Avi Yehuda

Exploring Data Filtering Techniques when Using Pandas to Read Parquet Files. How they work, and which of the ways gives the best results.

In Spark, there are two common transformations to change the number of tasks; coalesce and repartition. They are very similar but not identical. Let's see how.

Apache Spark is a wonderful tool, but sometimes it needs a bit of tuning. We look at a use case involving reading data from a JDBC source.

clover is a great tool for generating code coverage reports from your unit tests. it can be executed as a plugin in eclipse, maven or ant. however, not everyone knows that it can also be used to collect coverage data of integration tests. this post explains how to collect coverage data with clover at runtime. this post assumes that you already know what are unit and integration tests. this post assumes that you know what clover is, and already used it either with eclipse, ant or maven. * let me assured you that even though the directions bellow seems complicated and clumsy at first, after doing them once or twice it is really easy to repeat them. motivation the default action of clover is to gather code coverage information during build time or compile time. therefore, this information includes just the coverage data created by unit tests. if you are developing web applications, you probably use more technologies to test your applications beside unit tests. these technologies may include httpunit/ htmlunit or automation technologies (like selenium). these technologies do not work at build time, they can only work during run time, where a web server is up and running and http calls are made. as a result, the code coverage made during build time is not reflecting the actual code coverage. we should be able to test the coverage while a server is running. the idea the idea is that we will first run clover regularly during build time. we will than take the clover artifacts, put them in our server and then run the integration tests. while running the integration tests, the clover database will be updated and we would be able to generate reports from it which will reflect both unit and integration tests. step 1 – preparation make sure that you have a web/application server (tomcat/jboss/weblogic…) with your web application already deployed. execute clover on your application as you would normally do (either by compiling the code on eclipse or by building with maven or ant, it doesn’t matter). the result of this action would be: clover db files. one of the outcome of executing the clover on your code are the db files. the db files hold all the information about your code and the coverage itself. the location of those files may change depending on the way you use clover and according to the way you configured clover in your environment. this is how the files looks like the .db file holds the information regarding your code (classes, methods and so on). all the other files hold all the coverage data. it is important that you will locate those files because we are going to use them in the next step. an instrumented code. another outcome of clover is that it instruments your code. a clover call is injected into each method so it would be reported in the coverage calculation. we will need this code. we will use this instrumented code in runtime to update the coverage data. if you use eclipse than the generated classes would be instrumented. if you use maven or ant than most chances are that a jar with all the instrumented code would be generated separately. search the instrument code jar. again, i can’t tell you exactly where it is located, but usually it generates a jar with a ‘clover’ postfix. example: if your jar name is my_app.jar, than the generated instrumented code jar will probably be something like my_app-clover.jar. so you will need to do some detective work here to find the instrumented classes/jar. if you are not sure whether the classes are instrumented or not, just decompile one of them with jad and search for the word clover inside of it. a code coverage report. this is a report with the unit test coverage. we don’t really need this, but it would be good so that we would be able to compare it with the report we will generate at the end. step 2 – updating the server the next steps are very important, please make sure you do them properly. replacing the existing application jar/classes with the instrumented jar/classes. take the instrumented jar/classes that were created by the clover and add it to your server’s classpath instead of the original jar/classes. the instrumented code will cause the clover db to be updated with the runtime data. adding clover jars and license. since we will use clover on runtime we will need also the clover jars in our server’s classpath. so add the clover own jars to your server’s classpath. if you are using clover with eclipse than these jars are located in the plugin folder of eclipse. if you are using maven than they will be loacted in your repository. * make sure you are using the same version of clover in your server as you used to generate the db files and instrumented code, otherwise it will not work and you will get error messages. also add the clover license file to the same location as the jars. adding the clover java argument. add the following java arg -dclover.initstring.basedir={ location of the db files that were created by the clover } . * notice – the path you have entered above is the path of the folder which contains the db files. example: -dclover.initstring.basedir=c:/workspace/mywebapp/target/clover. this java argument is used by the clover to locate the db files and update them. step 3 – restarting the server and running the tests now that hopefully all is set properly all that you need to do now is to restart your server and than running your integration tests. the tests should trigger the instrumented code which will call the clover api’s and will update the clover db. while running the tests: look at your log/console and search for error messages from clover. look at the folder which holds the clover db files. if everything is going as it should, new files will be created in this folder while running the tests. if not everything is going well the first time, don’t discourage, just go over each of the steps again. step 4 – generating an updated report if everything went well and new files were created in the db folder than that means you just need to generate a new report. if you are using clover with eclipse than you can simply push the reload button to reload the coverage data. if you are using maven or ant you can execute just the task which generates the report. another way is to use the clover htmlreporter to generate a report easily. now compare the new report to the old report. you should see that the new report coverage is much bigger than the old one since it contains also the integration tests coverage. * notice that not all the data is updated, even though the percentages are being updated, for some reason the calls counter does not. to summarize. as mentioned; yes, these instructions seems a bit complicated but after you succeed the first time, it is very easy to repeat it. in the company i work for we even made this whole process automatic and we are able to generate a full coverage report with unit and integrated tests combined. source: http://www.aviyehuda.com/2011/12/use-clover-to-generate-code-coverage-reports-of-your-integrationautomation-tests/

Recently I had to choose a validation framework or write one by myself. First I thought, no big deal, validation is not a complicated issue. But the more you think about it, the more you come to the conclusion that it is not as shallow as you think – you need to validate different types, you have different groups and many more issues… In short, writing a validation framework by yourself demands a lot of work. Luckily, JSR 303 solves this and the Hibernate implementation of the JSR does a pretty good job. Hibernate Validator is a JSR 303 implementation for bean validation. The way to work with this framework is first, to define constraints for java bean fields, and then, validating the bean. JSR 303 JSR 303 – defines a metadata model and API for entity validation. The default metadata source is annotations, with the ability to override and extend the meta-data through the use of XML. The API is not tied to a specific application tier or programming model. It is specifically not tied to either the web tier or the persistence tier, and is available for both server-side application programming, as well as for rich client Swing application developers. Hibernate Validator features Defining validation data using annotation and/or XML. Full object validation (including inner objects using recursion) Create customized constraints and validators. Customized error messages. Define groups(profiles). Create a Traversable Resolver. Using constraints Using XML Here is a simple example of a constraint using xml: com.mytest 2 In this example the field x can not be null. The field y can not be less than 2. Notice that the “Min” constraint has inner element – “value”. Notice the tag “”. It indicates the root path of all the beans. See also directions on how to load the XML file while using the validator. Using annotations Here is a simple code example: public class MyBean{@NotNullString x;@Min(2)int y;} This example is the equivalent to the previous xml example. Using both Using both annotations and XML constraints is possible. By default if you are using both only the XML is taken, unless you are using the attribute ‘ignore-annotations’ in the XML. Example: com.mytest.beans 2 2 Notice that the attribute ignore-annotations appears twice – for a bean and for a field. The default for a bean is ignore-annotations=”true” – this means that if you have an XML constraint for a bean, it will cancel the attribute constraint, Unless you will indicate that by ignore-annotations=”false” (look at the example). The default for a field is ignore-annotations=”false”. This means that by default annotations for a field are stronger (this is of course after you indicated that that the bean itself wont ignore annotations). If you wont that the XML will be stronger than you have to indicate that by ignore-annotations=”true” (look at the example in the “x2″ constraint). Existing constraints These constraints are a part of the hibernate validation framework: Constraint path Parameters javax.validation.constraints.AssertTrue (none) javax.validation.constraints.AssertFalse (none) javax.validation.constraints.NotNull (none) javax.validation.constraints.Null (none) javax.validation.constraints.Max value(mandatory) javax.validation.constraints.Min value(mandatory) javax.validation.constraints.DecimalMax value(mandatory) javax.validation.constraints.DecimalMin value(mandatory) javax.validation.constraints.Pattern regexp(mandatory) flags(optional) javax.validation.constraints.Past (none) javax.validation.constraints.Future (none) javax.validation.constraints.Size min(optional) max(optional) javax.validation.constraints.Digits integer(mandatory) fraction(mandatory) org.hibernate.constraints.Email (none) org.hibernate.constraints.Length min(optional) max(optional) org.hibernate.constraints.NotEmpty (none) org.hibernate.constraints.Range min(optional) max(optional) Inner objects constraints If you have nested beans (beans which contain other beans) you can easily let the system validate also the inner objects by using the constraint ‘valid’. XML example: com.mytest.beans Annotation example: public class MyBean{@Valid //inner bean to be validated separatelyprivate InnerBean innerBean; public InnerBean getInnerBean() {return innerBean;}public void setInnerBean(InnerBean innerBean) {this.innerBean = innerBean;}public class InnerBean { @NotNullString xx; public String getXx() {return xx;}public void setXx(String xx) {this.xx = xx;} Validating Example of a simple validation: ValidatorFactory factory = Validation.buildDefaultValidatorFactory();Validator validator = factory.getValidator();Set> constraintViolations = validator.validate(bean); Loading a constraints XML file As mentioned, you don’t have to use an XML file for defining constraints, you can just use annotations. But if you do want an XML file, you will have to load the file. Example: Configuration config = Validation.byDefaultProvider().configure();FileInputStream in = new FileInputStream( new File("resources/demo-constraints.xml"));config.addMapping(in);// Building the customized factory// (along with the changed configuration)ValidatorFactory factory = config.buildValidatorFactory();Validator validator = factory.getValidator(); The result The result(as you can see in the example above) is a collection of ConstraintViolation. Each ConstraintViolation holds the problematic field, it’s value and the error message itself. Example of reading the result: Set> constraintViolations = validator.validate(bean);//printing the resultsfor (ConstraintViolation constraintViolation : constraintViolations) {System.out.println(constraintViolation.getPropertyPath() + " -> " +constraintViolation.getMessage());} This object can be easily transformed to a more generic object like ValidationException or CyotaSoapException and so on. Customized constraints and validators If you want to create a new constraint you will have to create the constraint annotation interface and the validator class. Creating the constraint interface Here is a simple constraint example @Target( { METHOD, FIELD, ANNOTATION_TYPE })@Retention(RUNTIME)@Constraint(validatedBy = MyValidator.class)@Documentedpublic @interface MyConstraint{// These next parameters exist in every constraintString message() default "{com.mytest.MyConstraint.message}";Class[] groups() default {};Class[] payload() default {};// These next parameters are added// They are the constraint's attributesString myOptionalValue() default ""; //this parameter has a default valueString myMustValue(); //this parameters need to get input from the user} Notice the @Constraint annotation. It signifies the class that suppose to validates this constraint. Notice the message class member. It holds an error message or, like in this case, an error code. It will later be interpreted as a literal error message. The payload member holds payload objects. These objects carry additional data attached to the constraints that can be fetched when validating. Nested constraints You can also overload constraints very easily. For example, let’s say I want to create a new constraint which also checks that the value is not null. In this case, all I have to do is this: @Target( { METHOD, FIELD, ANNOTATION_TYPE })@Retention(RUNTIME)@Constraint(validatedBy = MyValidator.class)@Documented@NotNullpublic @interface MyConstraint{String message() default "{com.mytest.MyConstraint.message}";Class[] groups() default {};Class[] payload() default {};} In the example, notice the @NotNull annotation. Creating the validator class Here is an example of a simple validator: public class MyValidatorimplements ConstraintValidator {MyConstraint MyConstraint;/** * This function recives the constraint instance * (along with the user values) */public void initialize(MyConstraint MyConstraint) {this.MyConstraint=MyConstraint;}/** * The value is the actual object instance. * */public boolean isValid(String value, ConstraintValidatorContext arg1) { //using input from the userreturn value!=null && value.startsWith(MyConstraint.myMustValue());} The above code shows an example of a validator which validates that the value of the given string starts with a given character. The validator implements the ConstraintValidator interface. Notice that the constraint is given as input to the initialize() function. The value itself is input to the isValid() function Customizing error messages Each error has an error template. The error template is defined in the constraint annotation interface. This error template is later translated into an error message. The actual error message may be defined in 2 places: 1. Inside the constraint deceleration. The error message be defined when defining the constraint, whether it you are using XML or annotations. XML example: x is too small 2 Java example: public class MyBean{@Min(value = 2, message="x is too small")private String x;public String getX() {return x;}public void setX(String x) {this.x = x;} 2. Using a separate properties message. You may want to load a separate properties file containing the error messages according to the error template. loading the messages properties file is done using the validation factory configuration: Configuration config = Validation.byDefaultProvider().configure();// Using a properties file for customized error messagesFileInputStream in = new FileInputStream(new File("resources/messages.properties"));ResourceBundleMessageInterpolator messageInterpolator =new ResourceBundleMessageInterpolator( new PropertyResourceBundle(in));// Setting a messages properties fileconfig.messageInterpolator(messageInterpolator);in = new FileInputStream(new File("resources/demo-constraints.xml"));config.addMapping(in);// Building the customized factory (along with the changed configuration)ValidatorFactory factory = config.buildValidatorFactory();Validator validator = factory.getValidator(); Using groups There may be occasions when you will want to create a single constraint but with different values for different situations. For example, let’s say you want to create a username field and let him a minimum constraint. But, one time it will have minimum of 5 characters and another time it will have minimum of 6 characters. For cases like this you will want to use groups. To do so, you will have to create a group, create constraints for that group and last, validate objects by attaching the group. Please follow the steps below Create a group To create a group you simply create a new interface. example: public interface MyBeanGroup{} Create a constraint for the group XML example: com.mytest.groups.MyBeanGroup Annotations example: public class MyBean{@NotNull(groups = MyBeanGroup.class)private String x; public String getX() {return x;}public void setX(String x) {this.x = x;} Validate an object using the group Set> constraintViolations = validator.validate(bean, MyBeanGroup.class); The default group The default group is javax.validation.groups.Default. If you don’t assign a constraint any group it applies to the default group. If you are validating an object without using any group, it is validated as a part of the default group. Groups inheritance You can also create an group inheritance tree. In this way, if you validate an object using a group it will prefer constraints that are defined to it. But if there are no such constraints, then it will also take the constraints of it’s parents. Example, this is a group which inherits the default group: public interface MyBeanGroupextends javax.validation.groups.Default{} All the constraints that are defined for that group will apply to it. But also all the constraints which do not apply to any group(and by which apply to the default group), will also apply to it, since it exteds the default group. Jar dependency validation-api-1.0.0.GA.jar hibernate-validator-4.0.2.GA slf4j-api-1.4.2.jar slf4j-simple-1.4.2.jar log4j-1.2.15.jar Only for java5 jaxb-xjc-2.1.6.jar jaxb-impl-2.1.6.jar jaxb-api-2.1.jar activation-1.1.jar geronimo-stax-api_1.0_spec-1.0.1.jar References JSR 303 specification- Bean validation Product main page Hibernate Validator documentation Download demo project from http://www.aviyehuda.com/