Over the years I have been a panelist in many of the interviews for Java Developers. I have previously written a post titled Top 7 tips for succeeding in a technical interview for software engineers which covers few of the general guidelines. In this post I will share a mind map containing general topics covered in a Java developer interview. I have prepared this as a general reference for myself to remember the pointers and to keep a common standard across the multiple interviews. XMind gives a nice listing of the map. You can find the map here. Here is Image which you can download and use. Finally here is a old fashioned tabbed content list which is easier to copy paste. Java-Topics OOPs Encapsulation Abstraction Inheritance Interface - Abstract Class Casting IS-A vs HAS-A Relationships Aggregation vs Composition Plymorphism Method overloading vs Method Overloading Compile time vs Runtime Threads Creating threads Multitasking Synchronization Thread Transitions Marker Interface Serialization Clonnable Shallow copy vs Deep Copy Collections Map, List and Set Equals - Hashcode Legacy - Synchronized Classes JVM Stack vs Heap Memory Garbage Collection JRE, JVM, JDK Class loaders Exception Checked Vs Unchecked Exceptions Exception handling best practices try, catch, finally, throw, throws APIs Files String - StringBuffer - String Builder Java IO XML SAX Based & DOM Based JAXB - Java API for XML Binding Access specifier Access modifier public protected deafult private final static synchronized abstract transient volatile Inner/Nested Classes JavaEE Basics Packaging the Applications WAR EAR Basics MVC Servlets Listeners Lifecycle JSPs APIs JPA JAX-WS SOAP, WSDL Webservices basics Contract first vs JAX-RS RESTful and its advantages JSF This is a work in progress and I hope to refine it further. Let me know if you have any comments. - See more at: http://jyops.blogspot.ie/2013/10/a-mindmap-for-java-developer-interviews.html#sthash.K0A5wDAz.dpuf

In this seventh post of my series on addressing the issue of too many parameters in a Java method or constructor, I look at using state to reduce the need to pass parameters. One of the reasons I have waited until the 7th post of this series to address this is that it is one of my least favorite approaches for reducing parameters passed to methods and constructors. That stated, there are multiple flavors of this approach and I definitely prefer some flavors over others. Perhaps the best known and most widely scorned approach in all of software development for using state to reduce parameter methods is the use global variables. Although it may be semantically accurate to say thatJava does not have global variables, the reality is that for good or for bad the equivalent of global variables is achieved in Java via public static constructs. A particularly popular way to achieve this in Java is via the Stateful Singleton. In Patterns of Enterprise Application Architecture, Martin Fowler wrote that "any global data is always guilty until proven innocent." Global variables and "global-like" constructs in Java are considered bad form for several reasons. They can make it difficult for developers maintaining and reading code to know where the values are defined or last changed or even come from. By their very nature and intent, global data violates the principles of encapsulation and data hiding. Miško Hevery has written the following regarding the problems of static globals in an object-oriented language: Accessing global state statically doesn’t clarify those shared dependencies to readers of the constructors and methods that use the Global State. Global State and Singletons make APIs lie about their true dependencies. ... The root problem with global state is that it is globally accessible. In an ideal world, an object should be able to interact only with other objects which were directly passed into it (through a constructor, or method call). Having state available globally reduces the need for parameters because there is no need for one object to pass data to another object if both objects already have direct access to that data. However, as Hevery put it, that's completely orthogonal to the intent of object-oriented design. Mutable state is also an increasing problem as concurrent applications become more common. In his JavaOne 2012 presentation on Scala, Scala creator Martin Odersky stated that "every piece of mutable state you have is a liability" in a highly concurrent world and added that the problem is "non-determinism caused by concurrent threads accessing shared mutable state." Although there are reasons to avoid mutable state, it still remains a generally popular approach in software development. I think there are several reasons for this including that it's superfically easy to write mutable state sharing code and mutable shared code does provide ease of access. Some types of mutable data are popular because those types of mutable data have been taught and learned as effective for years. Finally, three are times when mutable state may be the most appropriate solution. For that last reason and to be complete, I now look at how the use of mutable state can reduce the number of parameters a method must expect. Stateful Singleton and Static Variables A Java implementation of Singleton and other public Java static fields are generally available to any Java code within the same Java Virtual Machine (JVM) and loaded with the same classloader [for more details, see When is a Singleton not a Singleton?]. Any data stored universally (at least from JVM/classloader perspective) is already available to client code in the same JVM and loaded with the same class loader. Because of this, there is no need to pass that data between clients and methods or constructors in that same JVM/classloader combination. Instance State While "statics" are considered "globally available," narrower instance-level state can also be used in a similar fashion to reduce the need to pass parameters between methods of the same class. An advantage of this over global variables is that the accessibility is limited to instances of the class (private fields) or instances of the class's children (private fields). Of course, if the fields are public, accessibility is pretty wide open, but the same data is not automatically available to other code in the same JVM/classloader. The next code listing demonstrates how state data can and sometimes is used to reduce the need for parameters between two methods internal to a given class. Example of Instance State Used to Avoid Passing Parameters /** * Simple example of using instance variable so that there is no need to * pass parameters to other methods defined in the same class. */ public void doSomethingGoodWithInstanceVariables() { this.person = Person.createInstanceWithNameAndAddressOnly( new FullName.FullNameBuilder(new Name("Flintstone"), new Name("Fred")).createFullName(), new Address.AddressBuilder(new City("Bedrock"), State.UN).createAddress()); printPerson(); } /** * Prints instance of Person without requiring it to be passed in because it * is an instance variable. */ public void printPerson() { out.println(this.person); } The above example is somewhat contrived and simplified, but does illustrate the point: the instance variableperson can be accessed by other instance methods defined in the same class, so that instance does not need to be passed between those instance methods. This does reduce the signature of potentially (public accessibility means it may be used by external methods) internal methods, but also introduces state and now means that the invoked method impacts the state of that same object. In other words, the benefit of not having to pass the parameter comes at the cost of another piece of mutable state. The other side of the trade-off, needing to pass the instance of Person because it is not an instance variable, is shown in the next code listing for comparison. Example of Passing Parameter Rather than Using Instance Variable /** * Simple example of passing a parameter rather than using an instance variable. */ public void doSomethingGoodWithoutInstanceVariables() { final Person person = Person.createInstanceWithNameAndAddressOnly( new FullName.FullNameBuilder(new Name("Flintstone"), new Name("Fred")).createFullName(), new Address.AddressBuilder(new City("Bedrock"), State.UN).createAddress()); printPerson(person); } /** * Prints instance of Person that is passed in as a parameter. * * @param person Instance of Person to be printed. */ public void printPerson(final Person person) { out.println(person); } The previous two code listings illustrate that parameter passing can be reduced by using instance state. I generally prefer to not use instance state solely to avoid parameter passing. If instance state is needed for other reasons, than the reduction of parameters to be passed is a nice side benefit, but I don't like introducing unnecessary instance state simply to remove or reduce the number of parameters. Although there was a time when the readability of reduced parameters might have justified instance state in a large single-threaded environment, I feel that the slight readability gain from reduced parameters is not worth the cost of classes that are not thread-safe in an increasingly multi-threaded world. I still don't like to pass a whole lot of parameters between methods of the same class, but I can use the parameters object (perhaps with a package-private scope class) to reduce the number of these parameters and pass that parameters object around instead of the large number of parameters. JavaBean Style Construction The JavaBeans convention/style has become extremely popular in the Java development community. Many frameworks such as Spring Framework and Hibernate rely on classes adhering to the JavaBeans conventions and some of the standards like Java Persistence API also are built around the JavaBeans conventions. There are multiple reasons for the popularity of the JavaBeans style including its ease-of-use and the ability to usereflection against this code adhering to this convention to avoid additional configuration. The general idea behind the JavaBean style is to instantiate an object with a no-argument constructor and then set its fields via single-argument "set" methods and access it fields via no-argument "get" methods. This is demonstrated in the next code listings. The first listing shows a simple example of a PersonBean class with no-arguments constructor and getter and setter methods. That code listing also includes some of the JavaBeans-style classes it uses. That code listing is followed by code using that JavaBean style class. Examples of JavaBeans Style Class public class PersonBean { private FullNameBean name; private AddressBean address; private Gender gender; private EmploymentStatus employment; private HomeownerStatus homeOwnerStatus; /** No-arguments constructor. */ public PersonBean() {} public FullNameBean getName() { return this.name; } public void setName(final FullNameBean newName) { this.name = newName; } public AddressBean getAddress() { return this.address; } public void setAddress(final AddressBean newAddress) { this.address = newAddress; } public Gender getGender() { return this.gender; } public void setGender(final Gender newGender) { this.gender = newGender; } public EmploymentStatus getEmployment() { return this.employment; } public void setEmployment(final EmploymentStatus newEmployment) { this.employment = newEmployment; } public HomeownerStatus getHomeOwnerStatus() { return this.homeOwnerStatus; } public void setHomeOwnerStatus(final HomeownerStatus newHomeOwnerStatus) { this.homeOwnerStatus = newHomeOwnerStatus; } } /** * Full name of a person in JavaBean style. * * @author Dustin */ public final class FullNameBean { private Name lastName; private Name firstName; private Name middleName; private Salutation salutation; private Suffix suffix; /** No-args constructor for JavaBean style instantiation. */ private FullNameBean() {} public Name getFirstName() { return this.firstName; } public void setFirstName(final Name newFirstName) { this.firstName = newFirstName; } public Name getLastName() { return this.lastName; } public void setLastName(final Name newLastName) { this.lastName = newLastName; } public Name getMiddleName() { return this.middleName; } public void setMiddleName(final Name newMiddleName) { this.middleName = newMiddleName; } public Salutation getSalutation() { return this.salutation; } public void setSalutation(final Salutation newSalutation) { this.salutation = newSalutation; } public Suffix getSuffix() { return this.suffix; } public void setSuffix(final Suffix newSuffix) { this.suffix = newSuffix; } @Override public String toString() { return this.salutation + " " + this.firstName + " " + this.middleName + this.lastName + ", " + this.suffix; } } package dustin.examples; /** * Representation of a United States address (JavaBeans style). * * @author Dustin */ public final class AddressBean { private StreetAddress streetAddress; private City city; private State state; /** No-arguments constructor for JavaBeans-style instantiation. */ private AddressBean() {} public StreetAddress getStreetAddress() { return this.streetAddress; } public void setStreetAddress(final StreetAddress newStreetAddress) { this.streetAddress = newStreetAddress; } public City getCity() { return this.city; } public void setCity(final City newCity) { this.city = newCity; } public State getState() { return this.state; } public void setState(final State newState) { this.state = newState; } @Override public String toString() { return this.streetAddress + ", " + this.city + ", " + this.state; } } Example of JavaBeans Style Instantiation and Population public PersonBean createPerson() { final PersonBean person = new PersonBean(); final FullNameBean personName = new FullNameBean(); personName.setFirstName(new Name("Fred")); personName.setLastName(new Name("Flintstone")); person.setName(personName); final AddressBean address = new AddressBean(); address.setStreetAddress(new StreetAddress("345 Cave Stone Road")); address.setCity(new City("Bedrock")); person.setAddress(address); return person; } The examples just shown demonstrate how the JavaBeans style approach can be used. This approach makes some concessions to reduce the need to pass a large number of parameters to a class's constructor. Instead, no parameters are passed to the constructor and each individual attribute that is needed must be set. One of the advantages of the JavaBeans style approach is that readability is enhanced as compared to a constructor with a large number of parameters because each of the "set" methods is hopefully named in a readable way. The JavaBeans approach is simple to understand and definitely achieves the goal of reducing lengthy parameters in the case of constructors. However, there are some disadvantages to this approach as well. One advantage is a lot of tedious client code for instantiating the object and setting its attributes one-at-a-time. It is easy with this approach to neglect to set a required attribute because there is no way for the compiler to enforce all required parameters be set without leaving the JavaBeans convention. Perhaps most damaging, there are several objects instantiated in this last code listing and these objects exist in different incomplete states from the time they are instantiated until the time the final "set" method is called. During that time, the objects are in what is really an "undefined" or "incomplete" state. The existence of "set" methods necessarily means that the class's attributes cannot be final, rendering the entire object highly mutable. Regarding the prevalent use of the JavaBeans pattern in Java, several credible authors have called into questionits value. Allen Holub's controversial article Why getter and setter methods are evil starts off with no holds barred: Though getter/setter methods are commonplace in Java, they are not particularly object oriented (OO). In fact, they can damage your code's maintainability. Moreover, the presence of numerous getter and setter methods is a red flag that the program isn't necessarily well designed from an OO perspective. Josh Bloch, in his less forceful and more gently persuasive tone, says of the JavaBeans getter/setter style: "The JavaBeans pattern has serious disadvantages of its own" (Effective Java, Second Edition, Item #2). It is in this context that Bloch recommends the builder pattern instead for object construction. I'm not against using the JavaBeans get/set style when the framework I've selected for other reasons requires it and the reasons for using that framework justify it. There are also areas where the JavaBeans style class is particularly well suited such as interacting with a data store and holding data from the data store for use by the application. However, I am not a fan of using the JavaBeans style for instantiating a question simply to avoid the need to pass parameters. I prefer one of the other approaches such as builder for that purpose. Benefits and Advantages I've covered different approaches to reducing the number of arguments to a method or constructor in this post, but they also share the same trade-off: exposing mutable state to reduce or eliminate the number of parameters that must be passed to a method or to a constructor. The advantages of these approaches are simplicity, generally readable (though "globals" can be difficult to read), and ease of first writing and use. Of course, their biggest advantage from this post's perspective is that they generally eliminate the need for any parameter passing. Costs and Disadvantages The trait that all approaches covered in this post share is the exposure of mutable state. This can lead to an extremely high cost if the code is used in a highly concurrent environment. There is a certain degree of unpredictability when object state is exposed for anyone to tinker with it as they like. It can be difficult to know which code made the wrong change or failed to make a necessary change (such as failing to call a "set" method when populating a newly instantiated object). Conclusion Some of the approaches covered in this post are highly popular despite their drawbacks. This may be for a variety of reasons including prevalence of use in popular frameworks (forcing users of the framework to use that style and also providing examples to others for their own code development). Other reasons for these approaches' popularity is the relative ease of initial development and the seemingly (deceptively) relatively little thought that needs to go into design with these approaches. In general, I prefer to spend a little more design and implementation effort to use builders and less mutable approaches when practical. However, there are cases where these mutable approaches work well in reducing the number of parameters passed around and introduce no more risk than was already present. My feeling is that Java developers should carefully consider use of any mutable Java classes and ensure that the mutability is either desired or is a cost that is justified by the reasons for using a mutable state approach.

JavaScript’s prototype-based inheritance is interesting and has its uses, but sometimes one just wants to express classical inheritance, familiar from C++ and Java. This need has been recognized by the ECMAScript committee and classes are being discussed for inclusion in the next version of the standard. It was surprisingly hard for me to find a good and simple code sample that shows how to cleanly and correctly express inheritance with ES5 (a lot of links discuss how to implement the pre-ES5 tools required for that) and explains why the thing works. Mozilla’s Object.Create reference came close, but not quite there because it still left some open questions. Hence this short post. Without further ado, the following code defines a parent class named Shape with a constructor and a method, and a derived class named Circle that has its own method: // Shape - superclass // x,y: location of shape's bounding rectangle function Shape(x, y) { this.x = x; this.y = y; } // Superclass method Shape.prototype.move = function(x, y) { this.x += x; this.y += y; } // Circle - subclass function Circle(x, y, r) { // Call constructor of superclass to initialize superclass-derived members. Shape.call(this, x, y); // Initialize subclass's own members this.r = r; } // Circle derives from Shape Circle.prototype = Object.create(Shape.prototype); Circle.prototype.constructor = Circle; // Subclass methods. Add them after Circle.prototype is created with // Object.create Circle.prototype.area = function() { return this.r * 2 * Math.PI; } The most interesting part here, the one that actually performs the feat of inheritance is these two lines, so I’ll explain them a bit: Circle.prototype = Object.create(Shape.prototype); Circle.prototype.constructor = Circle; The first line is the magic – it sets up the prototype chain. To understand it, you must first understand that "the prototype of an object" and "the .prototype property of an object" are different things. If you don’t, go read on that a bit. The first line, interpreted very technically, says: the prototype of new objects created with the Circle constructor is an object whose prototype is the prototype of objects created by Shape constructor. Yeah, that’s a handful. But it can be simplified as: each Circle has a Shape as its prototype. What about the second line? While not strictly necessary, it’s there to preserve some useful invariants, as we’ll see below. Since the assignment to Circle.prototype kills the existing Circle.prototype.constructor (which was set to Circle when the Circle constructor was created), we restore it. Let’s whip up a JavaScript console and load that code inside, to quickly try some stuff: > var shp = new Shape(1, 2) undefined > [shp.x, shp.y] [1, 2] > shp.move(1, 1) undefined > [shp.x, shp.y] [2, 3] … but we’re here for the circles: > var cir = new Circle(5, 6, 2) undefined > [cir.x, cir.y, cir.r] [5, 6, 2] > cir.move(1, 1) undefined > [cir.x, cir.y, cir.r] [6, 7, 2] > cir.area() 12.566370614359172 So far so good, a Circle initialized itself correctly using the Shape constructor; it responds to the methods inherited from Shape, and to its own area method too. Let’s check that the prototype shenanigans worked as expected: > var shape_proto = Object.getPrototypeOf(shp) undefined > var circle_proto = Object.getPrototypeOf(cir) undefined > Object.getPrototypeOf(circle_proto) === shape_proto true Great. Now let’s see what instanceof has to say: > cir instanceof Shape true > cir instanceof Circle true > shp instanceof Shape true > shp instanceof Circle false Finally, here are some things we can do with the constructor property that wouldn’t have been possible had we not preserved it: > cir.constructor === Circle true // Create a new Circle object based on an existing Circle instance > var new_cir = new cir.constructor(3, 4, 1.5) undefined > new_cir Circle {x: 3, y: 4, r: 1.5, constructor: function, area: function} A lot of existing code (and programmers) expect the constructor property of objects to point back to the constructor function used to create them with new. In addition, it is sometimes useful to be able to create a new object of the same class as an existing object, and here as well the constructor property is useful. So that is how we express classical inheritance in JavaScript. It is very explicit, and hence on the long-ish side. Hopefully the future ES standards will provide nice sugar for succinct class definitions.

I wanted to know: how would Java EE compare to Node.js in this particular case?

This article will be useful if you want to support both PostgreSQL and SQLite using JDBC. It will be especially useful if you: Are already accessing values from your (PostgreSQL) database using the regular JDBC ResultSet interface, like: Date d = rs.getDate("date_field"); BigDecimal bd = rs.getBigDecimal("bigdecimal_field"); And it is creating trouble when doing the same for SQLite, but you don't want to change that code. Are already retrieving autogenerated keys in PostgreSQL with a RETURNING clause, but this won't work in SQLite. You want a unified solution that works for both databases. Thought foreign keys are enforced in SQLite by default (like in PostgreSQL) and crashed with a wall. SQLite is allowing you to delete entries from your tables even when they are referenced in another table and you have explicitly told SQLite about it with a REFERENCES table_name(field_name) clause. Are having trouble with the differences between PostgreSQL and SQLite dialects (mostly concerning data types), for example, when making query filters with boolean values. Had your own way to manage exceptions for PostgreSQL and it is not working for SQLite (obviously). You want SQLite to fit into the model you already have. Other stuff might appear if you keep up... A few months ago I wanted to migrate an app to use SQLite as a data backend. In fact, I wanted it to work with both PostgreSQL and SQLite indistinctly (but not at the same time). I wanted to switch between these two databases easily without changing any code. I did it, but along the way I had to solve some problems that might be interesting to many other people. Many solutions I found were spread across the web, but there was no single place that explained how to completely achieve what I wanted. So, the aim of this post is to try to condense my learning into one article that may be of help to others as a (semi) complete guide. This guide might be useful not only to those creating their own frameworks, but for anyone who doesn't use any and are willing to try some quirks and tricks to make their app work. THE BEGINNING There are many cross-database incompatibilities between PostgreSQL and SQLite, most notably on data types. If you want to have the same code to work for both databases, you better use a framework that manages this for you. But here's the thing: the framework I use is created by myself, and didn't (completely) take these differences into account, since I mainly use PostgreSQL as database; that's how and why my problems arose. My framework conveys many things, but I focus here in the data access part. It uses some JDBC driver to connect to the databases, but it provides more abstract ways to do it; that's pretty much the data access part of the framework. A basic DAO class for my framework would look like this: public class MyDAO extends BaseDAO { public MyDAO() { super("context_alias", new DefaultDataMappingStrategy() { @Override public Object createResultObject(ResultSet rs) throws SQLException { MyModel model = (MyModel)ObjectsFactory.getObject("my_model_alias"); model.setStringField(rs.getString("string_field")); model.setIntegerField(rs.getInt("integer_field")); model.setBigDecimalField(rs.getBigDecimal("bigdecimal_field")); model.setDateField(rs.getDate("date_field")); model.setBooleanField(rs.getBoolean("boolean_field")); return model; } }); } @Override public String getTableName() { return "table_name"; } @Override public String getKeyFields() { return "string_field|integer_field"; } @Override protected Map getInsertionMap(Object obj) { Map map = new HashMap(); MyModel model = (MyModel) obj; map.put("string_field", model.getStringField()); map.put("integer_field", model.getIntegerField()); map.put("bigdecimal_field", model.getBigDecimalField()); map.put("date_field", model.getDateField()); map.put("boolean_field", model.getBooleanField()); return map; } @Override protected Map getUpdateMap(Object obj) { Map map = new HashMap(); MyModel model = (MyModel) obj; map.put("bigdecimal_field", model.getBigDecimalField()); map.put("date_field", model.getDateField()); map.put("boolean_field", model.getBooleanField()); return map; } @Override public String getFindAllStatement() { return "SELECT * FROM :@ "; } So, that I wanted to switch between databases without changing code means that I wanted to switch without changing my DAO classes. For SQLite, I used the xerial-jdbc-sqlite driver. I talk about drivers because there are some things that might be driver-specific when solving some problems; so when I say 'SQlite does it this way', I generally mean 'xerial-jdbc-sqlite driver does it this way'. Now, let's start. WARNING: Some of the solutions I give here fit into my framework, but might not directly fit into your code. It's up to you to imagine how to adapt what I provide here. DATA TYPES Since there are some differences between PostgreSQL and SQLite regarding data types, and I wanted to continue to access database values through the regular ResultSet interface, I had to have some mechanism to intercept the call to, for instance, resultset.getDate("date_field"). So I created a ResultSetWrapper class that would redefine the methods I was interested in, like this: public class ResultSetWrapper implements ResultSet { // The wrappped ResultSet ResultSet wrapped; /* I will use this DateFormat to format dates. I'm assuming an SQLite style pattern. I should not */ SimpleDateFormat df = new SimpleDateFormat("yyyy-mm-dd"); public ResultSetWrapper(ResultSet wrapped) { this.wrapped = wrapped; } /* Lots of ResultSet methods implementations go here, but this is an example of redefining a method I'm interested in changing its behavior: */ public Date getDate(String columnLabel) throws SQLException { Object value = this.wrapped.getObject(columnLabel); return (Date)TypesInferreer.inferDate(value); } } The getDate() method in ResultSetWrapper relies on TypesInferreer to convert the value retrieved to a Date value. All data types convertions would be encapsulated inside TypesInferreer, which would have methods to convert from different data types as needed. For instance, it would have a method like this one: public static Object inferDate(Object value) { java.util.Date date; // Do convertions here (convert value and asign to date) return date; } Which tries to convert any value to a Date (I'll show the actual implementation further). Now, instead of using the original resultset retrieved from saying preparedStatement.executeQuery(), you use new ResultSetWrapper(preparedStatement.executeQuery()). That's what my framework does: it passes this new resultset to DAO objects. Now let's see some type conversions. Mixing PostgreSQL Date and SQLite Long/String You could store Date values as text in a SQLite database (eg. '2013-10-09'); this you can do manually when creating the database, but when SQLite stores a Date object, by default it converts it to a Long value. There is no problem with this when saving the value to the SQLite database, but if you try to retrieve it using resultset.getDate("date_field"), then things get messy; It simply won't work (CastException). How do you access Date values, then? You create this method in TypesInfereer, which covers both String and Long variations: public static Object inferDate(Object value) { java.util.Date date = null; if(value == null) return null; if(value instanceof String) { try { date = df.parse((String)value); } catch (ParseException ex) { // Deal with ex } } else if(value instanceof Long) { date = new java.util.Date((Long)value); } else { date = (Date)value; } return new Date(date.getTime()); } And as you saw, the getDate() function in ResultSetWrapper is redefined like this: @Override public Date getDate(String columnLabel) throws SQLException { Object value = this.wrapped.getObject(columnLabel); return (Date)TypesInferreer.inferDate(value); } Now all DAOs can retrieve Date values from both databases indistinctly, using resultset.getDate("date_field"). Mixing PostgreSQL Numeric and SQLite Integer/Double/... My SQLite driver didn't implement the getBigDecimal() function. It complained like this when I called it: java.sql.SQLException: not implemented by SQLite JDBC driver. So I had to come up with a solution that was valid for both PostgreSQL and SQlite. This is what I did in ResultSetWrapper: @Override public BigDecimal getBigDecimal(String columnLabel) throws SQLException { Object value = this.wrapped.getObject(columnLabel); return (BigDecimal)TypesInferreer.inferBigDecimal(value); } But value would get different types depending on the actual value stored in the database; it could be an Integer, or a Double, or perhaps something else. I solved all the cases by doing this in TypesInfereer: public static Object inferBigDecimal(Object value) { if(value == null) return null; if(value instanceof BigDecimal == false) { return new BigDecimal(String.valueOf(value)); } return value; } Anyway, the String constructor of BigDecimal is the recommended one, so everything's fine with this. Now you can retrieve BigDecimal values using resultset.getBigDecimal("bigdecimal_field") from both databases. Mixing PostgreSQL Boolean and SQLite Integer SQLite doesn't have boolean values. Instead, it interprets any other value as boolean by following some rules. When SQLite saves a Boolean value to the database, it saves it as 0 or 1 for false or true respectively. Also, because drivers can interpret any value as boolean, you can use resultset.getBoolean("boolean_field") and it will work as expected by the rules. But the problem I faced was when creating filters. If a value for true is stored as 1 in the SQLite database, you can't expect the clause WHERE boolean_field = true to work. You will never find a match. Instead, you should have said WHERE boolean_field = 1. In my app, I created filters like this: dao.addFilter(new FilterSimple("boolean_field", true)); Now I needed FilterSimple to infer that, for SQLite, I meant 1 instead of true. So I created what I called a DatasourceVariation. These are objects that are specific for each type of database and are used accross all data accesses, by DAOs, Filters, and other objects. These objects would take care of managing all my cross-database incompatibilities, including: The way to reference a database object: in PostgreSQL you must prepend the schema name to every database object you refer in your queries. In SQLite you don't. The way to manage exeptions: explained further in this post. The way to backup and restore data: explained further in this post. Expressing BETWEEN clauses: Explained further in this post. And also, infering boolean values. For VariationSQLite, I did this: @Override public Object getReplaceValue(Object value) { if(value instanceof Boolean) { if((Boolean)value == true) return new Integer(1); else return new Integer(0); } return value; } Now we can say dao.addFilter(new FilterSimple("boolean_field", true)) for both databases, assuming that FilterSimple uses the variation to adapt the value before constructing the clause. RETRIEVING AUTOGENERATED KEYS When you have autonumeric fields (eg. Serial), in PostgreSQL you can specify a RETURNING clause at the end of an INSERT statement to automatically retrieve the values of autogenerated fields by doing this: PreparedStatement pstm = conn.prepareStatement(queryWithReturningClause); // ex. select * from table_x returning field_x ResultSet rs = statement.executeQuery(); if(rs.next()) { // Get autogenerated fields from rs } But that won't work with SQLite. In SQLite, retrieving autogenerated fields conveys a process that goes from creating the statement, executing the query and explicitly asking for the generated values. Like this: PreparedStatement pstm = conn.prepareStatement(queryWITHOUTreturningClause, Statement.RETURN_GENERATED_KEYS); pstm.executeUpdate(); ResultSet rs = pstm.getGeneratedKeys(); if (rs != null && rs.next()) { // Get autogenerated fields from rs } The good news is that this code works both for PostgreSQL and SQLite, so I replaced my previous code for this, and didn't have to make any distinction between databases. ENFORCING FOREIGN KEYS You'd think that using a REFERENCES table_name(field_name) clause when creating a SQLite database table makes foreign keys to be checked when deleting, updating, etc. You're wrong! Foreign keys are not enforced in SQLite by default. You have to explicitly say it, and it's done when creating the connection (WARNING: This is very driver-specific): SQLiteConfig config = new SQLiteConfig(); config.enforceForeignKeys(true); Connection conn = DriverManager.getConnection("jdbc:sqlite:" + dataSourcePath, config.toProperties()); For PostgreSQL it's different, so you better have a connection pool for each type of database, and decide which one to use at runtime. My framework does exactly that. NOTE: If you are capable of getting the connection depending on the database type, then you can enforce foreign keys transparently for both databases (for PostgreSQL it happens naturally without extra code). For instance, you could have an abstract getConnection() method, and each database's connection pool would return the connection in its own way. MANAGING EXCEPTIONS I had defined some different types of database exceptions in my framework: ExceptionDBDuplicateEntry, ExceptionDBEntryReferencedElsewhere, etc, which would be thrown and raised to upper layers in my architecture. For PostgreSQL, these exceptions directly mapped to some constant codes (which normally are vendor/driver specific): UNIQUE_VIOLATION = "23505", FOREIGN_KEY_VIOLATION = "23503", etc. So, for PostgreSQL, I managed database exceptions something like this: @Override public void manageException(SQLException ex) throws ExceptionDBDuplicateEntry, ExceptionDBEntryReferencedElsewhere { if (ex.getSQLState() == null) { ex = (SQLException) ex.getCause(); } if (ex.getSQLState().equals(UNIQUE_VIOLATION)) { throw new ExceptionDBDuplicateEntry(); } else if(ex.getSQLState().equals(FOREIGN_KEY_VIOLATION)) { throw new ExceptionDBEntryReferencedElsewhere(); } else { DAOPackage.log(ex); throw new ExceptionDBUnknownError(ex); } } That won't work for SQLite, obviously! So, what I did was move the database exceptions management to the DataSourceVariation. The VariationPostgresql class would have a method similar to the one above. For VarialtionSQLite, I did sort of a hack, but it's something that has worked until now (maybe until I change my driver). @Override public void manageException(SQLException ex) throws ExceptionDBDuplicateEntry, ExceptionDBEntryReferencedElsewhere { // This is a hack (is it???) String message = ex.getMessage().toLowerCase(); if(message.contains("sqlite_constraint")) { if(message.contains("is not unique")) throw new ExceptionDBDuplicateEntry(); else if(message.contains("foreign key constraint failed")) throw new ExceptionDBEntryReferencedElsewhere(); else { DAOPackage.log(ex); throw new ExceptionDBUnknownError(ex); } } else { DAOPackage.log(ex); throw new ExceptionDBUnknownError(ex); } } Update: This technique might have some flaws. But hey, can you find a better approach right away? FIXING BETWEEN CLAUSE The problem with the BETWEEN clause appeared while using a filter like this: dao.addFilter(new FilterBetween("date_field", date1, date2)); // date1 and date2 are java.util.Date objects FilterBetween would create a BETWEEN clause by formatting Dates as Strings, normally with the format 'yyyy-MM-dd' (although this should be configurable). Since dates in SQLite are long values, we can't create a clause like date_field BETWEEN '2013-01-01' AND '2013-02-01'. It had to be something like date_field >=1357016400000 AND date_field <= 1359694800000. So, I moved the creation of BETWEEN clauses to.... that's right, to DataSourceVariation. VariationSQLite does it like this: @Override public String getBetweenExpression(String fieldName, Object d1, Object d2) { String filter = ""; try { Date dd1 = null; Date dd2 = null; SimpleDateFormat df = new SimpleDateFormat("yyyy-mm-dd"); // Remember, this should be configurable if(d1 instanceof String) dd1 = df.parse((String)d1); else dd1 = (Date)d1; if(d2 instanceof String)dd2 = df.parse((String)d2); else dd2 = (Date)d2; filter = fieldName + " >= " + dd1.getTime() + " AND " + fieldName + " <= " + dd2.getTime(); } catch (ParseException ex) { DAOPackage.log(ex); throw new ExceptionDBUnknownError(ex); } return filter; } CONCLUSIONS As you can see, there are many intricacies when making an app support multiple database types. All I did here was only to support PostgreSQL and SQLite, but who knows what is needed to support other databases at the same time too. You can't expect JDBC alone will do all the work, so be prepared to solve some problems (and another problem, and another, ...) to make a database migration. And please, share your journey.

Introduction One of the nice recent features of Spring (2.x era) is support for custom XML. This is the way that Spring itself has added all kinds of new tags such as and . The way this works is pretty elegant, to the point that it makes an interesting alternative for configuring Java using XML, particularly if the application already uses Spring. I’ve written an example application to try to give an easily-copied example of how it’s done. The example uses Spring and a custom XML parser to build dynamic Swing menus. It makes a nice comparison to doing dynamic Swing menus using the Digester version I posted a while back. Of course, this is not a good way to make Java menus in general! In most applications, this would be an example of Soft Coding. This would really only make sense in an application where it was really important to be able to add or remove menus without changing Java code. So treat it as a nice example, but please don’t start making your GUIs this way. Spring Custom XML Custom XML works in a Spring configuration file because Spring can dynamically validate and parse XML. To do this, Spring first has to be able to validate the XML it parses against a schema. It does this by looking for all files on the classpath called META-INF/spring.schemas. These files provide a location on the classpath for the XML schema that goes with a given namespace. For example, the “core” XML for Spring is defined in the beansnamespace. The META-INF/spring.schemas file in the spring-beans JAR has entries like this one: http\://www.springframework.org/schema/beans/spring-beans-3.0.xsd=org/springframework/beans/factory/xml/spring-beans-3.0.xsd So when we use the beans schema in our Spring XML, it knows where on the classpath to hunt down the schema so it can validate that XML. Once the schema is validated, Spring needs to find a “handler” that knows how to make Spring beans based on the XML. Spring finds handlers by looking through all the files on the classpath called META-INF/spring.handlers. Thespring.handlers file in the spring-beans JAR has entries like this one: http\://www.springframework.org/schema/p=org.springframework.beans.factory.xml.SimplePropertyNamespaceHandler It’s really the job of the handler to make bean definitions, not the regular Java objects that will live as beans in the Spring application context. This is because Spring still has to manage things like beans depending on other beans, which means Spring has to parse all the XML to figure out the dependency graph before any objects can be instantiated. Example Application Our example application has several parts: The spring.schemas and spring.handlers files in META-INF. An XML schema defining what is valid in our custom namespace. MenuNamespaceHandler, the entry class that allows us to register what XML elements go with what parser classes. MenuDefinitionParser, the actual XML parser for our custom XML namespace. A regular Spring XML configuration file that also includes our custom XML. A main class to get the whole thing kicked off. There’s also a Java class called MenuItem that we use to store the ID, the title, and any children of the menu item. It doesn’t know anything about Spring or XML; it’s just a POJO. Defining the custom XML The spring.schemas file is pretty simple. Note that it’s matching to a file on the classpath; Spring is not going to be looking out on the Internet for your XML schema at runtime. http\://anvard.org/springxml/menu.xsd=org/anvard/springxml/menu.xsd The spring.handlers file is also pretty simple. It just points to the right handler class: http\://anvard.org/springxml/menu=org.anvard.springxml.MenuNamespaceHandler The XML schema is omitted here; it’s an XML schema and not much need be said. Of note is that it allows for arbitrary nesting of elements inside other elements. One more piece of boilerplate; the namespace handler. Since our namespace is really simple and only contains one top-level element (menu), it’s a one-liner: public void init() { registerBeanDefinitionParser("menu", new MenuDefinitionParser()); } The parser is where it gets interesting. The parser will get called while Spring is reading the XML file, whenever it comes across an element that belongs to the matching namespace. However, it will only be called for the top-level element; it’s up to us to handle any nested elements as required. protected AbstractBeanDefinition parseInternal(Element element, ParserContext context) { BeanDefinitionBuilder builder = parseItem(element); List childElements = DomUtils.getChildElementsByTagName( element, "menu"); if (null != childElements && childElements.size() > 0) { ManagedList children = new ManagedList<>( childElements.size()); for (Element child : childElements) { children.add(parseInternal(child, context)); } builder.addPropertyValue("children", children); } return builder.getBeanDefinition(); } private BeanDefinitionBuilder parseItem(Element element) { BeanDefinitionBuilder builder = BeanDefinitionBuilder .rootBeanDefinition(MenuItem.class); String id = element.getAttribute("id"); if (StringUtils.hasText(id)) { builder.addPropertyValue("id", id); } String title = element.getAttribute("title"); if (StringUtils.hasText(title)) { builder.addPropertyValue("title", title); } String listener = element.getAttribute("listener"); if (StringUtils.hasText(listener)) { builder.addPropertyReference("listener", listener); } return builder; } In this case, because we allowed for the idea that a menu could contain child menus, we have to handle that here with some recursion. Note that for every element at whatever level, we are creating a separate Spring bean definition (that’s one purpose of the rootBeanDefinition() static method call). The really important thing to notice is that as we build the bean definition, we are not creating a MenuItem object directly, nor are we setting any properties directly. In fact, in the case of the children property, we are not even building a list of the correct type, as the MenuItem class expects to receive a list of MenuItem children, but we are building a list ofAbstractBeanDefinition. Spring handles all of the necessary wiring when it actually instantiates our MenuList objects, including looking up each of the references in the list and populating a new list with the real objects. One other thing that’s slightly confusing is that a reference to a single other Spring bean uses addPropertyReference(), while a managed list of Spring bean definitions uses addPropertyValue(). Using the custom XML Now that these items are in place, we can use the custom XML just the same as any other XML in a Spring configuration file. For example: Note that we can make our custom XML the default namespace so we don’t have to prefix our XML elements; we can also make the bean namespace the default as is more typical in a Spring XML configuration file. We can mix our custom XML freely with standard Spring XML. Also note that our custom XML can make references back to ordinary Spring beans as long as we do the right thing in our parser to make this work. We use a list called toplevel as a handy way of finding the outermost menu items for our menu bar. Once the XML is parsed, the beans are all loaded into the Spring application context and the structure of the XML no longer really applies. Using this file from our main class looks just the same as any Spring code: ClassPathXmlApplicationContext ctx = new ClassPathXmlApplicationContext("/menuDefinition.xml"); All of our menu items are available in the Spring application context, so we could do ctx.getBean("menu9") and get back the menu item with the title “Child 5”. Conclusion Even though many Spring users are shifting toward annotation-driven configuration, there are still things that are easier to do in XML, like creating many instances of a class with different properties. A custom XML namespace is a way to make Spring XML configuration more compact and more readable.

These days, no community member writes or speaks about using DataTables and DataSets for data operations. But, there are a number of real projects built using them, and many developers still feel happy when they use them in their projects. Sometimes it is not easy to completely replace DataTables with typed generic lists, particularly in bulky projects. But now is the right time to move, as future developers may not even learn about DataTables :). Generic collections have a number of advantages over DataTables. One cannot imagine a day without generic collections once he/she gets to know how beneficial they are. The following is a list of the reasons to move from DataTables to collections that I could think of now: DataTable stores boxed objects, and one needs to unbox values when needed. This adds overhead on the runtime environment. However, values in generic collections are strongly typed, so no boxing involved. Unboxing happens at runtime, as does the type checking. If there is a mismatch between types of source and target, it leads to a runtime exception. This may lead to a number of issues while using DataTables. In case of collections, as the types are checked at the compile time, such type mismatches are caught during compilation. .NET languages got very nice support for creating collections, like object initializer and collection initializer. We don’t have such features for DataTables. LINQ queries can be used on both DataTables and collections. But the experience of writing the queries on generic collections is better because of IntelliSense support provided by Visual Studio. DataTables are framework specific; we often see issues with serializing and de-serializing them in web services. Generic collections are easier to serialize and de-serialize, so they can be easily used in any service and consumed from a client written in any language. ORMs are becoming increasingly popular, and they use generic collections for all data operations. Mocking DataTables in unit tests is a pain, as it involves creating the structure of the table wherever needed. But a generic collection needs a class defined just once. These are my opinions on preferring collections over DataTables. Any feedback is welcome. Happy coding!

In my two immediately previous posts, I looked at reducing the number of parameters required for a constructor or method invocation via custom types and parameter objects. In this post, I look at use of thebuilder pattern to reduce the number of parameters required for a constructor with some discussion on how this pattern can even help with non-constructor methods that take too many parameters. In the Second Edition of Effective Java, Josh Bloch introduces use of the builder pattern in Item #2 for dealing with constructors that require too many parameters. Bloch not only demonstrates how to use the Builder, but explains it advantages over constructors accepting a large number of parameters. I will get to those advantages at the end of this post, but think it's important to point out that Bloch has devoted an entire item in his book to this practice. To illustrate the advantages of this approach, I'll use the following example Person class. It doesn't have all the methods I would typically add to such a class because I want to focus on its construction. Person.java (without Builder Pattern) package dustin.examples; /** * Person class used as part of too many parameters demonstration. * * @author Dustin */ public class Person { private final String lastName; private final String firstName; private final String middleName; private final String salutation; private final String suffix; private final String streetAddress; private final String city; private final String state; private final boolean isFemale; private final boolean isEmployed; private final boolean isHomewOwner; public Person( final String newLastName, final String newFirstName, final String newMiddleName, final String newSalutation, final String newSuffix, final String newStreetAddress, final String newCity, final String newState, final boolean newIsFemale, final boolean newIsEmployed, final boolean newIsHomeOwner) { this.lastName = newLastName; this.firstName = newFirstName; this.middleName = newMiddleName; this.salutation = newSalutation; this.suffix = newSuffix; this.streetAddress = newStreetAddress; this.city = newCity; this.state = newState; this.isFemale = newIsFemale; this.isEmployed = newIsEmployed; this.isHomewOwner = newIsHomeOwner; } } This class's constructor works, but it is difficult for client code to use properly. The Builder pattern can be used to make the constructor easier to use. NetBeans will refactor this for me as I have written about previously. An example of the refactored code is shown next (NetBeans does this by creating all new Builder class). PersonBuilder.java package dustin.examples; public class PersonBuilder { private String newLastName; private String newFirstName; private String newMiddleName; private String newSalutation; private String newSuffix; private String newStreetAddress; private String newCity; private String newState; private boolean newIsFemale; private boolean newIsEmployed; private boolean newIsHomeOwner; public PersonBuilder() { } public PersonBuilder setNewLastName(String newLastName) { this.newLastName = newLastName; return this; } public PersonBuilder setNewFirstName(String newFirstName) { this.newFirstName = newFirstName; return this; } public PersonBuilder setNewMiddleName(String newMiddleName) { this.newMiddleName = newMiddleName; return this; } public PersonBuilder setNewSalutation(String newSalutation) { this.newSalutation = newSalutation; return this; } public PersonBuilder setNewSuffix(String newSuffix) { this.newSuffix = newSuffix; return this; } public PersonBuilder setNewStreetAddress(String newStreetAddress) { this.newStreetAddress = newStreetAddress; return this; } public PersonBuilder setNewCity(String newCity) { this.newCity = newCity; return this; } public PersonBuilder setNewState(String newState) { this.newState = newState; return this; } public PersonBuilder setNewIsFemale(boolean newIsFemale) { this.newIsFemale = newIsFemale; return this; } public PersonBuilder setNewIsEmployed(boolean newIsEmployed) { this.newIsEmployed = newIsEmployed; return this; } public PersonBuilder setNewIsHomeOwner(boolean newIsHomeOwner) { this.newIsHomeOwner = newIsHomeOwner; return this; } public Person createPerson() { return new Person(newLastName, newFirstName, newMiddleName, newSalutation, newSuffix, newStreetAddress, newCity, newState, newIsFemale, newIsEmployed, newIsHomeOwner); } } I prefer to have my Builder as a nested class inside the class whose object it builds, but the NetBeans automatic generation of a standalone Builder is very easy to use. Another difference between the NetBeans-generated Builder and the Builders I like to write is that my preferred Builder implementations have required fields provided in the Builder's constructor rather than provide a no-arguments constructor. The next code listing shows my Person class from above with a Builder added into it as a nested class. Person.java with Nested Person.Builder package dustin.examples; /** * Person class used as part of too many parameters demonstration. * * @author Dustin */ public class Person { private final String lastName; private final String firstName; private final String middleName; private final String salutation; private final String suffix; private final String streetAddress; private final String city; private final String state; private final boolean isFemale; private final boolean isEmployed; private final boolean isHomewOwner; public Person( final String newLastName, final String newFirstName, final String newMiddleName, final String newSalutation, final String newSuffix, final String newStreetAddress, final String newCity, final String newState, final boolean newIsFemale, final boolean newIsEmployed, final boolean newIsHomeOwner) { this.lastName = newLastName; this.firstName = newFirstName; this.middleName = newMiddleName; this.salutation = newSalutation; this.suffix = newSuffix; this.streetAddress = newStreetAddress; this.city = newCity; this.state = newState; this.isFemale = newIsFemale; this.isEmployed = newIsEmployed; this.isHomewOwner = newIsHomeOwner; } public static class PersonBuilder { private String nestedLastName; private String nestedFirstName; private String nestedMiddleName; private String nestedSalutation; private String nestedSuffix; private String nestedStreetAddress; private String nestedCity; private String nestedState; private boolean nestedIsFemale; private boolean nestedIsEmployed; private boolean nestedIsHomeOwner; public PersonBuilder( final String newFirstName, final String newCity, final String newState) { this.nestedFirstName = newFirstName; this.nestedCity = newCity; this.nestedState = newState; } public PersonBuilder lastName(String newLastName) { this.nestedLastName = newLastName; return this; } public PersonBuilder firstName(String newFirstName) { this.nestedFirstName = newFirstName; return this; } public PersonBuilder middleName(String newMiddleName) { this.nestedMiddleName = newMiddleName; return this; } public PersonBuilder salutation(String newSalutation) { this.nestedSalutation = newSalutation; return this; } public PersonBuilder suffix(String newSuffix) { this.nestedSuffix = newSuffix; return this; } public PersonBuilder streetAddress(String newStreetAddress) { this.nestedStreetAddress = newStreetAddress; return this; } public PersonBuilder city(String newCity) { this.nestedCity = newCity; return this; } public PersonBuilder state(String newState) { this.nestedState = newState; return this; } public PersonBuilder isFemale(boolean newIsFemale) { this.nestedIsFemale = newIsFemale; return this; } public PersonBuilder isEmployed(boolean newIsEmployed) { this.nestedIsEmployed = newIsEmployed; return this; } public PersonBuilder isHomeOwner(boolean newIsHomeOwner) { this.nestedIsHomeOwner = newIsHomeOwner; return this; } public Person createPerson() { return new Person( nestedLastName, nestedFirstName, nestedMiddleName, nestedSalutation, nestedSuffix, nestedStreetAddress, nestedCity, nestedState, nestedIsFemale, nestedIsEmployed, nestedIsHomeOwner); } } } The Builder can be even nicer when enhanced through use of custom types and parameters objects as outlined in my first two posts on the "too many parameters" problem. This is shown in the next code listing. Person.java with Nested Builder, Custom Types, and Parameters Object package dustin.examples; /** * Person class used as part of too many parameters demonstration. * * @author Dustin */ public class Person { private final FullName name; private final Address address; private final Gender gender; private final EmploymentStatus employment; private final HomeownerStatus homeOwnerStatus; /** * Parameterized constructor can be private because only my internal builder * needs to call me to provide an instance to clients. * * @param newName Name of this person. * @param newAddress Address of this person. * @param newGender Gender of this person. * @param newEmployment Employment status of this person. * @param newHomeOwner Home ownership status of this person. */ private Person( final FullName newName, final Address newAddress, final Gender newGender, final EmploymentStatus newEmployment, final HomeownerStatus newHomeOwner) { this.name = newName; this.address = newAddress; this.gender = newGender; this.employment = newEmployment; this.homeOwnerStatus = newHomeOwner; } public FullName getName() { return this.name; } public Address getAddress() { return this.address; } public Gender getGender() { return this.gender; } public EmploymentStatus getEmployment() { return this.employment; } public HomeownerStatus getHomeOwnerStatus() { return this.homeOwnerStatus; } /** * Builder class as outlined in the Second Edition of Joshua Bloch's * Effective Java that is used to build a {@link Person} instance. */ public static class PersonBuilder { private FullName nestedName; private Address nestedAddress; private Gender nestedGender; private EmploymentStatus nestedEmploymentStatus; private HomeownerStatus nestedHomeOwnerStatus; public PersonBuilder( final FullName newFullName, final Address newAddress) { this.nestedName = newFullName; this.nestedAddress = newAddress; } public PersonBuilder name(final FullName newName) { this.nestedName = newName; return this; } public PersonBuilder address(final Address newAddress) { this.nestedAddress = newAddress; return this; } public PersonBuilder gender(final Gender newGender) { this.nestedGender = newGender; return this; } public PersonBuilder employment(final EmploymentStatus newEmploymentStatus) { this.nestedEmploymentStatus = newEmploymentStatus; return this; } public PersonBuilder homeOwner(final HomeownerStatus newHomeOwnerStatus) { this.nestedHomeOwnerStatus = newHomeOwnerStatus; return this; } public Person createPerson() { return new Person( nestedName, nestedAddress, nestedGender, nestedEmploymentStatus, nestedHomeOwnerStatus); } } } The last couple of code listings show how a Builder is typically used - to construct an object. Indeed, the item on the builder (Item #2) in Joshua Bloch's Second Edition of Effective Java is in the chapter on creating (and destroying) object. However, the builder can help indirectly with non-constructor methods by allowing an easier way to build parameters objects that are passed to methods. For example, in the last code listing, the methods have some parameters objects (FullName and Address) passed to them. It can be tedious for clients to have to construct these parameters objects and the builder can be used to make that process less tedious. So, although the builder is used for construction in each case, it indirectly benefits non-constructor methods by allowing for easier use of the parameters objects that reduce a method's argument count. The new definitions of the FullName and Address classes to be used as parameters objects and using the Builder themselves are shown next. FullName.java with Builder package dustin.examples; /** * Full name of a person. * * @author Dustin */ public final class FullName { private final Name lastName; private final Name firstName; private final Name middleName; private final Salutation salutation; private final Suffix suffix; private FullName( final Name newLastName, final Name newFirstName, final Name newMiddleName, final Salutation newSalutation, final Suffix newSuffix) { this.lastName = newLastName; this.firstName = newFirstName; this.middleName = newMiddleName; this.salutation = newSalutation; this.suffix = newSuffix; } public Name getLastName() { return this.lastName; } public Name getFirstName() { return this.firstName; } public Name getMiddleName() { return this.middleName; } public Salutation getSalutation() { return this.salutation; } public Suffix getSuffix() { return this.suffix; } @Override public String toString() { return this.salutation + " " + this.firstName + " " + this.middleName + this.lastName + ", " + this.suffix; } public static class FullNameBuilder { private final Name nestedLastName; private final Name nestedFirstName; private Name nestedMiddleName; private Salutation nestedSalutation; private Suffix nestedSuffix; public FullNameBuilder( final Name newLastName, final Name newFirstName) { this.nestedLastName = newLastName; this.nestedFirstName = newFirstName; } public FullNameBuilder middleName(final Name newMiddleName) { this.nestedMiddleName = newMiddleName; return this; } public FullNameBuilder salutation(final Salutation newSalutation) { this.nestedSalutation = newSalutation; return this; } public FullNameBuilder suffix(final Suffix newSuffix) { this.nestedSuffix = newSuffix; return this; } public FullName createFullName() { return new FullName( nestedLastName, nestedFirstName, nestedMiddleName, nestedSalutation, nestedSuffix); } } } Address.java with Builder package dustin.examples; /** * Representation of a United States address. * * @author Dustin */ public final class Address { private final StreetAddress streetAddress; private final City city; private final State state; private Address(final StreetAddress newStreetAddress, final City newCity, final State newState) { this.streetAddress = newStreetAddress; this.city = newCity; this.state = newState; } public StreetAddress getStreetAddress() { return this.streetAddress; } public City getCity() { return this.city; } public State getState() { return this.state; } @Override public String toString() { return this.streetAddress + ", " + this.city + ", " + this.state; } public static class AddressBuilder { private StreetAddress nestedStreetAddress; private final City nestedCity; private final State nestedState; public AddressBuilder(final City newCity, final State newState) { this.nestedCity = newCity; this.nestedState = newState; } public AddressBuilder streetAddress(final StreetAddress newStreetAddress) { this.nestedStreetAddress = newStreetAddress; return this; } public Address createAddress() { return new Address(nestedStreetAddress, nestedCity, nestedState); } } } With the above builders included in the classes, a Person instance can be created as shown in the next code listing. A more traditional instantiation of a Person instance is shown after that for comparison. Two Examples of Client Code Instantiating a Person with Builders final Person person1 = new Person.PersonBuilder( new FullName.FullNameBuilder( new Name("Dynamite"), new Name("Napoleon")).createFullName(), new Address.AddressBuilder( new City("Preston"), State.ID).createAddress()).createPerson(); final Person person2 = new Person.PersonBuilder( new FullName.FullNameBuilder( new Name("Coltrane"), new Name("Rosco")).middleName(new Name("Purvis")).createFullName(), new Address.AddressBuilder( new City("Hazzard"), State.GA).createAddress()) .gender(Gender.MALE).employment(EmploymentStatus.EMPLOYED).createPerson(); Instantiating a Person Without a Builder final person = new Person("Coltrane", "Rosco", "Purvis", null, "Hazzard", "Georgia", false, true, true); As the previous code snippets show, the client code for calling a traditional Java constructor is far less readable and far easier to mess up than use of the builder classes. The variety of the same types (strings and booleans) and the necessity to place nulls in the constructor call for optional attributes make provide many ways for this approach to end badly. Benefits and Advantages There is a considerable cost to the Builder pattern in that one must essentially double the number of lines of code each attribute and for setting those attributes. This price pays off, however, when the client code benefits greatly in terms of usability and readability. The parameters to the constructor are reduced and are provided in highly readable method calls. Another advantage of the Builder approach is the ability to acquire an object in a single statement and state without the object in multiple states problem presented by using "set" methods. I am increasingly appreciating the value of immutability in a multi-core world and the Builder pattern is perfectly suited for an immutable class when that class features a large number of attributes. I also like that there is no need to pass in null for optional parameters to the constructor. The Builder pattern not only makes the code more readable, but makes it even easier to apply an IDE's code completion feature. Further benefits of the Builder pattern when used with constructors are outlined in Item #2 of the Second Edition of Effective Java. Costs and Disadvantages As shown and mentioned above, the number of lines of code of a given class must be essentially doubled for "set" methods with the builder approach. Furthermore, although client code is more readable, the client code is also more verbose. I consider the benefit of greater readability worth the cost as the number of arguments increase or as more arguments share the same type or as the number of optional arguments increase. More lines of code in the class with the builder sometimes mean that developers may forget to add support for a new attribute to the builder when they add that attribute to the main class. To try to help with this, I like to nest my builders inside the class that they build so that it's more obvious to the developer that there is a relevant builder that needs to be similarly updated. Although there is still risk of the developer forgetting to add support for a new attribute to the builder, this is really no different than the risk of forgetting to add a new attribute to a class's toString(), equals(Object), hashCode() or other methods often based on all attributes of a class. In my implementation of the Builder, I made the client pass required attributes into the builder's constructor rather than via "set" methods. The advantage of this is that the object is always instantiated in a "complete" state rather than sitting in an incomplete state until the developer calls (if ever calls) the appropriate "set" method to set additional fields. This is necessary to enjoy the benefits of immutability. However, a minor disadvantage of that approach is that I don't get the readability advantages of methods named for the field I am setting. The Builder, as its name suggests, is really only an alternative to constructors and not directly used to reduce the number of non-constructor method parameters. However, the builder can be used in conjunction with parameters objects to reduce the number of non-constructor method arguments. Further arguments against use of the Builder for object construction can be found in a comment on the A dive into the Builder pattern post. Conclusion I really like the Builder pattern for constructing objects when I have a lot of parameters, especially if many of these parameters are null and when many of them share the same data type. A developer might feel that the extra code to implement a Builder might not justify its benefits for a small number of parameters, especially if the few parameters are required and of different types. In such cases, it might be considered desirable to use traditional constructors or, if immutability is not desired, use a no-argument constructor and require the client to know to call the necessary "set" methods.

simple talk - How to get SQL Railroad Diagrams from MSDN BNF syntax notation. On SQL Server Books-On-Line, in the Transact-SQL Reference (database Engine), every SQL Statement has its syntax represented in ‘Backus–Naur Form’ notation (BNF) syntax. For a programmer in a hurry, this should be ideal because It is the only quick way to understand and appreciate all the permutations of the syntax. It is a great feature once you get your eye in. It isn’t the only way to get the information; You can, of course, reverse-engineer an understanding of the syntax from the examples, but your understanding won’t be complete, and you’ll have wasted time doing it. BNF is a good start in representing the syntax: Oracle and SQLite go one step further, and have proper railroad diagrams for their syntax, which is a far more accessible way of doing it. There are three problems with the BNF on MSDN. Firstly, it is isn’t a standard version of BNF, but an ancient fork from EBNF, inherited from Sybase. Secondly, it is excruciatingly difficult to understand, and thirdly it has a number of syntactic and semantic errors. The page describing DML triggers, for example, currently has the absurd BNF error that makes it state that all statements in the body of the trigger must be separated by commas. There are a few other detail problems too. Here is the offending syntax for a DML trigger, pasted from MSDN. ... I’ve been trying to create railroad diagrams for all the important SQL Server SQL statements, as good as you’d find for Oracle, and have so far published the CREATE TABLE and ALTER TABLE railroad diagrams based on the BNF. Although I’ve been aware of them, I’ve never realised until recently how many errors there are. Then, Colin Daley created a translator for the SQL Server dialect of BNF which outputs standard EBNF notation used by the W3C. The example MSDN BNF for the trigger would be rendered as … ... Colin’s intention was to allow anyone to paste SQL Server’s BNF notation into his website-based parser, and from this generate classic railroad diagrams via Gunther Rademacher's Railroad Diagram Generator. Colin's application does this for you: you're not aware that you are moving to a different site. Because Colin's 'translator' it is a parser, it will pick up syntax errors. Once you’ve fixed the syntax errors, you will get the syntax in the form of a human-readable railroad diagram and, in this form, the semantic mistakes become flamingly obvious. Gunter’s Railroad Diagram Generator is brilliant. To be able, after correcting the MSDN dialect of BNF, to generate a standard EBNF, and from thence to create railroad diagrams for SQL Server’s syntax that are as good as Oracle’s, is a great boon, and many thanks to Colin for the idea. Here is the result of the W3C EBNF from Colin’s application then being run through the Railroad diagram generator. Now that’s much better, you’ll agree. This is pretty easy to understand, and at this point any error is immediately obvious. This should be seriously useful, and it is to me. However there is that snag. The BNF is generally incorrect, and you can’t expect the average visitor to mess about with it. The answer is, of course, to correct the BNF on MSDN and maybe even add railroad diagrams for the syntax. Stop giggling! I agree it won’t happen. In the meantime, we need to collaboratively store and publish these corrected syntaxes ourselves as we do them. How? GitHub? SQL Server Central? Simple-Talk? What should those of us who use the system do with our corrected EBNF so that anyone can use them without hassle? Grammar Translator If you are familiar with the Grammar Translator, go ahead and create railroad diagrams from the Transact-SQL Reference. Otherwise, please see the FAQ. In particular, be sure to try thetutorial. Welcome to Railroad Diagram Generator! This is a tool for creating syntax diagrams, also known as railroad diagrams, from context-free grammars specified in EBNF. Syntax diagrams have been used for decades now, so the concept is well-known, and some tools for diagram generation are in existence. The features of this one are usage of the W3C's EBNF notation, web-scraping of grammars from W3C specifications, online editing of grammars, diagram presentation in SVG, and it was completely written in web languages (XQuery, XHTML, CSS, JavaScript). There's nothing like a diagram to help grok something (and the MSDN BNF SQL stuff really makes my brain hurt...)

I’ve been doing some work that involves reading in CSV files, for which I’ve been using OpenCSV, and my initial approach was to read through the file line by line, parse the contents, and save it into a list of maps. This works when the contents of the file fit into memory, but is problematic for larger files where I needed to stream the file and process each line individually, rather than all of them after the file was loaded. I initially wrote a variation on totallylazy’s Strings#lines to do this, and while I was able to stream the file, I made a mistake somewhere which meant the number of maps on the heap was always increasing. After spending a few hours trying to fix this, Michael suggested that it’d be easier to use an iterator instead, and I ended up with the following code: public class ParseCSVFile { public static void main(String[] args) throws IOException { final CSVReader csvReader = new CSVReader( new BufferedReader( new FileReader( "/path/to/file.csv" ) ), '\t' ); final String[] fields = csvReader.readNext(); Iterator>() lazilyLoadedFile = return new Iterator>() { String[] data = csvReader.readNext(); @Override public boolean hasNext() { return data != null; } @Override public Map next() { final Map properties = new HashMap(); for ( int i = 0; i < data.length; i++ ) { properties.put(fields[i], data[i]); } try { data = csvReader.readNext(); } catch ( IOException e ) { data = null; } return properties; } @Override public void remove() { throw new UnsupportedOperationException(); } }; } } Although this code works, it’s not the most readable function I’ve ever written, so any suggestions on how to do this in a cleaner way are welcome.

With these changes, we can access the REST API equally well fromhttp://:9999 and https://:9998.

I have recently started the Coursera Functional Programming with Scala course (taught by Martin Odersky - the creator of Scala) - which is actually serving as an introduction to both FP and Scala at the same time having done neither before. The course itself is great, however, trying to watch the videos and take in both the new Scala syntax and the FP concepts at the same time can take a bit of effort. I wanted to work through some of the core FP concepts in a more familiar context, so am going to apply some of the lessons/principles/exercises in Groovy. Functions: If you have done any Groovy programming then you will have come across Groovy functions/closures. As Groovy is dynamically typed (compared to Scala's static typing), you can play it fairly fast and loose. For example, if we take the square root function that is demonstrated in the Scala course, it is defined as follows: def sqrt(x: Double): Double = ... As you can see, Scala expects the values to be typed (aside, you don't actually always need to provide a return type in Scala). But in the Groovy function it is: def sqrt = {x-> ... } A groovy function can be defined and assigned to any variable, thereby allowing it to be passed around as a first class object. If we look at the complete solution for calculating the square root of a number (using Newton's method - To compute the square root of "x" we start with an estimate of the square root, "y" and continue to improve the the guess by taking the mean of x and x/y ) def sqrtIter(guess: Double, x: Double): Double = if (isGoodEnough(guess, x)) guess else sqrtIter(improve(guess, x), x) def improve(guess: Double, x: Double) = (guess + x / guess) / 2 def isGoodEnough(guess: Double, x: Double) = abs(guess * guess - x) < 0.001 def sqrt(x: Double) = srqtIter(1.0, x) So that's in Scala, if we try Groovy we will see we can achieve pretty much the same thing easily: //Improve the guess using Newton's method def improve = { guess, x -> (guess + x / guess) / 2 } //Check if our guess is good enough, within a chosen threshold def isGoodEnough = {guess, x -> abs(guess * guess - x) < 0.001 } //iterate over guesses until our guess is good enough def sqrtIter = { guess, x -> if (isGoodEnough(guess, x)) guess else sqrtIter(improve(guess, x), x) } //wrap everythin in the square root function call def sqrt = {x -> srqtIter(1.0, x)} Recursion (and tail-recursion): As FP avoids having mutable state, the most common approach to solve problems is to break the problem down in to simple functions and call them recursively - This avoids having to maintain state whilst iterating through loops, and each function call is given its input and produces an output. If we again consider an example from the Scala course, with the example of a simple function that calculates the factorial for a given number. def factorial ={ n -> if (n == 0) 1 else n * factorial(n - 1) } factorial(4) This simple function recursively calculates the factorial, continuing to call itself until all numbers to zero have been considered. As you can see, there is no mutable state - every call to the factorial function simply takes the input and returns an output (the value of n is never changed or re-assigned, n is simply used to calculate output values) There is a problem here, and that is as soon as you attempt to calculate the factorial of a significantly large enough number you will encounter a StackOverflow exception - this is because in the JVM every time a function is called, a frame is added to the stack, so working recursively its pretty easy to hit upon the limit of the stack and encounter this problem. The common way to solve this is by using Tail-Call recursion. This trick is simply to have the last code that is evaluated in the function to be the recursive call - normally in FP languages the compiler/interpreter will recognise this pattern and under the hood, it will really just run the code as a loop (e.g. if we know the very last piece of code in the block of code is calling itself, its really not that different to just having the block of code/function inside a loop construct) In the previous factorial example, it might look like the last code to be executed is the recursive callfactorial(n-1) - however, the value of that call is actually returned to the function and THENmultiplied by n - so actually the last piece of code to be evaluated in the function call is actually n * return value of factorial(n-1). Let's have a look at re-writing the function so it is tail-recursive. def factorial ={ n, accumulator=1 -> if (n == 1) accumulator else factorial(n-1, n*accumulator) } factorial(4) Now, using an accumulator, the last code to be evaluated in the function is our recursive function call. In most FP languages, including Scala, this is enough - however, the JVM doesn't automatically support tail-call recursion, so you actually need to use a rather clunkier approach in Groovy: def factorial ={ n, accumulator=1 -> if (n == 1) accumulator else factorial.trampoline(n-1, n*accumulator) }.trampoline() factorial(4) The use of the trampoline() method means that the function will now be called using tail-call recursion, so there should never be a StackOverflow exception. It's not as nice as in Scala or other languages, but the support is there so we can continue. Currying: This is like function composition - the idea being you take a generic function, and then you curry it with some value to make a more specific application of the function. For example, if we look at a function that given values x and y, it returns z which is the value x percent of y (e.g. given x=10, y=100, it returns the 10 percent of 100, z=10) def percentage = { percentage, x -> x/100 * percentage } The above simple function is a generic mechanism to get a percentage value of another, but if we consider that we wanted a common application of this function was to always calculate 10% of a given value - rather than write a slightly modified version of the function we can simply curry the function as follows: def tenPercent = percentage.curry(10) Now, if the function tenPercent(x) is called, it uses the original percentage() function, but curries the value 10 as the first argument. (If you need to curry other argument positions you can also use the rcurry() function to curry the right most argument, or ncurry() which also takes an argument position - check the Groovy docs on currying for more info) Immutability: Immutability is partially supported in Java normally with use of the final keyword (meaning variables can't be changed after being initially set on object instantiation). Groovy also provides a quick and easy @Immutable annotation that can be added to a class to easily make it immutable. But really, there is more to avoiding immutable state than just having classes as immutable - As we have functions as first class objects, we can easily assign variables and mutate them within a function - so this is more of a mindset or philosophy that you have to get used to. For example: def list = ['groovy', 'functional'] //This mutates the original list list.add('programming') //This creates a new list leaving the original unchanged def newList = list.plus(2, 'programming') The first example is probably more like the Groovy/Java code we are used to writing, but that is mutating the state of the list - where as the second approach leaves the original list unchanged. Map Reduce: As a final note, there are some functions in FP that are pretty common techniques - the most famous of which these days (in part thanks to Google) is Map-Reduce, but the trio of functions are actually Map, Reduce(also known as Fold) & Filter - you can read more about the functions here (or just google them!), but these functions actually correlate pretty nicely to core Groovy functions that you probably use a lot of (assuming you are groovy programmers). Map map is the easiest to understand of the three. It takes in two inputs - a function, and a list. It then applies this function to every element in the list. You can basically do the same thing with a list comprehension however. Sound familiar? This is basically the .collect{} function in Groovy Reduce/Fold fold takes in a function and folds it in between the elements of a list. It's a bit hard to understand at first This one is a bit more complicated to descibe, but is the same as the .inject{} function in groovy Filter filter is easy. It takes in a 'test' and a list, and it chucks out any elements of the list which don't satisfy that test. And another simple one - filtering out a list for desired elements, this is Groovy's .findAll{} function As I said at the start, I am new to FP and coming from an OO background, but hopefully the above isn't too far from the truth! As I get further through the Coursera course I will try to post again, maybe with some of the assignments attempted in Groovy to see how it really stands up. Some useful references: Groovy docs on FP: http://groovy.codehaus.org/Functional+Programming+with+Groovy Coursera Scala course: https://www.coursera.org/course/progfun

Recently I needed to quickly create some temporary JSON objects from some preexisting data. So, I turned to Json.NET, a very popular JSON framework for .NET. Since I got my data on the fly (think parsing some other data structures and you have lots of anonymous results), I didn’t want to create classes for such instances. Luckily, Json.NET supports serializing anonymous objects. First add the library via NuGet – it is called Newtonsoft.Json. For example: var obj = new { user = new { name = "John", age = 21, data = new[] { 1, 2, 3, 4 } } }; var result = JsonConvert.SerializeObject(obj, Formatting.Indented); If we did that in a console application, we could print out the result. The printout is: { "user": { "name": "John", "age": 21, "data": [ 1, 2, 3, 4 ] } } The code above works on Windows 8, Windows Phone 7 and 8. This way it is dead simple to create JSON representation of your anonymous data. Think REST services.

this post comes from ernie souhrada at the mysql performance blog. note: this is part 1 of what will be a two-part series on the performance implications of using in-flight data encryption. some of you may recall my security webinar from back in mid-august; one of the follow-up questions that i was asked was about the performance impact of enabling ssl connections. my answer was 25%, based on some 2011 data that i had seen over on yassl’s website, but i included the caveat that it is workload-dependent, because the most expensive part of using ssl is establishing the connection. not long thereafter, i received a request to conduct some more specific benchmarks surrounding ssl usage in mysql, and today i’m going to show the results. first, the testing environment. all tests were performed on an intel core i7-2600k 3.4ghz cpu (8 cores, ht included) with 32gb of ram and centos 6.4. the disk subsystem is a 2-disk raid-0 of samsung 830 ssds, although since we’re only concerned with measuring the overhead added by using ssl connections, we’ll only be conducting read-only tests with a dataset that fits completely in the buffer pool. the version of mysql used for this experiment is community edition 5.6.13, and the testing tools are sysbench 0.5 and perl. we conduct two tests, each one designed to simulate one of the most common mysql usage patterns. first, we examine connection pooling, often seen in the java world, where some small set of connections are established by, for example, the servlet container and then just passed around to the application as needed, and one-request-per-connection, typical in the lamp world, where the script that displays a given page might connect to the database, run a couple of queries, and then disconnect. test 1: connection pool for the first test, i ran sysbench in read-only mode at concurrency levels of 1, 2, 4, 8, 16, and 32 threads, first with no encryption and then with ssl enabled and key lengths of 1024, 2048, and 4096 bits. 8 sysbench tables were prepared, each containing 100,000 rows, resulting in a total data size of approximately 256mb. the size of my innodb buffer pool was 4gb, and before conducting each official measurement run, i ran a warm-up run to prime the buffer pool. each official test run lasted 10 minutes; this might seem short, but unlike, say, a pcie flash storage device, i would not expect the variable under observation to really change that much over time or need time to stabilize. the basic sysbench syntax used is shown below. #!/bin/bash for ssl in on off ; do for threads in 1 2 4 8 16 32 ; do sysbench --test=/usr/share/sysbench/oltp.lua --mysql-user=msandbox$ssl --mysql-password=msandbox \ --mysql-host=127.0.0.1 --mysql-port=5613 --mysql-db=sbtest --mysql-ssl=$ssl \ --oltp-tables-count=8 --num-threads=$threads --oltp-dist-type=uniform --oltp-read-only=on \ --report-interval=10 --max-time=600 --max-requests=0 run > sb-ssl_${ssl}-threads-${threads}.out done done if you’re not familiar with sysbench, the important thing to know about it for our purposes is that it does not connect and disconnect after each query or after each transaction. it establishes n connections to the database (where n is the number of threads) and runs queries though them until the test is over. this behavior provides our connection-pool simulation. the assumption, given what we know about where ssl is the slowest, is that the performance penalty here should be the lowest. first, let’s look at raw throughput, measured in queries per second: the average throughput and standard deviation (both measured in queries per second) for each test configuration is shown below in tabular format: # of threads ssl key size 1 2 4 8 16 32 ssl off 9250.18 (1005.82) 18297.61 (689.22) 33910.31 (446.02) 50077.60 (1525.37) 49844.49 (934.86) 49651.09 (498.68) 1024-bit 2406.53 (288.53) 4650.56 (558.58) 9183.33 (1565.41) 26007.11 (345.79) 25959.61 (343.55) 25913.69 (192.90) 2048-bit 2448.43 (290.02) 4641.61 (510.91) 8951.67 (1043.99) 26143.25 (360.84) 25872.10 (324.48) 25764.48 (370.33) 4096-bit 2427.95 (289.00) 4641.32 (547.57) 8991.37 (1005.89) 26058.09 (432.86) 25990.13 (439.53) 26041.27 (780.71) so, given that this is an 8-core machine and io isn’t a factor, we would expect throughput to max out at 8 threads, so the levelling-off of performance is expected. what we also see is that it doesn’t seem to make much difference what key length is used, which is also largely expected. however, i definitely didn’t think the encryption overhead would be so high. the next graph here is 95th-percentile latency from the same test: and in tabular format, the raw numbers (average and standard deviation): # of threads ssl key size 1 2 4 8 16 32 ssl off 1.882 (0.522) 1.728 (0.167) 1.764 (0.145) 2.459 (0.523) 6.616 (0.251) 27.307 (0.817) 1024-bit 6.151 (0.241) 6.442 (0.180) 6.677 (0.289) 4.535 (0.507) 11.481 (1.403) 37.152 (0.393) 2048-bit 6.083 (0.277) 6.510 (0.081) 6.693 (0.043) 4.498 (0.503) 11.222 (1.502) 37.387 (0.393) 4096-bit 6.120 (0.268) 6.454 (0.119) 6.690 (0.043) 4.571 (0.727) 11.194 (1.395) 37.26 (0.307) with the exception of 8 and 32 threads, the latency introduced by the use of ssl is constant at right around 5ms, regardless of the key length or the number of threads. i’m not surprised that there’s a large jump in latency at 32 threads, but i don’t have an immediate explanation for the improvement in the ssl latency numbers at 8 threads. test 2: connection time for the second test, i wrote a simple perl script to just connect and disconnect from the database as fast as possible. we know that it’s the connection setup which is the slowest part of ssl, and the previous test already shows us roughly what we can expect for ssl encryption overhead for sending data once the connection has been established, so let’s see just how much overhead ssl adds to connection time. the basic script to do this is quite simple (non-ssl version shown): #!/usr/bin/perl use dbi; use time::hires qw(time); $start = time; for (my $i=0; $i<100; $i++) { my $dbh = dbi->connect("dbi:mysql:host=127.0.0.1;port=5613", "msandbox","msandbox",undef); $dbh->disconnect; undef $dbh; } printf "%.6f\n", time - $start; as with test #1, i ran test #2 with no encryption and ssl encryption of 1024, 2048, and 4098 bits, and i conducted 10 trials of each configuration. then i took the elapsed time for each test and converted it to connections per second. the graph below shows the results from each run: here are the averages and standard deviations: encryption average connections per second standard deviation none 2701.75 165.54 1024-bit 77.04 6.14 2048-bit 28.183 1.713 4096-bit 5.45 0.015 yes, that’s right, 4096-bit ssl connections are 3 orders of magnitude slower to establish than unencrypted connections. really, the connection overhead for any level of ssl usage is quite high when compared to the unencrypted test, and it’s certainly much higher than my original quoted number of 25%. analysis and parting thoughts so, what do we take away from this? the first thing is, of course, is that ssl overhead is a lot higher than 25%, particularly if your application uses anything close to the one-connection-per-request pattern. for a system which establishes and maintains long-running connections, the initial connection overhead becomes a non-factor, regardless of the encryption strength, but there’s still a rather large performance penalty compared to the unencrypted connection. this leads directly into the second point, which is that connection pooling is by far a more efficient method of using ssl if your application can support it. but what if connection pooling isn’t an option, mysql’s ssl performance is insufficient, and you still need full encryption of data in-flight? run the encryption component of your system at a lower layer – a vpn with hardware crypto would be the fastest approach, but even something as simple as an ssh tunnel or openvpn *might* be faster than ssl within mysql. i’ll be exploring some of these solutions in a follow-up post. and finally… when in doubt, run your own benchmarks. i don’t have an explanation for why the yassl numbers are so different from these (maybe yassl is a faster ssl library than openssl, or maybe they used a different cipher – if you’re curious, the original 25% number came from slides 56-58 of this presentation ), but in any event, this does illustrate why it’s important to run tests on your own hardware and with your own workload when you’re interested in finding out how well something will perform rather than taking someone else’s word for it.

The following question will again test your knowledge of the Oracle Weblogic threading model. I’m looking forward for your comments and experience on the same. If you are a Weblogic administrator, I’m certain that you heard of this common problem: stuck threads. This is one of the most common problems you will face when supporting a Weblogic production environment. A Weblogic stuck thread simply means a thread performing the same request for a very long time and more than the configurable Stuck Thread Max Time. Question: How can you detect the presence of STUCK threads during and following a production incident? Answer: As we saw from our last article “Weblogic Thread Monitoring Tips”, Weblogic provides functionalities allowing us to closely monitor its internal self-tuning thread pool. It will also highlight you the presence of any stuck thread. This monitoring view is very useful when you do a live analysis but what about after a production incident? The good news is that Oracle Weblogic will also log any detected stuck thread to the server log. Such information includes details on the request and more importantly, the thread stack trace. This data is crucial and will allow you to potentially better understand the root cause of any slowdown condition that occurred at a certain time. < ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)'> <[STUCK] ExecuteThread: '35' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "608" seconds working on the request "Workmanager: default, Version: 0, Scheduled=true, Started=true, Started time: 608213 ms POST /App1/jsp/test.jsp HTTP/1.1 Accept: application/x-ms-application... Referer: http://.. Accept-Language: en-US User-Agent: Mozilla/4.0 .. Content-Type: application/x-www-form-urlencoded Accept-Encoding: gzip, deflate Content-Length: 539 Connection: Keep-Alive Cache-Control: no-cache Cookie: JSESSIONID= ]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace: ................................... javax.servlet.http.HttpServlet.service(HttpServlet.java:727) javax.servlet.http.HttpServlet.service(HttpServlet.java:820) weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227) weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125) weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:301) weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:184) weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.... weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run() weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321) weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120) weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2281) weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2180) weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1491) weblogic.work.ExecuteThread.execute(ExecuteThread.java:256) weblogic.work.ExecuteThread.run(ExecuteThread.java:221) Here is one more tip: the generation and analysis of a JVM thread dump will also highlight you stuck threads. As we can see from the snapshot below, the Weblogic thread state is now updated to STUCK, which means that this particular request is being executed since at least 600 seconds or 10 minutes. This is very useful information since the native thread state will typically remain to RUNNABLE. The native thread state will only get updated when dealing with BLOCKED threads etc. You have to keep in mind that RUNNABLE simply means that this thread is healthy from a JVM perspective. However, it does not mean that it truly is from a middleware or Java EE container perspective. This is why Oracle Weblogic has its own internal ExecuteThread state. Finally, if your organization or client is using any commercial monitoring tool, I recommend that you enable some alerting around both hogging thread and stuck thread. This will allow your support team to take some pro-active actions before the affected Weblogic managed server(s) become fully unresponsive.

I’m going to start by discussing the Spring WebMVC configuration and move on from there in future posts.

When putting together data sets to play around with, one of the more boring tasks is stripping out characters that you’re not interested in and more often than not those characters are white spaces. Since I’ve been building data sets using Clojure I wanted to write a function that would do this for me. I started out with the following string: (def word " with a little bit of space we can make it through the night ") which I wanted to format in such a way that there would be a maximum of one space between each word. I start out by using the trim function but that only removes white space from the beginning and end of a string: > (clojure.string/trim word) "with a little bit of space we can make it through the night" I wanted to get rid of the space in between ‘a’ and ‘little’ as well so I wrote the following code to split on a space and filter out any excess spaces that still remained before joining the words back together: > (clojure.string/join " " (filter #(not (clojure.string/blank? %)) (clojure.string/split word #" "))) "with a little bit of space we can make it through the night" I wanted to try and make it a bit easier to read by using the thread last (->>) macro but that didn’t work as well as I’d hoped because clojure.string/split doesn’t take the string in as its last parameter: > (->> (clojure.string/split word #" ") (filter #(not (clojure.string/blank? %))) (clojure.string/join " ")) "with a little bit of space we can make it through the night" I worked around it by creating a specific function for splitting on a space: (defn split-on-space [word] (clojure.string/split word #"\s")) which means we can now chain everything together nicely: > (->> word split-on-space (filter #(not (clojure.string/blank? %))) (clojure.string/join " ")) "with a little bit of space we can make it through the night" I couldn’t find a cleaner way to do this but I’m sure there is one and my googling just isn’t up to scratch so do let me know in the comments!

In the previous post, we have seen an example where dataProvider attribute has been used3 to test methods with different sets of input data for the same test method. TestNG provides another attribute dataProviderClass in conjunction with dataProvider to fetch the input data for the test methods from an external class. The actual class that holds input data is set to the dataProviderClass attribute and datProvider by itself holds the method name where the input data is actually fetched. Here is a quick example to show how to use dataProviderClass and dataProvide attribute Code Service Class ? view source print? 01.package com.skilledmonster.example; 02./** 03.* Simple calculator service to demonstrate TestNG Framework 04.* 05.* @author Jagadeesh Motamarri 06.* @version 1.0 07.*/ 08.public interface CalculatorService { 09.int sum(int a, int b); 10.int multiply(int a, int b); 11.int div(int a, int b); 12.int sub(int a, int b); 13.} Service Implementation Class ? view source print? 01.package com.skilledmonster.example; 02./** 03.* Simple calculator service implementation to demonstrate TestNG Framework 04.* 05.* @author Jagadeesh Motamarri 06.* @version 1.0 07.*/ 08.public class SimpleCalculator implements CalculatorService { 09.public int sum(int a, int b) { 10.return a + b; 11.} 12.public int multiply(int a, int b) { 13.return a * b; 14.} 15.public int div(int a, int b) { 16.return a / b; 17.} 18.public int sub(int a, int b) { 19.return a - b; 20.} 21.} Data Provider Class ? view source print? 01.package com.skilledmonster.common; 02.import org.testng.annotations.DataProvider; 03./** 04.* Data Provider class for TestNG test cases 05.* 06.* @author Jagadeesh Motamarri 07.* @version 1.0 08.*/ 09.public class TestNGDataProvider { 10./** 11.* Data Provider for testing sum of 2 numbers 12.* 13.* @return 14.*/ 15.@DataProvider 16.public static Object[][] testSumInput() { 17.return new Object[][] { { 5, 5 }, { 10, 10 }, { 20, 20 } }; 18.} 19./** 20.* Data Provider for testing multiplication of 2 numbers 21.* 22.* @return 23.*/ 24.@DataProvider 25.public static Object[][] testMultipleInput() { 26.return new Object[][] { { 5, 5 }, { 10, 10 }, { 20, 20 } }; 27.} 28.} Finally, test class that uses dataProviderClass attribute to feed the input data for the test methods ? package com.skilledmonster.example; import org.testng.Assert; import org.testng.annotations.BeforeClass; import org.testng.annotations.Test; import com.skilledmonster.common.TestNGDataProvider; /** * Example to demonstrate use of dataProviderClass and dataProvide attributes of TestNG framework * * @author Jagadeesh Motamarri * @version 1.0 */ public class TestNGAnnotationTestDataProviderExample { public CalculatorService service; @BeforeClass public void init() { System.out.println("@BeforeClass: The annotated method will be run before the first test method in the current class is invoked."); System.out.println("init service"); service = new SimpleCalculator(); } @Test(dataProviderClass = TestNGDataProvider.class, dataProvider = "testSumInput") public void testSum(int a, int b) { System.out.println("@Test : testSum()"); int result = service.sum(a, b); Assert.assertEquals(result, a + b); } @Test(dataProviderClass = TestNGDataProvider.class, dataProvider = "testMultipleInput") public void testMultiple(int a, int b) { System.out.println("@Test : testMultiple()"); int result = service.multiply(a, b); Assert.assertEquals(result, a * b); } } Output As shown in the above console output, each of the testSum() and testMutiple() methods are invoked with different sets of input data using an external class with dataProviderClass attribute. Advantage More flexibility and re-usability of commonly used data across several test classes. Download Download TestNG DataProvider Example

Here are some free HTML WYSIWYG editors based on the Mozilla Gecko Engine.

I wanted to do some date manipulation in Clojure recently and figured that since clj-time is a wrapper around Joda Time it’d probably do the trick. The first thing we need to do is add the dependency to our project file and then run lein reps to pull down the appropriate JARs. The project file should look something like this: project.clj (defproject ranking-algorithms "0.1.0-SNAPSHOT" :license {:name "Eclipse Public License" :url "http://www.eclipse.org/legal/epl-v10.html"} :dependencies [[org.clojure/clojure "1.4.0"] [clj-time "0.6.0"]]) Now let’s load the clj-time.format namespace into the REPL since we know we’ll be parsing dates: > (require '(clj-time [format :as f])) The string that I want to convert into a date looks like this: (def string-date "18 September 2012") The first thing we should do is check whether there is an existing formatter that we can use by evaluating the following function: > (f/show-formatters) ... :hour-minute 06:45 :hour-minute-second 06:45:22 :hour-minute-second-fraction 06:45:22.473 :hour-minute-second-ms 06:45:22.473 :mysql 2013-09-20 06:45:22 :ordinal-date 2013-263 :ordinal-date-time 2013-263T06:45:22.473Z :ordinal-date-time-no-ms 2013-263T06:45:22Z :rfc822 Fri, 20 Sep 2013 06:45:22 +0000 ... There are a lot of different built in formatters but unfortunately I couldn’t find one that exactly matched our date format so we’ll have to write our own one. For that we’ll need to refresh our knowledge of Java date formatting: We end up with the following formatter: > (f/parse (f/formatter "dd MMM YYYY") string-date) # It took me much longer than it should have to remember that ‘MMM’ is the pattern to match a short form of a month but it’s just the same as what we’d have to do in Java but with some neat wrapper functions.