I happen to read through a chapter on XML parsing and building APIs in Java. And I tried out the different parsers on a sample XML. Then I thought of sharing it on my blog so that I can have a reference to the code as well as a reference for anyone reading this. In this post I parse the same XML in different parsers to perform the same operation of populating the XML content into objects and then adding the objects to a list. The sample XML considered in the examples is: Rakesh Mishra Bangalore John Davis Chennai Rajesh Sharma Pune And the obejct into which the XML content is to be extracted is defined as below: class Employee{ String id; String firstName; String lastName; String location; @Override public String toString() { return firstName+" "+lastName+"("+id+")"+location; } } There are 3 main parsers for which I have given sample code: DOM Parser SAX Parser StAX Parser Using DOM Parser I am making use of the DOM parser implementation that comes with the JDK and in my example I am using JDK 7. The DOM Parser loads the complete XML content into a Tree structure. And we iterate through the Node and NodeList to get the content of the XML. The code for XML parsing using DOM parser is given below. public class DOMParserDemo { public static void main(String[] args) throws Exception { //Get the DOM Builder Factory DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance(); //Get the DOM Builder DocumentBuilder builder = factory.newDocumentBuilder(); //Load and Parse the XML document //document contains the complete XML as a Tree. Document document = builder.parse( ClassLoader.getSystemResourceAsStream("xml/employee.xml")); List empList = new ArrayList<>(); //Iterating through the nodes and extracting the data. NodeList nodeList = document.getDocumentElement().getChildNodes(); for (int i = 0; i < nodeList.getLength(); i++) { //We have encountered an tag. Node node = nodeList.item(i); if (node instanceof Element) { Employee emp = new Employee(); emp.id = node.getAttributes(). getNamedItem("id").getNodeValue(); NodeList childNodes = node.getChildNodes(); for (int j = 0; j < childNodes.getLength(); j++) { Node cNode = childNodes.item(j); //Identifying the child tag of employee encountered. if (cNode instanceof Element) { String content = cNode.getLastChild(). getTextContent().trim(); switch (cNode.getNodeName()) { case "firstName": emp.firstName = content; break; case "lastName": emp.lastName = content; break; case "location": emp.location = content; break; } } } empList.add(emp); } } //Printing the Employee list populated. for (Employee emp : empList) { System.out.println(emp); } } } class Employee{ String id; String firstName; String lastName; String location; @Override public String toString() { return firstName+" "+lastName+"("+id+")"+location; } } The output for the above will be: Rakesh Mishra(111)Bangalore John Davis(112)Chennai Rajesh Sharma(113)Pune Using SAX Parser SAX Parser is different from the DOM Parser where SAX parser doesn’t load the complete XML into the memory, instead it parses the XML line by line triggering different events as and when it encounters different elements like: opening tag, closing tag, character data, comments and so on. This is the reason why SAX Parser is called an event based parser. Along with the XML source file, we also register a handler which extends the DefaultHandler class. The DefaultHandler class provides different callbacks out of which we would be interested in: startElement() – triggers this event when the start of the tag is encountered. endElement() – triggers this event when the end of the tag is encountered. characters() – triggers this event when it encounters some text data. The code for parsing the XML using SAX Parser is given below: import java.util.ArrayList; import java.util.List; import javax.xml.parsers.SAXParser; import javax.xml.parsers.SAXParserFactory; import org.xml.sax.Attributes; import org.xml.sax.SAXException; import org.xml.sax.helpers.DefaultHandler; public class SAXParserDemo { public static void main(String[] args) throws Exception { SAXParserFactory parserFactor = SAXParserFactory.newInstance(); SAXParser parser = parserFactor.newSAXParser(); SAXHandler handler = new SAXHandler(); parser.parse(ClassLoader.getSystemResourceAsStream("xml/employee.xml"), handler); //Printing the list of employees obtained from XML for ( Employee emp : handler.empList){ System.out.println(emp); } } } /** * The Handler for SAX Events. */ class SAXHandler extends DefaultHandler { List empList = new ArrayList<>(); Employee emp = null; String content = null; @Override //Triggered when the start of tag is found. public void startElement(String uri, String localName, String qName, Attributes attributes) throws SAXException { switch(qName){ //Create a new Employee object when the start tag is found case "employee": emp = new Employee(); emp.id = attributes.getValue("id"); break; } } @Override public void endElement(String uri, String localName, String qName) throws SAXException { switch(qName){ //Add the employee to list once end tag is found case "employee": empList.add(emp); break; //For all other end tags the employee has to be updated. case "firstName": emp.firstName = content; break; case "lastName": emp.lastName = content; break; case "location": emp.location = content; break; } } @Override public void characters(char[] ch, int start, int length) throws SAXException { content = String.copyValueOf(ch, start, length).trim(); } } class Employee { String id; String firstName; String lastName; String location; @Override public String toString() { return firstName + " " + lastName + "(" + id + ")" + location; } } The output for the above would be: Rakesh Mishra(111)Bangalore John Davis(112)Chennai Rajesh Sharma(113)Pune Using StAX Parser StAX stands for Streaming API for XML and StAX Parser is different from DOM in the same way SAX Parser is. StAX parser is also in a subtle way different from SAX parser. The SAX Parser pushes the data but StAX parser pulls the required data from the XML. The StAX parser maintains a cursor at the current position in the document allows to extract the content available at the cursor whereas SAX parser issues events as and when certain data is encountered. XMLInputFactory and XMLStreamReader are the two class which can be used to load an XML file. And as we read through the XML file using XMLStreamReader, events are generated in the form of integer values and these are then compared with the constants in XMLStreamConstants. The below code shows how to parse XML using StAX parser: import java.util.ArrayList; import java.util.List; import javax.xml.stream.XMLInputFactory; import javax.xml.stream.XMLStreamConstants; import javax.xml.stream.XMLStreamException; import javax.xml.stream.XMLStreamReader; public class StaxParserDemo { public static void main(String[] args) throws XMLStreamException { List empList = null; Employee currEmp = null; String tagContent = null; XMLInputFactory factory = XMLInputFactory.newInstance(); XMLStreamReader reader = factory.createXMLStreamReader( ClassLoader.getSystemResourceAsStream("xml/employee.xml")); while(reader.hasNext()){ int event = reader.next(); switch(event){ case XMLStreamConstants.START_ELEMENT: if ("employee".equals(reader.getLocalName())){ currEmp = new Employee(); currEmp.id = reader.getAttributeValue(0); } if("employees".equals(reader.getLocalName())){ empList = new ArrayList<>(); } break; case XMLStreamConstants.CHARACTERS: tagContent = reader.getText().trim(); break; case XMLStreamConstants.END_ELEMENT: switch(reader.getLocalName()){ case "employee": empList.add(currEmp); break; case "firstName": currEmp.firstName = tagContent; break; case "lastName": currEmp.lastName = tagContent; break; case "location": currEmp.location = tagContent; break; } break; case XMLStreamConstants.START_DOCUMENT: empList = new ArrayList<>(); break; } } //Print the employee list populated from XML for ( Employee emp : empList){ System.out.println(emp); } } } class Employee{ String id; String firstName; String lastName; String location; @Override public String toString(){ return firstName+" "+lastName+"("+id+") "+location; } } The output for the above is: Rakesh Mishra(111) Bangalore John Davis(112) Chennai Rajesh Sharma(113) Pune With this I have covered parsing the same XML document and performing the same task of populating the list of Employee objects using all the three parsers namely: DOM Parser SAX Parser StAX Parser

Avro has the ability to generate Java code from Avro schema, IDL and protocol files. Avro also has a plugin which allows you to generate these Java sources directly from Maven, which is a good idea as it avoids issues that can arise if your schema/protocol files stray from the checked-in code generated equivalents. Today I created a simple GitHub project called avro-maven because I had to fiddle a bit to get Avro and Maven to play nice. The GitHub project is self-contained and also has a README which goes over the basics. In this post I’ll go over how to use Maven to generate code for schema, IDL and protocol files. pom.xml updates to support the Avro plugin Avro schema files only define types, whereas IDL and protocol files model types as well as RPC semantics such as messages. The only difference between IDL and protocol files is that IDL files are Avro’s DSL for specifying RPC, versus protocol files are the same in JSON form. Each type of file has an entry that can be used in the goals element as can be seen below. All three can be used together, or if you only have schema files you can safely remove the protocol and idl-protocol entries (and vice-versa). org.apache.avro avro-maven-plugin ${avro.version} generate-sources schema protocol idl-protocol ... org.apache.avro avro ${avro.version} org.apache.avro avro-maven-plugin ${avro.version} org.apache.avro avro-compiler ${avro.version} org.apache.avro avro-ipc ${avro.version} By default the plugin assumes that your Avro sources are located in ${basedir}/src/main/avro, and that you want your generated sources to be written to ${project.build.directory}/generated-sources/avro, where ${project.build.directory} is typically the target directory. Keep reading if you want to change any of these settings. Avro configurables Luckily Avro’s Maven plugin offers the ability to customize various code generation settings. The following table shows the configurables that can be used for any of the schema, IDL and protocol code generators. Configurable Default value Description sourceDirectory ${basedir}/src/main/avro The Avro source directory for schema, protocol and IDL files. outputDirectory ${project.build.directory}/generated-sources/avro The directory where Avro writes code-generated sources. testSourceDirectory ${basedir}/src/test/avro The input directory containing any Avro files used in testing. testOutputDirectory ${project.build.directory}/generated-test-sources/avro The output directory where Avro writes code-generated files for your testing purposes. fieldVisibility PUBLIC_DEPRECATED Determines the accessibility of fields (e.g. whether they are public or private). Must be one of PUBLIC, PUBLIC_DEPRECATED or PRIVATE. PUBLIC_DEPRECATED merely adds a deprecated annotation to each field, e.g. "@Deprecated public long time". In addition, the includes and testIncludes configurables can also be used to specify alternative file extensions to the defaults, which are **/*.avsc, **/*.avpr and **/*.avdl for schema, protocol and IDL files respectively. Let’s look at an example of how we can specify all of these options for schema compilation. org.apache.avro avro-maven-plugin ${avro.version} generate-sources schema ${project.basedir}/src/main/myavro/ ${project.basedir}/src/main/java/ ${project.basedir}/src/main/myavro/ ${project.basedir}/src/test/java/ PRIVATE **/*.avro **/*.test As a reminder everything covered in this blog article can be seen in action in the GitHub repo at https://github.com/alexholmes/avro-maven.

There are many good ideas and practices in Agile development, ideas and practices that definitely work: breaking projects into Small Releases to manage risk and accelerate feedback; time-boxing to limit WIP and keep everyone focused; relying only on working software as the measure of progress; simple estimating and using velocity to forecast team performance; working closely and constantly with the customer; and Continuous Integration – and Continuous Delivery – to ensure that code is always working and stable. But there are other commonly accepted ideas and best practices that aren’t important: if you don’t follow them, nothing bad will happen to you and your project will still succeed. And there are a couple that you are better off not following at all. Test-Driven Development Teams that need to move quickly need to depend on a fast, efficient testing safety net. With Test First Development or Test-Driven Development (TDD), there’s no excuse for not writing tests – after all, you have to write a failing test before you write the code. So you end up with a good set of working automated tests that ensure a high level of coverage and regression protection. TDD is not only a way of ensuring that developers test their code. It is also advocated as a design technique that leads to better quality code and a simpler, cleaner design. A study of teams at Microsoft and IBM (Realizing Quality Improvement through Test Driven Development, Microsoft Research, 2008) found that while TDD increased upfront development costs between 15-35% (TDD demands developers change the way that they think and work, which slows developers down, at least at first), it reduced defect density by 40% (IBM) or as much as 60-90% (Microsoft) over teams that did not follow disciplined unit testing. But in Making Software Chapter 12 “How Effective is Test-Driven Development” researchers led by Burak Turhan found that while TDD improves external quality (measured by one or more of test cases passed, number of defects, defect density, defects per test, effort required to fix defects, change density, % of preventative changes) and can improve the quality of the tests (fewer mistakes in the tests, tests that are easier to maintain), TDD does not consistently improve the quality of the design. TDD seems to reduce code complexity and improve reuse, however it also negatively impacts coupling and cohesion. And while method and class-level complexity is better in code developed using TDD, project/package level complexity is worse. People who like TDD like it a lot, so if you like it, do it. And even if you are not TDD-infected, there are times when working test first is natural – when you have to solve a specific problem in a specific way, or if you’re fixing a bug where the failing test case is already written up for you. But the important thing is that you write a good set of tests and keep them up to date and run them frequently – it doesn't matter if you write them before, or after, you write the code. Pair Programming According to the VersionOne State of Agile Development Survey 2012, almost 1/3 of teams follow pair programming – a surprisingly high number, given how disciplined pair programming is, and how few teams follow XP (2%) or Scrum/XP Hybrid (11%) methods where pair programming would be prescribed. There are good reasons for pairing: information sharing and improving code quality through continuous, informal code reviews as developers work together. And there are natural times to pair developers, or sometimes developers and testers, together: when you’re working through a hard design problem; or on code that you’ve never seen before and somebody who has worked on it is available to help; or when you’re over your head in troubleshooting a high-pressure problem; or testing a difficult part of the system; or when a new person joins the team and needs to learn about the code and coding practices. Some (extroverted) people enjoy pairing up, the energy it creates and the opportunities it provides to get to know others on the team. But forcing people who prefer working on their own or who don’t like each other to work closely together is definitely not a good idea. There are real social costs in pairing: you have to be careful to pair people up by skill, experience, style, personality type and work ethic. And sustained pair programming can be exhausting, especially over the long term – one study (Vanhanen and Lassenius 2007) found that people only pair between 1.5 and 4 hours a day on average, because it’s too intense to do all day long. In Pair Programming Considered Harmful? Jon Evans says that pairing can have also negative effects on creativity: Research strongly suggests that people are more creative when they enjoy privacy and freedom from interruption … What distinguished programmers at the top-performing companies wasn’t greater experience or better pay. It was how much privacy, personal workspace and freedom from interruption they enjoyed,” says a New York Times article castigating “the new groupthink”. And in “Still Questioning Extreme Programming” Pete McBreen points out some other disadvantages and weaknesses of pair programming: Exploration of ideas is not encouraged, pairing makes a developer focus on writing the code, so unless there is time in the day for solo exploration the team gets a very superficial level of understanding of the code. Developers can come to rely too much on the unit tests, assuming that if the tests pass then the code is OK. (This follows on from the lack of exploration.) Corner cases and edge cases are not investigated in detail, especially if they are hard to write tests for. Code that requires detail thinking about the design is hard to do when pairing unless one partner completely dominates the session. With the usual tradeoff between partners, it is hard to build technically complex designs unless they have been already been worked out in a solo session. Personal styles matter when pairing, and not all pairings are as productive as others. Pairs with different typing skills and proficiencies often result in the better typist doing all of the coding with the other partner being purely passive. And of course pairing in distributed teams doesn't work well if at all (depending on distance, differences in time zones, culture, working styles, language), although some people still try. While pairing does improve code quality over solo programming, you can get the same improvements in code quality, and at least some of the information sharing advantages, through code reviews, at less cost. Code reviews – especially lightweight, offline reviews – are easier to schedule, less expensive and less intrusive than pairing. And as Jason Cohen points out even if developers are pair programming, you may still need to do code reviews, because pair programming is really about joint problem solving, and doesn’t cover all of the issues that a code review would. Back to Jon Evans for the final word on pair programming: The true answer is that there is no one answer; that what works best is a dynamic combination of solitary, pair, and group work, depending on the context, using your best judgement. Paired programming definitely has its place. (Betteridge’s Law strikes again!) In some cases that place may even be “much of most days.” But insisting on 100 percent pairing is mindless dogma, and like all mindless dogma, ultimately counterproductive. Emergent Design and Metaphor Incremental development works, and trying to keep design simple makes good sense, but attempting to define an architecture on the fly is foolish and impractical. There’s a reason that almost nobody actually follows Emergent Design: it doesn't work. Relying on a high-level metaphor (the system is an "assembly line" or a "bill of materials" or a "hive of bees") shared by the team as some kind of substitute for architecture is even more ridiculous. Research from Carnegie Mellon University found that … natural language metaphors are relatively useless for either fostering communication among technical and non-technical project members or in developing architecture. Almost no one understands what a system metaphor is any ways, or how it is to be used, or how to choose a meaningful metaphor or how to change it if you got it wrong (and how you would know if you got it wrong), including one of the people who helped come up with the idea: Okay I might as well say it publicly - I still haven't got the hang of this metaphor thing. I saw it work, and work well on the C3 project, but it doesn't mean I have any idea how to do it, let alone how to explain how to do it. Martin Fowler, Is Design Dead? Agile development methods have improved development success and shown better ways to approach many different software development problems – but not architecture and design. Daily Standups When you have a new team and everyone needs to get to know each other and more time to understand what the project is about; or when the team is working under emergency conditions trying to fix something or finish something under extreme pressure, then getting everyone together in regular meetings, maybe even more than once a day, is necessary and valuable. But whether everyone stands up or sits down and what they end up talking about in a meeting should be up to you. If your team has been working well together for a while and everyone knows each other and knows what they are working on, and if developers update cards on a task board or a Kanban board or the status in an electronic system as they get things done, and if they are grown up enough to ask for help when they need it, then you don’t need to make them all stand up in a room every morning. Collective Code Ownership Letting everyone work on all of the code isn't always practical (because not everyone on the team has the requisite knowledge or experience to work on every problem) and collective code ownership can have negative effects on code quality. Share code where it makes sense to do so, but realize that not everybody can – or should – work on every part of the system. Writing All Requirements as Stories The idea that every requirement specification can be written as User Stories in 1 or 2 lines on cards, that requirements should be too short on purpose (so that the developer has to talk to someone to explain what’s really needed) and insisting that they should all be in the same template form “As a type of user I want some goal so that some reason…” is silly and unnecessary. This is the same kind of simple minded orthodoxy that led everyone to try to capture all requirements in UML Use Case format with stick men and bubbles 15 years ago. There are many different ways to effectively express requirements. Sometimes requirements need to be specified in detail (when you have to meet regulatory compliance or comply with a standard or integrate with an existing system or implement a specific algorithm or…). Sometimes it’s better to work from a test case or a detailed use case scenario or a wire frame or some other kind of model, because somebody who knows what’s going on has already worked out the details for you. So pick the format and level of detail that works best and get to work. Relying on a Product Owner Relying on one person as the Product Owner, as the single solitary voice of the customer and the “one throat to choke” when the project fails, doesn't scale, doesn't last, and puts the team and the project and eventually the business at risk. It’s a naïve, dangerous approach to designing a product and to managing a development project, and it causes more problems than it solves. Many teams have realized this and are trying to work around the Product Owner idea because they have to. To succeed, a team needs real and sustained customer engagement at multiple levels, and they should take responsibility themselves for making sure that they get what they need, rather than relying on one person to do it all.

In our previous article we have explained on how to protect the data while it is in transit through Transport Layer Security (TLS)/Secured Socket Layer (SSL). Now let us try to understand how to apply security mechanism for a JEE 6 based web application using LDAP server for authentication. Objective: • Configure a LDAP realm in the JEE Application Server • Apply JEE security to a sample web application. Products used: IDE: Netbeans 7.2 Java Development Kit (JDK): Version 6 Glassfish server: 3.1 Authentication Mechanism: Form Based authentication Authentication server: LDAP OpenDS v2.2 Apply JEE security to the sample web application: The JEE web applications can be secured either through Declarative security or Programmatic security. Declarative security can be implemented in JEE applications by using annotations or through deployment descriptor. This type of security mechanism is used when the roles and authentication process is simple, when it can make use of existing security providers (even external like LDAP, Kerberos). Programmatic security provides additional security mechanism when declarative security is not sufficient for the application in context. It is used when we require custom made security and when rich set of roles, authentication is required. Configure Realm in the Glassfish Application Server Before we configure a realm in the Glassfish Application server you will need to install and configure an LDAP server which we will be using for our project. You can get the complete instructions in the following article: “How to install and configure LDAP server”. Once the installation is successful start your Glassfish server and go to the admin console. Create a new LDAP Realm. Create new LDAP Realm Add the configuration settings as per the configurations set up done for the LDAP server. Glassfish Web App LDAP Realm JAAS Context – identifier which will be used in the application module to connect with the LDAP server. (e.g. ldapRealm) Directory – LDAP server URL path (e.g. ldap://localhost:389) Base DN: Distinguished name in the LDAP directory identifying the location of the user data. Applying JEE security to the web application Create a sample web application as per the following structure: SampleWebApp Directory Form based authentication mechanism will be used for authentication of the users. JEE Login and Authentication Let us explain the whole process with help of above diagram and the code. Set up a sample web application in Netbeans IDE. SampleWebApp in Netbeans IDE SampleWebApp Configuration Step 1: As explained in the above diagram a client browser tries to request for a protected resource from the websitehttp://{samplewebsite.com}/{contextroot}/index.jsp. The webserver goes into the web configuration file and figures out that the requested resource is protected. web.xml Code SecurityConstraint Secured resources /* GeneralUser Administrator NONE Step 2: The webserver presents the Login.jsp as a part of the Form based authentication mechanism to the client. These configurations are checked from the web configuration file. web.xml FORM ldapRealm /Login.jsp /LoginError.jsp Step 3: The client submits the login form to the web server. When the servers finds that the form action is “j_security_check” it processes the request to authenticate the client’s credential. The jsp form must contain the login elements j_username and j_password which will allow the web server to invoke the login authentication mechanism. Login.jsp username: password: While processing the request the webserver will send the authentication request to the LDAP server since LDAP realm is used in the login-config. The LDAP server will authenticate the user based on the username and password stored in the LDAP repository. Step 4: If the authentication is successful the secured resource (in this case index.jsp) is returned to the client and the container uses a session id to identify a login session for the client. The container maintains the login session with a cookie containing the session-id. The server sends this cookie back to the client, and as long as the client is able to show this cookie for subsequent requests, then the container easily recognize the client and hence maintains the session for this client. Step 5: Only if the authentication is unsuccessful the user will be redirected to the LoginError.jsp as per the configuration in the web.xml. /LoginError.jsp This shows how to apply form based security authentication to a sample web application. Now let us get a brief look on the secured resource which is used for this project. In this project the secured resource is index.jsp which accepts a username and forwards the request to LoginServlet. Login servlet dispatches the request to Success.jsp which then prints the username to the client. index.jsp Please type your name LoginServlet.java protected void processRequest(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { response.setContentType("text/html;charset=UTF-8"); PrintWriter out = response.getWriter(); try { RequestDispatcher requestDispatcher = getServletConfig().getServletContext(). getRequestDispatcher("/Success.jsp"); requestDispatcher.forward(request, response); } finally { out.close(); } } Success.jsp You have been successfully logged in as ${param.username} web.xml LoginServlet com.login.LoginServlet LoginServlet /LoginServlet You can download the complete working code from the below link. SampleWebApp-Code Download Hope our readers have enjoyed this article. Keep watching this space for more articles on JEE security.

With the release of ASP.NET MVC 4 one of the exciting features packed in the release was ASP.NET Web API.

This post will illustrate WSO2 ESB XSLT Transform Mediator with sample code. When we need XSLT Mediator is need in wso2 ESB? When we have to deal with a dynamic request for an ESB proxy. (If the proxy request is dynamic (not static/not predefined) we will have to use the XSLT mediator to mediate the message) What is XSLT Mediator ? The XSLT Mediator applies a specified XSLT transformation to a selected element of the current message payload. The source attribute specifies which element to have XSLT transformation. The feature element defines in to TransformerFactory. If source element is not specified then soap body as the selected element. Parameters can be passed into the transformations through the property elements and it is optional and those can be accessed during transformation by . What is the WSO2 ESB XSLT Mediator Syntax? * * * Source - Specifies which element to be selected to apply the given XSLT transformation. Property - Allows optional parameters to be passed into the transformations. Feature - Defines any features which should be explicitly set to the TransformerFactory. Resource - Resolves XSLT imports and includes from the repository. Here are some thing that I will be using in XSLT script The element applies a template to the current element or to the current element's child nodes. If we add a select attribute to the element it will process only the child element that matches the value of the attribute. We can use the select attribute to specify the order in which the child nodes are processed.[1] omit-xml-declaration values : yes OR no Optional. "yes" specifies that the XML declaration () should be omitted in the output. "no" specifies that the XML declaration should be included in the output. The default is "no" indent values: yes OR no Optional. "yes" indicates that the output should be indented according to its hierarchic structure. "no" indicates that the output should not be indented according to its hierarchic structure. This attribute is not supported by Netscape 6[2] Let do a sample We can try our Transport Services (BusServices)[3] Here Is the Expecting request 2 Back end Serive expecting request 2 1. Start BusServices in WSO2 AS 2. Start WSO2 ESB in offset 1 3. Go to Home > Manage > Service Bus > Local Entries in ESB 4. Then add Add In-lined XML Entry 5. Five name as "in_xslt" and Value as below (transform.xsl file contains) Now I need to change my respond from: 4Colombo Negombo Galle to: 2Colombo Negombo Galle Here is XSLT Script for that Add it to the out sequences in proxy Here is Final testing NOTE If any case If I get empty String in RootID I am adding new RootName Root text

Merge sort is a classical "divide and conquer" sorting algorithm. You should have to never write one because you'd be silly to do that when a standard library class already will already do it for you. But, it is useful to demonstrate a few characteristics of programming techniques in Scala. Firstly a quick recap on the merge sort. It is a divide and conquer algorithm. A list of elements is split up into smaller and smaller lists. When a list has one element it is considered sorted. It is then merged with the list beside it. When there are no more lists to merged the original data set is considered sorted. Now let's take a look how to do that using an imperative approach in Java. public void sort(int[] values) { int[] numbers = values; int[] auxillaryNumbers = new int[values.length]; mergesort(numbers, auxillaryNumbers, 0, values.length - 1); } private void mergesort(int [] numbers, int [] auxillaryNumbers, int low, int high) { // Check if low is smaller then high, if not then the array is sorted if (low < high) { // Get the index of the element which is in the middle int middle = low + (high - low) / 2; // Sort the left side of the array mergesort(numbers, auxillaryNumbers, low, middle); // Sort the right side of the array mergesort(numbers, auxillaryNumbers, middle + 1, high); // Combine them both // Alex: the first time we hit this when there is min difference between high and low. merge(numbers, auxillaryNumbers, low, middle, high); } } /** * Merges a[low .. middle] with a[middle..high]. * This method assumes a[low .. middle] and a[middle..high] are sorted. It returns * a[low..high] as an sorted array. */ private void merge(int [] a, int[] aux, int low, int middle, int high) { // Copy both parts into the aux array for (int k = low; k <= high; k++) { aux[k] = a[k]; } int i = low, j = middle + 1; for (int k = low; k <= high; k++) { if (i > middle) a[k] = aux[j++]; else if (j > high) a[k] = aux[i++]; else if (aux[j] < aux[i]) a[k] = aux[j++]; else a[k] = aux[i++]; } } public static void main(String args[]){ ... ms.sort(new int[] {5, 3, 1, 17, 2, 8, 19, 11}); ... } } Discussion... An auxillary array is used to achieve the sort. Elements to be sorted are copied into it and then once sorted copied back. It is important this array is only created once otherwise there can be a performance hit from extensive array created. The merge method does not have to create an auxiliary array however since it changes an object it means the merge method has side effects. Merge sort big(O) performance is N log N. Now let's have a go at a Scala solution. def mergeSort(xs: List[Int]): List[Int] = { val n = xs.length / 2 if (n == 0) xs else { def merge(xs: List[Int], ys: List[Int]): List[Int] = (xs, ys) match { case(Nil, ys) => ys case(xs, Nil) => xs case(x :: xs1, y :: ys1) => if (x < y) x::merge(xs1, ys) else y :: merge(xs, ys1) } val (left, right) = xs splitAt(n) merge(mergeSort(left), mergeSort(right)) } } Key discussion points: It is the same divide and conquer idea. The splitAt function is used to divide up the data up each time into a tuple. For every recursion this will new a new tuple. The local function merge is then used to perform the merging. Local functions are a useful feature as they help promote encapsulation and prevent code bloat. Neiher the mergeSort() or merge() functions have any side effects. They don't change any object. They create (and throw away) objects. Because the data is not been passed across iterations of the merging, there is no need to pass beginning and ending pointers which can get very buggy. This merge recursion uses pattern matching to great effect here. Not only is there matching for data lists but when a match happens the data lists are assigned to variables: x meaning the top element in the left list xs1 the rest of the left list y meaning the top element in the right list ys1 meaning the rest of the data in the right list This makes it very easy to compare the top elements and to pass around the rest of the date to compare. Would the iterative approach be possible in Java? Of course. But it would be much more complex. You don't have any pattern matching and you don't get a nudge to declare objects as immutable as Scala does with making you make something val or var. In Java, it would always be easier to read the code for this problem if it was done in an imperative style where objects are being changed across iterations of a loop. But Scala a functional recursive approach can be quite neat. So here we see an example of how Scala makes it easier to achieve good, clean, concise recursion and a make a functional approach much more possible.

intro i was aware of thread local but never had the occasion to really use it until recently. so i started digging a little bit on the subject because i needed an easy way of propagating some user information via the different layers of my web application without changing the signature of each method called. small prerequisite info a thread is an individual process that has its own call stack. in java, there is one thread per call stack or one call stack per thread. even if you don’t create any new threads in your program, threads are there running without your knowledge.best example is whenyou just start a simple java program via main method,then you do not implicitly call new thread().start(), but the jvm creates a main thread for you in order to run the main method. the main thread is quite special because it is the thread from which all the other thread will spawn and when this thread is finished, the application ends it’s lifecycle. in a web application server normally there is a pool of of threads ,because a thread is class quite heavyweight to create.all jee servers (weblogic,glassfish,jboss etc) have a self tuning thread pool, meaning that the thread pool increase and decrease when is needed so there is not thread created on each request, and existing ones are reused. understanding thread local in order to understand better thread local i will show very simplistic implementation of one custom thread local. package ccs.progest.javacodesamples.threadlocal.ex1; import java.util.hashmap; import java.util.map; public class customthreadlocal { private static map threadmap = new hashmap(); public static void add(object object) { threadmap.put(thread.currentthread(), object); } public static void remove(object object) { threadmap.remove(thread.currentthread()); } public static object get() { return threadmap.get(thread.currentthread()); } } so you can call anytime in your application the add method on customthreadlocal and what it will do is to put in a map the current thread as key and as value the object you want to associate with this thread. this object might be an object that you want to have access to from anywhere within the current executed thread, or it might be an expensive object you want to keep associated with the thread and reuse as many times you want. you define a class threadcontext where you have all information you want to propagate within the thread. package ccs.progest.javacodesamples.threadlocal.ex1; public class threadcontext { private string userid; private long transactionid; public string getuserid() { return userid; } public void setuserid(string userid) { this.userid = userid; } public long gettransactionid() { return transactionid; } public void settransactionid(long transactionid) { this.transactionid = transactionid; } public string tostring() { return "userid:" + userid + ",transactionid:" + transactionid; } } now is the time to use the threadcontext. i will start two threads and in each thread i will add a new threadcontext instance that will hold information i want to propagate for each thread. package ccs.progest.javacodesamples.threadlocal.ex1; public class threadlocalmainsampleex1 { public static void main(string[] args) { new thread(new runnable() { public void run() { threadcontext threadcontext = new threadcontext(); threadcontext.settransactionid(1l); threadcontext.setuserid("user 1"); customthreadlocal.add(threadcontext); //here we call a method where the thread context is not passed as parameter printthreadcontextvalues.printthreadcontextvalues(); } }).start(); new thread(new runnable() { public void run() { threadcontext threadcontext = new threadcontext(); threadcontext.settransactionid(2l); threadcontext.setuserid("user 2"); customthreadlocal.add(threadcontext); //here we call a method where the thread context is not passed as parameter printthreadcontextvalues.printthreadcontextvalues(); } }).start(); } } notice: customthreadlocal.add(threadcontext) is the line of code where the current thread is associated with the threadcontext instance as you will see executing this code the result will be: userid:user 1,transactionid:1 userid:user 2,transactionid:2 how this is possible because we did not passed as parameter threadcontext ,userid or trasactionid to printthreadcontextvalues ? package ccs.progest.javacodesamples.threadlocal.ex1; public class printthreadcontextvalues { public static void printthreadcontextvalues(){ system.out.println(customthreadlocal.get()); } } simple enough when customthreadlocal.get() is called from the internal map of customthreadlocal it is retrived the object associated with the current thread. now let’s see the samples when is used a real threadlocal class. (the above customthreadlocal class is just to understand the principles behind threadlocal class which is very fast and uses memory in an optimal way) package ccs.progest.javacodesamples.threadlocal.ex2; public class threadcontext { private string userid; private long transactionid; private static threadlocal threadlocal = new threadlocal(){ @override protected threadcontext initialvalue() { return new threadcontext(); } }; public static threadcontext get() { return threadlocal.get(); } public string getuserid() { return userid; } public void setuserid(string userid) { this.userid = userid; } public long gettransactionid() { return transactionid; } public void settransactionid(long transactionid) { this.transactionid = transactionid; } public string tostring() { return "userid:" + userid + ",transactionid:" + transactionid; } } as javadoc describes : threadlocal instances are typically private static fields in classes that wish to associate state with a thread package ccs.progest.javacodesamples.threadlocal.ex2; public class threadlocalmainsampleex2 { public static void main(string[] args) { new thread(new runnable() { public void run() { threadcontext threadcontext = threadcontext.get(); threadcontext.settransactionid(1l); threadcontext.setuserid("user 1"); //here we call a method where the thread context is not passed as parameter printthreadcontextvalues.printthreadcontextvalues(); } }).start(); new thread(new runnable() { public void run() { threadcontext threadcontext = threadcontext.get(); threadcontext.settransactionid(2l); threadcontext.setuserid("user 2"); //here we call a method where the thread context is not passed as parameter printthreadcontextvalues.printthreadcontextvalues(); } }).start(); } } when get is called , a new threadcontext instance is associated with the current thread,then the desired values are set the threadcontext instance. as you see the result is the same as for the first set of samples. userid:user 1,transactionid:1 userid:user 2,transactionid:2 (it might be the reverse order ,so don’t worry if you see ‘user 2′ first) package ccs.progest.javacodesamples.threadlocal.ex2; public class printthreadcontextvalues { public static void printthreadcontextvalues(){ system.out.println(threadcontext.get()); } } another very usefull usage of threadlocal is the situation when you have a non threadsafe instance of an quite expensive object.most poular sample i found was with simpledateformat (but soon i’ll come with another example when webservices ports will be used) package ccs.progest.javacodesamples.threadlocal.ex4; import java.text.simpledateformat; import java.util.date; public class threadlocaldateformat { // simpledateformat is not thread-safe, so each thread will have one private static final threadlocal formatter = new threadlocal() { @override protected simpledateformat initialvalue() { return new simpledateformat("mm/dd/yyyy"); } }; public string formatit(date date) { return formatter.get().format(date); } } conclusion: there are many uses for thread locals. here i describe only two: (i think the most used ones) genuine per-thread context, such as user id or transaction id. per-thread instances for performance.

A common feature that you will come across in websites is the ability to resize an image to fit an exact size so that it will be displayed correctly on your design. If you have a very large image and you are going to place it on your website in a space that is only 100px x 100px then you will want to be able to resize this image to fit in the space. One option is to just set the width and height attributes on the image tag in your HTML, this will force the image to be displayed at this size. This will work perfectly fine and the image will fit in the 100px x 100px space but the problem is that when the browser loads the image it will not resize the image but will just display it in the limited size. This means that the image will still need to be downloaded at full size, if the image is very large it can take some time for you to download a large image just to be displayed in a small space. A better solution would be to resize the image to 100px x 100px which will reduce it's size so that the browser doesn't need to download a large image just to display it in a small space. In this tutorial we are going to create a PHP class that will allow you to resize an image to any dimension you want, it will allow you to resize while keeping the aspect ratio of the image. When the class has resized the image you can either save the image on the server or download the image. Class Methods First lets plan out the class we are going to create: We need to pass an existing image to the class, this is the image that we will use to resize. We need to pass in the desired image dimensions so the class can work out what's the new size of the image. Then we need to be able to save the image to a location on the server, and choose to download the image. The Constructor This class relies on an original image being found and set on the class, without this the class will not work correctly. Because of this we should pass the image filename into the constructor of the class. This will then check if the file exists on the server, if it does then we can call the set image method where we can get the image and create a resource of the image and store this in a class variable. /** * Class constructor requires to send through the image filename * * @param string $filename - Filename of the image you want to resize */ public function __construct( $filename ) { if(file_exists($filename)) { $this->setImage( $filename ); } else { throw new Exception('Image ' . $filename . ' can not be found, try another image.'); } } Set The Image The set image method is used to create a image resource based on the image given to the class this uses the PHP functions imagecreatefromjpeg, imagecreatefromgif, imagecreatefrompng to create the image resource from the given image. We can then use this with the functions imagesx and imagesy to return the current width and height of the image. This will allow us to resize the image easier later in the script. /** * Set the image variable by using image create * * @param string $filename - The image filename */ private function setImage( $filename ) { $size = getimagesize($filename); $this->ext = $size['mime']; switch($this->ext) { // Image is a JPG case 'image/jpg': case 'image/jpeg': // create a jpeg extension $this->image = imagecreatefromjpeg($filename); break; // Image is a GIF case 'image/gif': $this->image = @imagecreatefromgif($filename); break; // Image is a PNG case 'image/png': $this->image = @imagecreatefrompng($filename); break; // Mime type not found default: throw new Exception("File is not an image, please use another file type.", 1); } $this->origWidth = imagesx($this->image); $this->origHeight = imagesy($this->image); } Resize The Image The resize function is what we will use to calculate the new values for the image width and height. This takes 3 parameters the new width, new height and the resize option, this will allow us to resize the image to exact dimensions, use the defined width with the height and keep aspect ratio, use the define height keeping the aspect ratio or let the class decide the best way of resizing the image. Once we have the new width and height of the image we can create a new image resource by using the PHP function imagecreatetruecolor(). Now we can create the new image from the old image resizing it to the new dimensions by using the imagecopyresampled() function. /** * Resize the image to these set dimensions * * @param int $width - Max width of the image * @param int $height - Max height of the image * @param string $resizeOption - Scale option for the image * * @return Save new image */ public function resizeTo( $width, $height, $resizeOption = 'default' ) { switch(strtolower($resizeOption)) { case 'exact': $this->resizeWidth = $width; $this->resizeHeight = $height; break; case 'maxwidth': $this->resizeWidth = $width; $this->resizeHeight = $this->resizeHeightByWidth($width); break; case 'maxheight': $this->resizeWidth = $this->resizeWidthByHeight($height); $this->resizeHeight = $height; break; default: if($this->origWidth > $width || $this->origHeight > $height) { if ( $this->origWidth > $this->origHeight ) { $this->resizeHeight = $this->resizeHeightByWidth($width); $this->resizeWidth = $width; } else if( $this->origWidth < $this->origHeight ) { $this->resizeWidth = $this->resizeWidthByHeight($height); $this->resizeHeight = $height; } } else { $this->resizeWidth = $width; $this->resizeHeight = $height; } break; } $this->newImage = imagecreatetruecolor($this->resizeWidth, $this->resizeHeight); imagecopyresampled($this->newImage, $this->image, 0, 0, 0, 0, $this->resizeWidth, $this->resizeHeight, $this->origWidth, $this->origHeight); } /** * Get the resized height from the width keeping the aspect ratio * * @param int $width - Max image width * * @return Height keeping aspect ratio */ private function resizeHeightByWidth($width) { return floor(($this->origHeight / $this->origWidth) * $width); } /** * Get the resized width from the height keeping the aspect ratio * * @param int $height - Max image height * * @return Width keeping aspect ratio */ private function resizeWidthByHeight($height) { return floor(($this->origWidth / $this->origHeight) * $height); } Save The Image With the new image now set in a class variable we can now use this to save the image on the server. This function will take 3 parameters the save path, the image quality and if we want to download the image. For each mime type PHP has a function imagejpeg(), imagegif(), imagepng() that will allow you to save the image by passing in the new image resource and the path the image is going to be saved. Once this image is saved on the server and we decided to download it we can change the headers to allow the browser to download the image on the clients machine. /** * Save the image as the image type the original image was * * @param String[type] $savePath - The path to store the new image * @param string $imageQuality - The qulaity level of image to create * * @return Saves the image */ public function saveImage($savePath, $imageQuality="100", $download = false) { switch($this->ext) { case 'image/jpg': case 'image/jpeg': // Check PHP supports this file type if (imagetypes() & IMG_JPG) { imagejpeg($this->newImage, $savePath, $imageQuality); } break; case 'image/gif': // Check PHP supports this file type if (imagetypes() & IMG_GIF) { imagegif($this->newImage, $savePath); } break; case 'image/png': $invertScaleQuality = 9 - round(($imageQuality/100) * 9); // Check PHP supports this file type if (imagetypes() & IMG_PNG) { imagepng($this->newImage, $savePath, $invertScaleQuality); } break; } if($download) { header('Content-Description: File Transfer'); header("Content-type: application/octet-stream"); header("Content-disposition: attachment; filename= ".$savePath.""); readfile($savePath); } imagedestroy($this->newImage); } Full Resize Image Class setImage( $filename ); } else { throw new Exception('Image ' . $filename . ' can not be found, try another image.'); } } /** * Set the image variable by using image create * * @param string $filename - The image filename */ private function setImage( $filename ) { $size = getimagesize($filename); $this->ext = $size['mime']; switch($this->ext) { // Image is a JPG case 'image/jpg': case 'image/jpeg': // create a jpeg extension $this->image = imagecreatefromjpeg($filename); break; // Image is a GIF case 'image/gif': $this->image = @imagecreatefromgif($filename); break; // Image is a PNG case 'image/png': $this->image = @imagecreatefrompng($filename); break; // Mime type not found default: throw new Exception("File is not an image, please use another file type.", 1); } $this->origWidth = imagesx($this->image); $this->origHeight = imagesy($this->image); } /** * Save the image as the image type the original image was * * @param String[type] $savePath - The path to store the new image * @param string $imageQuality - The qulaity level of image to create * * @return Saves the image */ public function saveImage($savePath, $imageQuality="100", $download = false) { switch($this->ext) { case 'image/jpg': case 'image/jpeg': // Check PHP supports this file type if (imagetypes() & IMG_JPG) { imagejpeg($this->newImage, $savePath, $imageQuality); } break; case 'image/gif': // Check PHP supports this file type if (imagetypes() & IMG_GIF) { imagegif($this->newImage, $savePath); } break; case 'image/png': $invertScaleQuality = 9 - round(($imageQuality/100) * 9); // Check PHP supports this file type if (imagetypes() & IMG_PNG) { imagepng($this->newImage, $savePath, $invertScaleQuality); } break; } if($download) { header('Content-Description: File Transfer'); header("Content-type: application/octet-stream"); header("Content-disposition: attachment; filename= ".$savePath.""); readfile($savePath); } imagedestroy($this->newImage); } /** * Resize the image to these set dimensions * * @param int $width - Max width of the image * @param int $height - Max height of the image * @param string $resizeOption - Scale option for the image * * @return Save new image */ public function resizeTo( $width, $height, $resizeOption = 'default' ) { switch(strtolower($resizeOption)) { case 'exact': $this->resizeWidth = $width; $this->resizeHeight = $height; break; case 'maxwidth': $this->resizeWidth = $width; $this->resizeHeight = $this->resizeHeightByWidth($width); break; case 'maxheight': $this->resizeWidth = $this->resizeWidthByHeight($height); $this->resizeHeight = $height; break; default: if($this->origWidth > $width || $this->origHeight > $height) { if ( $this->origWidth > $this->origHeight ) { $this->resizeHeight = $this->resizeHeightByWidth($width); $this->resizeWidth = $width; } else if( $this->origWidth < $this->origHeight ) { $this->resizeWidth = $this->resizeWidthByHeight($height); $this->resizeHeight = $height; } } else { $this->resizeWidth = $width; $this->resizeHeight = $height; } break; } $this->newImage = imagecreatetruecolor($this->resizeWidth, $this->resizeHeight); imagecopyresampled($this->newImage, $this->image, 0, 0, 0, 0, $this->resizeWidth, $this->resizeHeight, $this->origWidth, $this->origHeight); } /** * Get the resized height from the width keeping the aspect ratio * * @param int $width - Max image width * * @return Height keeping aspect ratio */ private function resizeHeightByWidth($width) { return floor(($this->origHeight/$this->origWidth)*$width); } /** * Get the resized width from the height keeping the aspect ratio * * @param int $height - Max image height * * @return Width keeping aspect ratio */ private function resizeWidthByHeight($height) { return floor(($this->origWidth/$this->origHeight)*$height); } } ?> Using The Resize Image PHP Class Because we have created this to allow you to resize the image in multiple ways it means that there are different ways of using the class. Resize the image to an exact size. Resize the image to a max width size keeping aspect ratio of the image. Resize the image to a max height size keeping aspect ratio of the image. Resize the image to a given width and height and allow the code to work out which way of resizing is best keeping the aspect ratio. You can save the created resize image on the server. You can download the created resize image on the server. Resize Exact Size To resize an image to an exact size you can use the following code. First pass in the image we want to resize in the class constructor, then define the width and height with the scale option of exact. The class will now have the create dimensions to create the new image, now call the function saveImage() and pass in the new file location to the new image. $resize = new ResizeImage('images/Be-Original.jpg'); $resize->resizeTo(100, 100, 'exact'); $resize->saveImage('images/be-original-exact.jpg'); Resize Max Width Size If you choose to set the image to be an exact size then when the image is resized it could lose it's aspect ratio, which means the image could look stretched. But if you know the max width that you want the image to be you can resize the image to a max width, this will keep the aspect ratio of the image. $resize = new ResizeImage('images/Be-Original.jpg'); $resize->resizeTo(100, 100, 'maxWidth'); $resize->saveImage('images/be-original-maxWidth.jpg'); Resize Max Height Size Just as you can select a max width for the image while keeping aspect ratio you can also select a max height while keeping aspect ratio. $resize = new ResizeImage('images/Be-Original.jpg'); $resize->resizeTo(100, 100, 'maxHeight'); $resize->saveImage('images/be-original-maxHeight.jpg'); Resize Auto Size From Given Width And Height You can also allow the code to work out the best way to resize the image, so if the image height is larger than the width then it will resize the image by using the height and keeping aspect ratio. If the image width is larger than the height then the image will be resized using the width and keeping the aspect ratio. $resize = new ResizeImage('images/Be-Original.jpg'); $resize->resizeTo(100, 100); $resize->saveImage('images/be-original-default.jpg'); Download The Resized Image The default behaviour for this class is to save the image on the server, but you can easily change this to download by passing in a true parameter to the saveImage method. $resize = new ResizeImage('images/Be-Original.jpg'); $resize->resizeTo(100, 100, 'exact'); $resize->saveImage('images/be-original-exact.jpg', "100", true);

As people know, I'm a huge fan of PhoneGap and what it allows me to do with JavaScript, HTML, and CSS. But I think it is crucial to remember that you don't always need PhoneGap. A great example of that is camera access. Did you know that recent mobile browsers support accessing the camera directly from HTML and JavaScript? Let's look at an example. Over a year ago I wrote a blog post where I created an application called "Color Thief." This application made use of PhoneGap's Camera API and a third party JavaScript library called Color Thief. I loved this example because it demonstrated how you could combine the extra power that PhoneGap provides along with existing JavaScript libraries. This morning I watched an excellent Google IO presentation (https://www.youtube.com/watch?v=EPYnGFEcis4&feature=youtube_gdata_player) on Mobile HTML. It was an overview of some of the exciting stuff you can now do with mobile HTML and JavaScript. To be clear, this was all without using wrappers like PhoneGap. In one of the examples the presenters discussed the new "capture" support for the input/file field type. This is rather simple to implement: If supported (recent Android and latest iOS), the user can then use their camera to select a picture. I decided to rebuild my old demo to skip PhoneGap completely and just make use of this feature. Here's the code: For the most part, this is pretty similar to the last version. I no longer wait for the deviceready event, but instead just listen for the document itself to load. Instead of listening for a button click, I've switched to a input field using type=file. I now listen for the change event, and on that, I see if I have access to a file. If I do, I can then use the URL object to create a pointer to the source and then simply add it to my DOM. After that, Color Thief takes over. The only tricky part I ran into was that in iOS the URL object is still prefixed. You can see how I get around that in the startup code. To be fair, this isn't 100% backwards compatible, I could add a few checks in here to ensure that things will work and gracefully let people on older phones know they can't use this feature. But the end result is nearly the exact same functionality in a web page - no PhoneGap, no native code. <br>

A few days ago I was going through a project's Maven dependencies, removing unused junk, checking jar file version numbers adding a little dependency management and generally tidying up (yes, I know that this isn't something we often get time to do, but even Maven dependencies can be a form of technical debt). After recompiling and running the unit tests I ran some end to end tests only to find that the whole thing fell apart... Big time. The exception I got was the usual one that all Spring developers get, a java.lang.IllegalStateException: Failed to load ApplicationContext ...exception. This is nothing new and as a Spring developer you find the problem, which is usually a missing bean definition and move on. Only this time it was something different, and that's because the cause was: java.lang.NoSuchFieldError: NULL ...which gives you no clues about what's going wrong. Now I knew that I'd been messing around with the project's dependencies, so I must have broken something somewhere. It turned out that it was a transient dependency problem. I was using Spring Security version 3.1.1-RELEASE, which is built using version 3.0.7-RELEASE of the Spring core libraries and not as you'd expect version 3.1.1-RELEASE. This meant that I'd ended up with different and incompatible versions of some of the Spring libraries on my classpath. You may well wonder why the Guys at Spring Security build their code with version 3.0.7-RELEASE and they say that this is intentional and that it's to do with backwards compatibility issues. As Rob Winch, Spring Security Lead at SpringSource, says: "Spring Security uses 3.0.x (intentionally to support users that require it). For this reason, if you build with Maven and want to use Spring 3.1 you must either exclude the Spring dependencies in your maven pom, explicitly add the Spring 3.1 dependencies to your pom, or add a dependency management section to your pom. This is not a bug. Even if Spring Security was changed to use Spring 3.1 by default, the users using Spring 3.0 would encounter the same problem. The reason this occurs is due to the algorithm that Maven uses to resolve transitive dependency versions [1]" Once you know how, the problem is easy to spot. If you're using STS/eclipse you can easily examine Maven dependencies using the POM editor. The fix is simple too, all you need to do is to explicitly define the wayward Spring libraries in your POM. For example: org.springframework spring-core 3.1.1-RELEASE Finally, you can check that it's fixed using STS/eclipse's POM file editor, where you'll see that the unwanted version is now labelled as "omitted". [1] http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#Transitive_Dependencies

Github has a nice API for inspecting repositories – it lets you read gists, issues, commit history, files and so on. Git repository data lends itself to demonstrating the power of combining full text and faceted search, as there is a mix of free text fields (commit messages, code) and enumerable fields (committers, dates, committer employers). Github APIs return JSON, which has the nice property of resembling a tree structure – results can be recursed over without fear of infinite loops. Note that to download the entire commit history for a repository, you need to page through it by sha hash. The API I use here lacks diffs, which must be retrieved elsewhere. To test this, access a URL like so. The configurable arguments are the repository owner and name fields. https://api.github.com/repos/torvalds/linux/commits This is what a commit looks like: { "sha": "7638417db6d59f3c431d3e1f261cc637155684cd", "url": "https://api.github.com/repos/octocat/Hello-World/git/commits/7638417db6d59f3c431d3e1f261cc637155684cd", "author": { "date": "2008-07-09T16:13:30+12:00", "name": "Scott Chacon", "email": "[email protected]" }, "committer": { "date": "2008-07-09T16:13:30+12:00", "name": "Scott Chacon", "email": "[email protected]" }, "message": "my commit message", "tree": { "url": "https://api.github.com/repos/octocat/Hello-World/git/trees/827efc6d56897b048c772eb4087f854f46256132", "sha": "827efc6d56897b048c772eb4087f854f46256132" }, "parents": [ { "url": "https://api.github.com/repos/octocat/Hello-World/git/commits/7d1b31e74ee336d15cbd21741bc88a537ed063a0", "sha": "7d1b31e74ee336d15cbd21741bc88a537ed063a0" } ] } To make the test simple, I download these as JSON locally, then start a python webserver. Were I to make many such calls on a public site, I’d set up a proxy to the github APIs. python -m SimpleHTTPServer This data has a number of nested objects and must be flattened to fit into the lunr.jsfull-text index. This example uses the commit number (0, 1, 2..N) as the location in the index, but a real environment should use the commit hash to allow partitioning the ingestion process. Nested objects are flattened by joining subsequent keys with underscores in between. A production-worthy solution needs to escape these to prevent collisions. var documents = []; function recurse(doc_num, base, obj, value) { if ($.isPlainObject(value)) { $.each(value, function (k, v) { recurse(doc_num, base + obj + "_", k, v); }); } else { process(doc_num, base + obj, value); } } function process(doc_num, key, value) { if (documents.length <= doc_num) documents[doc_num] = {}; if (value !== null) documents[doc_num][key] = value + ''; } $.each(data, function(doc_num, commit) { $.each(commit, function(k, v) { recurse(doc_num, '', k, v) }); }); Normally, one sets up a lunr full-text index by specifying all the fields, much like Solr’s numerous XML config files. Lunr doesn’t have nearly as many configuration options, since you only specify the ‘boost’ parameter to increase the value of certain fields in ranking. I imagine this will change as the project grows, at the very least to include type hints. Given the simplicity of field objects, you can infer infer the field list from JSON payloads. The code below provides two modes, one where you inspect the entire JSON payload, or one where you limit how many commits you check, a good option when JSON data is consistent. The function accepts configuration objects resembling ExtJS config objects, which lets you override as desired. If fields derived from existing data are required, they can be inserted after any documents are inserted. function inferIndex(documents, config) { return lunr(function() { this.ref('id'); var found = {}; var idx = this; $.each(documents, function(doc_num, doc) { if (config && config.limit && config.limit < doc_num) return; $.each(doc, function(k, v) { if (!found[k]) { if (config && config[k]) { idx.field(k, config[k]); } else { idx.field(k); } found[k] = true; } }); }); }); } var index = inferIndex(documents, {limit: 1, 'commit_author_name':{boost:10}); Inserting flattened documents into the index becomes simple. The method below provides a callback, should you desire to add calculated fields fields. $.each(documents, function(doc_num, attrs, doc_cb) { var doc = $.extend( {id: doc_num}, attrs); if (doc_cb) { doc = doc_cb(doc); } index.add(doc); }); At this point we’ve indexed the entire commit history from a git repository, which lets us search for commits by topic. While this is useful, it’d be really nice to be able to facet on fields, which would return the number of documents in a category, like a SQL group by. I’ve found it particularly convenient to facet on author, date, or author’s company. If you have access to the original documents, you can easily construct facets based on the results of a lunr search: function facet(index, query, data, field) { var results = index.search(query); var facets = {}; $.each(results, function(index, searchResult) { var doc = data[searchResult.ref]; facets[doc[field]] = (facets[doc[field]] === undefined ? 0 : facets[doc[field]]) + 1; } ); return facets; } Commit messages in repositories where I work often contain names of clients who requested a feature or bug fix. Consequently doing a search faceted by author provides a list of who worked with each client the most – this can also tell you who has worked with various pieces of technology. The following query demonstrates this technique: var facets = facet(index, 'driver', documents, 'commit_author_name'); {"Wolfram Sang":24,"Linus Torvalds":3} The approach shown here works well, but requires retrieving results requires access to the original document data. If we want to filter the results to a category, we need a richer search API than lunr currently provides, as well as callback options within the search API. In Solr there are also options to skip lower-casing data, as that may be inappropriate for category titles. Mitigating these issues will be explored further in future essays.

If you’re uploading blobs by splitting blobs into blocks and you get the error – The specified blob or block content is invalid, then this post is for you. Short Version If you’re uploading blobs by splitting blobs into blocks and you get the above mentioned error, ensure that your block ids of your blocks are of same length. If the block ids of your blocks are of different length, you’ll get this error. Long Version Now for the longer version of this post . A few days back I was working with storage client library especially around uploading blobs in chunks and with one particular blob I was constantly getting the error – The specified blob or block content is invalid. I tried numerous combinations even resorting to REST API directly but to no avail. It only happened with just one blob. Furthermore if I uploaded the same blob without splitting it into blocks, all was well. I was at my wits’ end. Tried searching the Internet for this error but could not find a conclusive answer to my problem. After much trial and error, I was able to simulate the same problem on other blobs as well. Here’s how you can recreate it: Start uploading the blob by splitting it into blocks. For block id, let’s do a 7 character long string e.g. intValue.ToString(“d7”). This will ensure that my block ids would be “0000001”, “0000002”, …, ”0000010” ….. After one or two blocks are uploaded, cancel the operation. Now re-upload the blob by splitting it into blocks. However this time for block id, let’s do a 6 character long string e.g. intValue.ToString(“d6”). You’ll get the error as soon as you try to upload the 1st block. Possible Solutions Now that we know the root cause of this problem, let’s look at some of the possible solutions to solve this problem. Wait out One possible solution is to wait out. I know its lame but still a possible solution. We know that Windows Azure Blob Storage Service keeps all uncommitted blocks for a duration of 7 days and if within 7 days those uncommitted blocks are not committed, the storage service purges them. I wish storage service provided some mechanism to purge uncommitted blocks programmatically. Commit uncommitted blocks You could possibly commit the blocks which are in uncommitted state so that at least you get a blob (which would not be the blob we wanted to upload in the first place). You can then delete that blob and re-upload the blob by specifying block ids which are of same length. To fetch the list of uncommitted blocks, if you’re using REST API directly you can perform “Get Block List” operation and pass “blocklisttype=uncommitted” as one of the query string parameters. If you’re using storage client library (assuming you’re using the version 2.x of .Net storage client library), you can do something like the code below: private static List GetUncommittedBlockIds(CloudBlockBlob blob) { var sasUri = blob.GetSharedAccessSignature(new SharedAccessBlobPolicy() { SharedAccessExpiryTime = DateTime.UtcNow.AddMinutes(5), Permissions = SharedAccessBlobPermissions.Read, }); var blobUri = new Uri(string.Format("{0}{1}", blob.Uri, sasUri)); List uncommittedBlockIds = new List(); var request = BlobHttpWebRequestFactory.GetBlockList(blobUri, null, null, BlockListingFilter.Uncommitted, null, null); //request.Headers.Add("Authorization", using (var resp = (HttpWebResponse)request.GetResponse()) { using (var stream = resp.GetResponseStream()) { var getBlockListResponse = new GetBlockListResponse(stream); var blocks = getBlockListResponse.Blocks; foreach (var block in blocks.Where(b => !b.Committed)) { uncommittedBlockIds.Add(Encoding.UTF8.GetString(Convert.FromBase64String(block.Name))); } } } return uncommittedBlockIds; } A few things to keep in mind here: Microsoft.WindowsAzure.Storage.Blob namespace does not have the capability to get the list of uncommitted blocks. You would need to make use ofMicrosoft.WindowsAzure.Storage.Blob.Protocol namespace. Because we’re kind of invoking the REST API by executing an HttpWebRequest, I created a shared access signature on the blob so that I don’t have to create “Authorization” header. Fetch uncommitted blocks to see block id length You could fetch the list of uncommitted blocks just to find out the length of the block id used. You could then use that block id length for your new upload session and do the upload. Please see the code snippet above to find this information. Upload another blob with same name without splitting it into blocks You could also upload another blob with the same name without splitting it into blocks. It could very well be a zero byte blob. That way your uncommitted block list will be wiped clean. Then you could delete that dummy blob and re-upload the actual blob. A Few Words About Blocks Since we’re talking about blocks, I thought it might be useful to mention a few points about them: Blocks and block related operations are only applicable for “Block Blobs”. Duh!! You’ll get an error if you’re trying to do these operations on a “Page Blob”. For uploading large blobs, it is recommended that you split your blob into blocks. In fact if your blob size is more than 64 MB, then you have to split it into blocks. Minimum size of a block is 1 Byte and the maximum size of a block is 4 MB. It is recommended that you choose a block size based on your internet connectivity and number of parallel threads you want use to upload these blocks. A blob can be split into a maximum of 50000 blocks. It’s important to remember this limitation because you are reminded of this limit when you’re trying to upload 50001st block. The length of all the block ids must be same. So if you’re using an integer value to denote block id, you make sure that you pad that integer value with “0” so that you get same length. So you could do something likeint.ToString(“d6”). When passing the block id as a parameter, it must be Base64 encoded. While the order in which the blocks are uploaded is not important, the order is important when you commit the block list because that’s when the blob is constructed by the service. For example, let’s say you’re uploading a blob by splitting it into 5 blocks (with ids “000001”, “000002”, “000003”, “000004”, and “000005”). You could upload these blocks in any order – 000004, 000001, 000003, 000005, 000002 however when you commit the block list, ensure that the block ids are passed in proper order i.e. 000001, 000002, 000003, 000004, 000005. Summary That’s it for this post. I hope you’ve found this information useful. I spent considerable amount of time trying to fix this problem so I hope it will help some folks out. As always, if you find any issues with the post please let me know and I’ll fix it ASAP.

this is the first blog post in a series which looks at some data organization patterns in mapreduce. we’ll look at how to bucket output across multiple files in a single task, how to multiplex data across multiple files, and also how to coalesce data. these are all common patterns that are useful to have in your mapreduce toolkit. we’ll kick things off with a look at bucketing data outputs in your map or reduce tasks. by default when using a fileoutputformat-derived outputformat (such as textoutputformat), all the outputs for a reduce task (or a map task in a map-only job) are written to a single file in hdfs. imagine a situation where you have user activity logs being streamed into hdfs, and you want to write a mapreduce job to better organize the incoming data. as an example a large organization with multiple products may want to bucket the logs based on the product. to do this you’ll need the ability to write to multiple output files in a single task. let’s take a look at how we can make that happen. multipleoutputformat there are a few ways you can achieve your goal, and the first option we’ll look at is the multipleoutputformat class in hadoop. this is an abstract class that lets you do the following: define the output path for each and every key/value output record being emitted by a task. incorporate the input paths into the output directory for map-only jobs. redefine the key and value that are used to write to the underlying recordwriter . this is useful in situations where you want to remove data from the outputs as it duplicates data in the filename. for each output path, define the recordwriter that should be used to write the outputs. ok enough with the words - let’s look at some data and code. first up is the simple data we’ll use in our example - imagine you work at a fruit market with locations in multiple cities, and you have a purchase transaction stream which contains the store location along with the fruit that was purchased. cupertino apple sunnyvale banana cupertino pear to help bucket your data for future analysis, you want to bin each record into city-specific files. for the simple data set above you don’t want to filter, project or transform your data, just bucket it out, so a simple identity map-only job will do the job. to force more than one mapper, we’ll write the data to two separate files. $ tab="$(printf '\t')" $ hdfs -put - file1.txt << eof cupertino${tab}apple sunnyvale${tab}banana eof $ hdfs -put - file2.txt << eof cupertino${tab}pear eof here’s the code which will let you write city-specific output files. import org.apache.commons.lang.stringutils; import org.apache.hadoop.conf.configuration; import org.apache.hadoop.conf.configured; import org.apache.hadoop.fs.filesystem; import org.apache.hadoop.fs.path; import org.apache.hadoop.io.text; import org.apache.hadoop.mapred.*; import org.apache.hadoop.mapred.lib.identitymapper; import org.apache.hadoop.mapred.lib.multipletextoutputformat; import org.apache.hadoop.util.progressable; import org.apache.hadoop.util.tool; import org.apache.hadoop.util.toolrunner; import java.io.ioexception; import java.util.arrays; /** * an example of how to use {@link org.apache.hadoop.mapred.lib.multipleoutputformat}. */ public class mofexample extends configured implements tool { /** * create output files based on the output record's key name. */ static class keybasedmultipletextoutputformat extends multipletextoutputformat { @override protected string generatefilenameforkeyvalue(text key, text value, string name) { return key.tostring() + "/" + name; } } /** * the main job driver. */ public int run(final string[] args) throws exception { string csvinputs = stringutils.join(arrays.copyofrange(args, 0, args.length - 1), ","); path outputdir = new path(args[args.length - 1]); jobconf jobconf = new jobconf(super.getconf()); jobconf.setjarbyclass(mofexample.class); jobconf.setnumreducetasks(0); jobconf.setmapperclass(identitymapper.class); jobconf.setinputformat(keyvaluetextinputformat.class); jobconf.setoutputformat(keybasedmultipletextoutputformat.class); fileinputformat.setinputpaths(jobconf, csvinputs); fileoutputformat.setoutputpath(jobconf, outputdir); return jobclient.runjob(jobconf).issuccessful() ? 0 : 1; } /** * main entry point for the utility. * * @param args arguments * @throws exception when something goes wrong */ public static void main(final string[] args) throws exception { int res = toolrunner.run(new configuration(), new mofexample(), args); system.exit(res); } } run this code and you’ll see the following files in hdfs, where /output is the job output directory: $ hadoop fs -lsr /output /output/cupertino/part-00000 /output/cupertino/part-00001 /output/sunnyvale/part-00000 if you look at the output files you’ll see that the files contain the correct buckets. $ hadoop fs -lsr /output/cupertino/* cupertino apple cupertino pear $ hadoop fs -lsr /output/sunnyvale/* sunnyvale banana awesome, you have your data bucketed by store. now that we have everything working, let’s look at what we did to get there. we had to do two things to get this working: extend multipletextoutputformat this is where the magic happened - let’s look at that class again. static class keybasedmultipletextoutputformat extends multipletextoutputformat { @override protected string generatefilenameforkeyvalue(text key, text value, string name) { return key.tostring() + "/" + name; } } you are working with text, which is why you extended multipletextoutputformat , a class that in turn extends multipleoutputformat . multipletextoutputformat is a simple class which instructs the multipleoutputformat to use textoutputformat as the underlying output format for writing out the records. if you were to use multipleoutputformat as-is it behaves as if you were using the regular textoutputformat , which is to say that it’ll only write to a single output file. to write data to multiple files you had to extend it, as with the example above. the generatefilenameforkeyvalue method allows you to return the output path for an input record. the third argument, name , is the original fileoutputformat -created filename, which is in the form “part-nnnnn”, where “nnnnn” is the task index, to ensure uniqueness. to avoid file collisions, it’s a good idea to make sure your generated output paths are unique, and leveraging the original output file is certainly a good way of doing this. in our example we’re using the key as the directory name, and then writing to the original fileoutputformat filename within that directory. specify the outputformat the next step was easy - specify that this output format should be used for your job: jobconf.setoutputformat(keybasedmultipletextoutputformat.class); earlier we also mentioned that you can use the input path as part of the output path, which we will look at next. using the input filename as part of the output filename in map-only jobs what if we wanted to keep the input filename as part of the output filename? this only works for map-only jobs, and can be accomplished by overriding the getinputfilebasedoutputfilename method. let’s look at the following code to understand how this method fits into the overall sequence of actions that the multipleoutputformat class performs: public void write(k key, v value) throws ioexception { // get the file name based on the key string keybasedpath = generatefilenameforkeyvalue(key, value, myname); // get the file name based on the input file name string finalpath = getinputfilebasedoutputfilename(myjob, keybasedpath); // get the actual key k actualkey = generateactualkey(key, value); v actualvalue = generateactualvalue(key, value); recordwriter rw = this.recordwriters.get(finalpath); if (rw == null) { // if we don't have the record writer yet for the final path, create // one // and add it to the cache rw = getbaserecordwriter(myfs, myjob, finalpath, myprogressable); this.recordwriters.put(finalpath, rw); } rw.write(actualkey, actualvalue); }; the getinputfilebasedoutputfilename method is called with the output of generatefilenameforkeyvalue , which contains our already-customized output file. our new keybasedmultipletextoutputformat can now be updated to override getinputfilebasedoutputfilename and append the original input filename to the output filename: static class keybasedmultipletextoutputformat extends multipletextoutputformat { @override protected string generatefilenameforkeyvalue(object key, object value, string name) { return key.tostring() + "/" + name; } @override protected string getinputfilebasedoutputfilename(jobconf job, string name) { string infilename = new path(job.get("map.input.file")).getname(); return name + "-" + infilename; } if you run with your modified outputformat class you’ll see the following files in hdfs, confirming that the input filenames are now concatenated to the end of each output file. $ hadoop fs -lsr /output /output/cupertino/part-00000-file1.txt /output/cupertino/part-00001-file2.txt /output/sunnyvale/part-00000-file1.txt the implementation of getinputfilebasedoutputfilename in multipleoutputformat doesn’t do anything interesting by default, but if you set the value of the mapred.outputformat.numoftrailinglegs configurable to an integer greater than 0, then the getinputfilebasedoutputfilename will use part of the input path as the output path. let’s see what happens when we set the value to 1: jobconf.setint("mapred.outputformat.numoftrailinglegs", 1); the output files in hdfs now exactly mirror the input files used for the job: $ hadoop fs -lsr /output /output/file1.txt /output/file2.txt if we set mapred.outputformat.numoftrailinglegs to 2, and our input files exist in the /inputs directory, then our output directory looks like this: $ hadoop fs -lsr /output /output/input/file1.txt /output/input/file2.txt basically as you keep incrementing mapred.outputformat.numoftrailinglegs , then multipleoutputformat will continue to go up the parent directories of the input file and use them in the output path. modifying the output key and value it’s very possible that the actual key and value you want to emit are different from those that were used to determine the output file. in our example, we took the output key and wrote to a directory using the key name. if you do that keeping the key in the output file may be redundant. how would we modify the output record so that the key isn’t written? multipleoutputformat has your back with the generateactualkey method. class keybasedmultipletextoutputformat extends multipletextoutputformat { @override protected string generatefilenameforkeyvalue(text key, text value, string name) { return key.tostring() + "/" + name; } @override protected text generateactualkey(text key, text value) { return null; } } the returned value from this method replaces the key that’s supplied to the underlying recordwriter , so if you return null as in the above example, no key will be written to the file. $ hadoop fs -lsr /output/cupertino/* apple pear $ hadoop fs -lsr /output/sunnyvale/* banana you can achieve the same result for the output value by overriding the generateactualvalue method. changing the recordwriter in our final step we’ll look at how you can leverage multiple recordwriter classes for different output files. this is accomplished by overriding the getrecordwriter method. in the example below we’re leveraging the same textoutputformat for all the files, but it gives you a sense of what can be accomplished. static class keybasedmultipletextoutputformat extends multipletextoutputformat { @override protected string generatefilenameforkeyvalue(text key, text value, string name) { return key.tostring() + "/" + name; } @override public recordwriter getrecordwriter(filesystem fs, jobconf job, string name, progressable prog) throws ioexception { if (name.startswith("apple")) { return new textoutputformat().getrecordwriter(fs, job, name, prog); } else if (name.startswith("banana")) { return new textoutputformat().getrecordwriter(fs, job, name, prog); } return super.getrecordwriter(fs, job, name, prog); } } conclusion when using multipleoutputformat , give some thought to the number of distinct files that each reducer will create. it would be prudent to plan your bucketing so that you have a relatively small number of files. in this post we extended multipletextoutputformat , which is a simple extension of multipleoutputformat that supports text outputs. multiplesequencefileoutputformat also exists to support sequencefiles in a similar fashion. so what are the shortcomings with the multipleoutputformat class? if you have a job that uses both map and reduce phases, then multipleoutputformat can’t be used in the map-side to write outputs. of course, multipleoutputformat works fine in map-only jobs. all recordwriter classes must support exactly the same output record types. for example, you wouldn’t be able to support a recordwriter that emitted for one output file, and have another recordwriter that emitted . multipleoutputformat exists in the mapred package, so it won’t work with a job that requires use of the mapreduce package. all is not lost if you bump into either one of these issues, as you’ll discover in the next blog post.

When building websites, you’ll often want users to be able to search for something by name. On LinerNotes, users can search for bands, albums, genres etc from a search bar that appears on the homepage and in the omnipresent nav bar. And we need a way to match those queries to entities in our Postgres database. At first, this might seem like a simple problem with a simple solution, especially if you’re using the ORM; just jam the user input into an ORM filter and retrieve every matching string. But there’s a problem: if you do Bands.objects.filter(name="beatles") You’ll probably get nothing back, because the name column in your “bands” table probably says “The Beatles” and as far as Postgres is concerned if it’s not exactly the same string, it’s not a match. Users are naturally terrible at spelling, and even if they weren’t they’d be bad at guessing exactly how the name is formatted in your database. Of course you can use the LIKE keyword in SQL (or the equivalent ‘__contains’ suffix in the ORM) to give yourself a little flexibility and make sure that “Beatles” returns “The Beatles”. But 1) the LIKE keyword requires you to evaluate a regex against every row in your table, or hope that you’ve configured your indices to support LIKE (a quick Google doesn’t tell me whether Django does that by default in the ORM) and 2) what if the user types “Beetles”? Well, then you’ve got a bit of a problem. No matter how obvious it is to human you that “beatles” is close to “beetles”[1], to the computer they’re just two non-identical byte sequences. If you want the computer to understand them as similar you’re going to have to give it a metric for similarity and a method to make the comparison. There are a few ways to do that. You can do what I did initially and whip out the power tools, i.e. a dedicated search system like Solr or ElasticSearch. These guys have notions of fuzziness built right in (Solr more automatically than ES). But they’re designed for full-text indexing of documents (e.g. full web pages) and they’re rather complex to set up and administer. ES has been enough of a hassle to keep running smoothly that I took the time to see if I could push the search workload to Postgres, and hence this article. Unless you need to do something real fancy, it’s probably overkill to use them for just matching names. Instead, we’re going to follow Starr Horne’s advice and use a Postgres EXTENSION that lets us build fuzziness into our query in a fast and fairly simple way. Specifically, we’re going to use an extension called pg_trgm (i.e. “Postgres Trigram”) which gives Postgres a “similarity” function that can evaluate how many three-character subsequences (i.e. “trigrams”) two strings share. This is actually a pretty good metric for fuzzy matching short strings like names. To use pg_trgm, you’ll need to install the “Postgres Contrib” package. On ubuntu: sudo apt-get install postgres-contrib **WARNING: THIS WILL TRY TO RESTART YOUR DATABASE** then pop open psql and install pg_trgm (NB: this only works on Postgres 9.1+; Google for the instructions if you’re on a lower version.) psql CREATE EXTENSION pg_trgm; \dx # to check it's installed Now you can do SELECT * FROM types_and_labels_view WHERE label %'Mountain Goats' ORDER BY similarity(label,'Mountain Goats') DESC LIMIT 100; And out will pop the 100 most similar names. This will still take a long time if your table is large, but we can improve that with a special type of index provided by pg_trgm: CREATE INDEX labels_trigram_index ON types_and_labels_table USING gist (label gist_trgm_ops); or CREATE INDEX labels_trigram_index ON types_and_labels_table USING gin (label gin_trgm_ops); (GIN is slower than GIST to build, but answers queries faster. That’ll take a while to build (possibly quite a while), but once it does you should be able to fuzzy search with ease and speed. If you’re using Django, you will have to drop into writing SQL to use this (until someone, maybe you, writes a Django extension to do this in the ORM.) And as a frustrating finishing note, my attempt to implement this on LinerNotes was not ultimately succesful. It seems that that index query performance is at least O(n) and with 50 million entities in my database queries take at least 10 seconds. I’ve read that performance is great up to about 100k records then drops off sharply from there. There are some apparently additional options for improving query performance, but I’ll be sticking with ElasticSearch for now. [1] Sorry, Googlebot! Not sorry, Bingbot.

There are several advantages in utilizing the features of Interfaces in general programming. As you may already know, interfaces define a set of functionality as a rule or a contract. When you implement an interface all of these functionality must be implemented in the concrete class. The ideas discussed here can be applied using any programming language that supports Interfaces. Java has been used in examples for simplicity. In general when one writes a simple program, one may not think about the need of using an Interface. But when you are building a larger system or a library which keeps evolving, it is a good idea to use Interface. A particular advantage of using interface in Java is that it allows multiple inheritance. The full power of Interface is utilized when dependency injection techniques is used to inject required implementation on run time. Using references to interfaces instead of their concrete implementation classes helps in minimizing ripple effects, as the user of an interface reference doesn't have to worry about the changes in the underlying concrete implementation. To give an example consider our application has a layered architecture with Controller layer on top then Service layer, Repository layer and domain layer in that order. Here controller layer interacts with service layer and service layer interacts with repository layer. Each layer is aware of the services provided by the layer below it. We expose this information using Interface and hide the concrete implementation. Let us say we have a UserRegistrationService, which registers a new user in our system. Our requirement is to save the user information in a relational database and also in xml files. If we were not using interface, the UserRegistrationService may be implemented with two functions saveToXML() and saveToDatabase(). public class UserRegistrationService { public void saveToXML(UserInfo userInfo) { //save to xml using service exposed by Repository layer } public void saveToDatabase(UserInfo userInfo) { //save to db using service exposed by Repository layer } } In this case, the UserRegistrationController should be aware of the concrete implementation of these two functions in UserRegistrationService to use them. //Controller becomes complicated when additional features are required public class UserRegistrationController { //Controller should be aware of the implementation when no Interface is used UserRegistrationService userRegistrationService = new UserRegistrationService(); public void processRequest(UserInfo userInfo) { this.saveToXml(userInfo); } private void saveToXml(UserInfo userInfo) { userRegistrationService.saveToXml(userInfo); } private void saveToDatabase(UserInfo userInfo) { userRegistrationService.saveToDatabase(); } } If an additional functionality to save the information as JSON is required then you will have to add a new function saveToJson() in the Service class as well as make changes in the Controller. This adds lots of complication to maintenance of our huge application with hundreds of controllers and services. When using interface this becomes much simpler. We define our UserRegistrationService like this: public interface UserRegistrationService { public void save(); } The controller layer is only aware of this interface, which has a save method. Let us say we have two implementations like the following: public class UserRegistraionServiceXmlImpl implements UserRegistrationService { @Override public void save(UserInfo userInfo) { //save to xml using service exposed by Repository layer } } public class UserRegistraionServiceRelDbImpl implements UserRegistrationService { @Override public void save(UserInfo userInfo) { //save to relational db using service exposed by Repository layer } } Here we can choose any of these two implementation on run time using dependency injection. One may use @Inject or @Resource annotation to mark an implementation is to be injected. If using Spring, one may also use Xml bean definitions. //Controller becomes much simpler when using Interfaces in the service layer public class UserRegistrationController { @Resource(name = "userRegistrationServiceXmlImpl") UserRegistrationService userRegistrationService; public void processRequest(UserInfo userInfo) { userRegistrationService.save(userInfo); } public void setUserRegistrationService(UserRegistrationService userRegistrationService) { this.userRegistrationService = userRegistrationService; } } Now when we need to add the additional functionality of saving to Json, we just add another implementation and select that implementation to be injected. (You may use an IOC{Inversion Of Control} container like Spring to achieve this seamlessly.) public class UserRegistraionServiceJsonImpl implements UserRegistrationService { @Override public void save() { //save to json using service exposed by Repository layer } } This highly reduces the software modification and extension cost. As changes in one layer does not effect other layer and new functionalities are made available to other layer immediately. Thus using interface gives you more power over extending and maintaining your application, utilize abstraction and implement good software development practices. This article was originally published to http://tech.deepumohan.com/2013/01/when-to-use-interfaces-in-your.html

Let's say you have a NetBeans Platform application that puts a premium on vertical space. Maybe a Heads Up Display on a Touch Screen? Wouldn't it be great to have the menu slide out from the edge of the screen only when you need it? Well the NetBeans Platform provides slide-in TopComponents, of course, but a JMenu just isn't going to work out so well inside one. We can use JavaFX as part of the solution as it provides some capabilities that the base Swing components available in the NetBeans Platform do not. Let's say we take all of our root MenuBar items and place them within an Accordion type pane. Each collapsible TitledPane of the Accordion control could then contain the sub-menu items, maybe represented by a JavaFX MenuButton. This would allow for a recursive Menu like effect but the overall container could be placed anywhere. Something like the screenshot below: What we see here is the described effect sliding out and overlayed on top of the Favorites tab. I sprinkled in some transparency for good measure. Notice how we are able to completely eliminate the Menu Bar and Tool Bar gaining potentially valuable real estate? The rest of this tutorial will explain the steps necessary to achieve something like this. That article was written by Geertjan Wielenga and it will become clear that much of the base code to accomplish this article was extended from Geertjan's example. Thanks again Geertjan! Similar articles to this that may be helpful are below: https://dzone.com/articles/javafx-fxml-meets-netbeans https://dzone.com/articles/how-embed-javafx-chart-visual All these articles are loosely coupled in a tutorial arc towards explaining and demonstrating the advantages of integrating JavaFX into the NetBeans Platform. The following two steps are borrowed exactly as found from Geertjan's tutorial: Step 1. Remove the default menubar and replace with your own: import org.openide.modules.ModuleInstall; public class Installer extends ModuleInstall { @Override public void restored() { System.setProperty("netbeans.winsys.menu_bar.path", "LookAndFeel/MenuBar.instance"); } } Step 2: In the layer.xml file define your Swing replacement menubar. I have also taken the liberty to hide the Toolbars as well. Now why are we replacing the old MenuBar with a new MenuBar if we intend to hide it? Well if you hide the MenuBar via the layer.xml as I did the Toolbars the filesystem folder tree will not be instantiated. That means we won't be able to dynamically determine the Menu Folder tree to rebuild our custom AccordionMenu. The solution? Make an empty Menubar. package polaris.javafxwizard.jfxmenu; import javax.swing.JMenuBar; /** * * @author SPhillips (King of Australia) */ public class HiddenMenuBar extends JMenuBar { public HiddenMenuBar() { super(); } } Step 3: Build an "AccordionMenu" using JavaFX This is where the tutorials diverge and this process gets a bit more complicated. Our task is to use the JavaFX/Swing Interop pattern to create a component that extends JFXPanel yet can give the user access to all the items that were once in the Menu Bar. The basic algorithm is as such: Create a component that extends JFXPanel Implement the standard Platform.runLater() pattern for creating a JavaFX scene Loop through each top level file object in the Menu folder of the application file system: Create a JavaFX Flow Pane for each file object Recursively create JavaFX ButtonMenu items for submenus Add ButtonMenu items to FlowPanes Add FlowPane to JavaFX TitledPane Add TitledPane to JavaFX Accordion component Add Accordion to scene So instead of Menus and SubMenus, we are using MenuButtons which can be recursively added to other MenuButtons and MenuItems. The Accordion control gives us a space saving collapsible view with some nice animation. The FlowPane makes it easy to layout the MenuButtons horizontally in a way that maximizes space. Below is the code for my AccordionMenu class. You will see where I borrowed heavily from Geertjan's example: polaris.javafxwizard.jfxmenu; import java.lang.reflect.InvocationTargetException; import java.util.ArrayList; import java.util.Arrays; import java.util.List; import javafx.application.Platform; import javafx.embed.swing.JFXPanel; import javafx.event.ActionEvent; import javafx.event.EventHandler; import javafx.geometry.Orientation; import javafx.scene.Group; import javafx.scene.Scene; import javafx.scene.control.Accordion; import javafx.scene.control.Button; import javafx.scene.control.MenuButton; import javafx.scene.control.MenuItem; import javafx.scene.control.TitledPane; import javafx.scene.effect.DropShadow; import javafx.scene.layout.FlowPane; import javafx.scene.paint.Color; import javax.swing.Action; import javax.swing.SwingUtilities; import org.openide.awt.Actions; import org.openide.filesystems.FileObject; import org.openide.filesystems.FileUtil; import org.openide.loaders.DataFolder; import org.openide.loaders.DataObject; import org.openide.util.Exceptions; /** * * @author SPhillips (King of Australia) */ public class AccordionMenu extends JFXPanel{ public Accordion accordionPane; public String transparentCSS = "-fx-background-color: rgba(0,100,100,0.1);"; public AccordionMenu() { super(); // create JavaFX scene Platform.setImplicitExit(false); Platform.runLater(new Runnable() { @Override public void run() { createScene(); //Standard Swing Interop Pattern } }); } private void createScene() { FileObject menuFolder = FileUtil.getConfigFile("Menu"); FileObject[] menuKids = menuFolder.getChildren(); //for each Menu folder need to create a TilePane and add it to an Accordion List titledPaneList = new ArrayList<>(); for (FileObject menuKid : FileUtil.getOrder(Arrays.asList(menuKids), true)) { //Build a Flow pane based on menu children //TOP level menu items should all be flow panes FlowPane flowPane = buildFlowPane(menuKid); flowPane.setStyle(transparentCSS); TitledPane newTitledPaneFromFileObject = new TitledPane(menuKid.getName(), flowPane); newTitledPaneFromFileObject.setAnimated(true); newTitledPaneFromFileObject.autosize(); newTitledPaneFromFileObject.setStyle(transparentCSS); titledPaneList.add(newTitledPaneFromFileObject); } Group g = new Group(); Scene scene = new Scene(g, 400, 400,new Color(0.0,0.0,0.0,0.0)); scene.setFill(null); g.setStyle(transparentCSS); accordionPane = new Accordion(); accordionPane.setStyle(transparentCSS); accordionPane.getPanes().addAll(titledPaneList); g.getChildren().add(accordionPane); setScene(scene); validate(); this.setOpaque(true); this.setBackground(new java.awt.Color(0.0f, 0.0f, 0.0f, 0.0f)); } private FlowPane buildFlowPane(FileObject fo) { //FlowPanes are made up of Buttons and MenuButtons built from actions and sub menus FlowPane flowPane = new FlowPane(Orientation.HORIZONTAL,5,5); flowPane.setStyle(transparentCSS); //If anything at the Flow Pane level is an action we need to add it as a button //otherwise we can recursively build it as a MenuButton DataFolder df = DataFolder.findFolder(fo); DataObject[] childs = df.getChildren(); for (DataObject oneChild : childs) { //If child is folder we need to build recursively if (oneChild.getPrimaryFile().isFolder()) { FileObject childFo = oneChild.getPrimaryFile(); MenuButton newMenuButton = new MenuButton(childFo.getName()); buildMenuButton(childFo, newMenuButton); flowPane.getChildren().add(newMenuButton); } else { Object instanceObj = FileUtil.getConfigObject(oneChild.getPrimaryFile().getPath(), Object.class); if (instanceObj instanceof Action) { //If it is an Action we have reached an endpoint final Action a = (Action) instanceObj; String name = (String) a.getValue(Action.NAME); String cutAmpersand = Actions.cutAmpersand(name); Button buttonItem = new Button(cutAmpersand); MenuEventHandler meh = new MenuEventHandler(a); buttonItem.setOnAction(meh); buttonItem.setEffect(new DropShadow()); flowPane.getChildren().add(buttonItem); } } } return flowPane; } private void buildMenuButton(FileObject fo, MenuButton menuButton) { DataFolder df = DataFolder.findFolder(fo); DataObject[] childs = df.getChildren(); for (DataObject oneChild : childs) { //If child is folder we need to build recursively if (oneChild.getPrimaryFile().isFolder()) { FileObject childFo = oneChild.getPrimaryFile(); //Menu newMenu = new Menu(childFo.getName()); MenuButton newMenuButton = new MenuButton(childFo.getName()); //menu.getItems().add(newMenu); buildMenuButton(childFo, newMenuButton); } else { Object instanceObj = FileUtil.getConfigObject(oneChild.getPrimaryFile().getPath(), Object.class); if (instanceObj instanceof Action) { //If it is an Action we have reached an endpoint final Action a = (Action) instanceObj; String name = (String) a.getValue(Action.NAME); String cutAmpersand = Actions.cutAmpersand(name); MenuItem menuItem = new MenuItem(cutAmpersand); MenuEventHandler meh = new MenuEventHandler(a); menuItem.setOnAction(meh); menuButton.getItems().add(menuItem); } } } } private class MenuEventHandler implements EventHandler { public Action theAction; public MenuEventHandler(Action action) { super(); theAction = action; } @Override public void handle(final ActionEvent t) { try { SwingUtilities.invokeAndWait(new Runnable() { @Override public void run() { java.awt.event.ActionEvent event = new java.awt.event.ActionEvent( t.getSource(), t.hashCode(), t.toString()); theAction.actionPerformed(event); } }); } catch ( InterruptedException | InvocationTargetException ex) { Exceptions.printStackTrace(ex); } } } } I took the liberty of placing a few CSS stylings here and there, trying to play with the transparency. Also I found that it looked better if a JavaFX Button was used for any Actions found at the very top level, instead of a MenuButton with a single item. Step 4: Build a Slide in TopComponent for the new AccordionMenu Now that you have a JFXPanel Swing Interop component, your NetBeans Platform TopComponent doesn't need to know about JavaFX. However in this scenario the Platform also is contributing via its wonderful docking framework. Use the Window wizard and select Left Sliding In as a mode. I would also advise making this component not closable, otherwise the user could lose the ability to use the menu. Here are the annotations and constructor code in my TopComponent: @ConvertAsProperties( dtd = "-//polaris.javafxwizard.jfxmenu//SlidingAccordion//EN", autostore = false) @TopComponent.Description( preferredID = "SlidingAccordionTopComponent", iconBase="polaris/javafxwizard/jfxmenu/categories.png", persistenceType = TopComponent.PERSISTENCE_ALWAYS) @TopComponent.Registration(mode = "leftSlidingSide", openAtStartup = true) @ActionID(category = "Window", id = "polaris.javafxwizard.jfxmenu.SlidingAccordionTopComponent") @ActionReference(path = "Menu/JavaFX" /*, position = 333 */) @TopComponent.OpenActionRegistration( displayName = "#CTL_SlidingAccordionAction", preferredID = "SlidingAccordionTopComponent") @Messages({ "CTL_SlidingAccordionAction=SlidingAccordion", "CTL_SlidingAccordionTopComponent=SlidingAccordion Window", "HINT_SlidingAccordionTopComponent=This is a SlidingAccordion window" }) public final class SlidingAccordionTopComponent extends TopComponent { public AccordionMenu accordionMenu; public SlidingAccordionTopComponent() { initComponents(); setName(Bundle.CTL_SlidingAccordionTopComponent()); setToolTipText(Bundle.HINT_SlidingAccordionTopComponent()); putClientProperty(TopComponent.PROP_CLOSING_DISABLED, Boolean.TRUE); putClientProperty(TopComponent.PROP_DRAGGING_DISABLED, Boolean.TRUE); putClientProperty(TopComponent.PROP_MAXIMIZATION_DISABLED, Boolean.TRUE); putClientProperty(TopComponent.PROP_UNDOCKING_DISABLED, Boolean.TRUE); putClientProperty(TopComponent.PROP_KEEP_PREFERRED_SIZE_WHEN_SLIDED_IN, Boolean.TRUE); setLayout(new BorderLayout()); //Standard JFXPanel Swing Interop Pattern accordionMenu = new AccordionMenu(); //transparency Color transparent = new Color(0.0f, 0.0f, 0.0f, 0.0f); accordionMenu.setOpaque(true); accordionMenu.setBackground(transparent); this.add(accordionMenu); this.setOpaque(true); this.setBackground(transparent); } Step 5. See how great it looks We now have a slide out collapsible application menu provided by JavaFX components. These components can be "skinned" using CSS stylings and as such the menu can be crafted differently for different applications. (By the way if anyone reading this has some ideas please contact me because I am not a CSS guy at all) Best of all we have adapted our application to work nicely with a Heads Up Display or Kiosk view that typically run on touchscreen computers. This is because we have saved real estate and implemented an interface that is more condusive to single touches versus mouse drag events. Hey let's see how it might look with an application that needs all the space it can get?

The Reportlab library is a great way to generate PDFs in Python. Recently, I noticed that it has the ability to do barcodes. I had heard about it being able to generate QR codes, but I hadn’t really dug under the covers to see what else it could do. In this tutorial, we’ll take a look at some of the barcodes that Reportlab can generate. If you don’t already have Reportlab, go to their website and get it before jumping into the article. Reportlab’s Barcode Library Reportlab provides for several different types of bar codes: code39 (i.e. code 3 of 9), code93, code 128, EANBC, QR, and USPS. I saw one called “fourstate” as well, but I couldn’t figure out how to get it to work. Underneath some of these types, there are sub-types such as Standard, Extended or MultiWidth. I didn’t have much luck getting the MultiWidth one to work for the code128 bar code as it kept giving me an attribute error, so we’ll just ignore that one. If you know how to do it, ping me in the comments or via my contact form and let me know. I’ll update the article if anyone can show me how to add that or the fourstate barcode. Anyway, the best way to learn is to just write some code. Here’s a pretty straight-forward example: from reportlab.graphics.barcode import code39, code128, code93 from reportlab.graphics.barcode import eanbc, qr, usps from reportlab.graphics.shapes import Drawing from reportlab.lib.pagesizes import letter from reportlab.lib.units import mm from reportlab.pdfgen import canvas from reportlab.graphics import renderPDF #---------------------------------------------------------------------- def createBarCodes(): """ Create barcode examples and embed in a PDF """ c = canvas.Canvas("barcodes.pdf", pagesize=letter) barcode_value = "1234567890" barcode39 = code39.Extended39(barcode_value) barcode39Std = code39.Standard39(barcode_value, barHeight=20, stop=1) # code93 also has an Extended and MultiWidth version barcode93 = code93.Standard93(barcode_value) barcode128 = code128.Code128(barcode_value) # the multiwidth barcode appears to be broken #barcode128Multi = code128.MultiWidthBarcode(barcode_value) barcode_usps = usps.POSTNET("50158-9999") codes = [barcode39, barcode39Std, barcode93, barcode128, barcode_usps] x = 1 * mm y = 285 * mm x1 = 6.4 * mm for code in codes: code.drawOn(c, x, y) y = y - 15 * mm # draw the eanbc8 code barcode_eanbc8 = eanbc.Ean8BarcodeWidget(barcode_value) bounds = barcode_eanbc8.getBounds() width = bounds[2] - bounds[0] height = bounds[3] - bounds[1] d = Drawing(50, 10) d.add(barcode_eanbc8) renderPDF.draw(d, c, 15, 555) # draw the eanbc13 code barcode_eanbc13 = eanbc.Ean13BarcodeWidget(barcode_value) bounds = barcode_eanbc13.getBounds() width = bounds[2] - bounds[0] height = bounds[3] - bounds[1] d = Drawing(50, 10) d.add(barcode_eanbc13) renderPDF.draw(d, c, 15, 465) # draw a QR code qr_code = qr.QrCodeWidget('www.mousevspython.com') bounds = qr_code.getBounds() width = bounds[2] - bounds[0] height = bounds[3] - bounds[1] d = Drawing(45, 45, transform=[45./width,0,0,45./height,0,0]) d.add(qr_code) renderPDF.draw(d, c, 15, 405) c.save() if __name__ == "__main__": createBarCodes() Let’s break this down a bit. The code39.Extended39 doesn’t really accept much beyond the value itself. On the other hand, code39.Standard39, code93.Standard93 and code128.Code128 all have basically the same API. You can change the barWidth, barHeight, turn on the start/stop symbols and add “quiet” zones. The usps bar code module provides two types of bar code: FIM and POSTNET. FIM or Facing ID Marks only encodes one letter (A-D) which I personally didn’t find it very interesting. So I just show the POSTNET version which should be pretty familiar to people in the United States as it appears on the bottom of most envelopes. POSTNET encodes the zip code! The next three bar codes use a different API to draw them on the PDF that I discovered viaStackOverflow. Basically you create a Drawing object of a certain size and then add the bar code to the drawing. Finally you use the renderPDF module to place the drawing on the PDF. It’s pretty convoluted, but it works pretty well. The EANBC codes are ones you’ll see on some manufactured products, such as tissue boxes. If you’d like to see the result of the code above, you can download the PDF here. Wrapping Up At this point you should be able to go forth and create your own bar codes in your PDFs. Reportlab is pretty handy and I hope you’ll find this additional tool helpful in your endeavors. Additional Reading A step by step Reportlab tutoral Reportlab: Mixing Fixed Content and Flowables Reportlab Tables – Creating Tables in PDFs with Python Creating QR Codes with Python StackOverflow question on Python barcode generation StackOverflow question on reportlab, QR codes and django Get the Source! barcodes.tar

How to get visibility into an important component of any complex system: the messaging queue Here at AppNeta, we get to see a lot about how people build their web applications. From simple PHP scripts to heavily service-oriented Java clouds to monolithic Django apps, everybody’s product is architected a little differently. We’re still out to trace everything, and today I want to talk how to get visibility into an important component of any complex system: the messaging queue. Specifically, let’s look at how to trace a job from Rails using Resque. Messaging Queues If you haven’t used a messaging queue in your app, the idea is simple. Instead of forcing all the work to happen during the request, while the user is waiting, you can delay some of the more time-consuming tasks. You can do anything in these tasks, ranging from a simple insert to kicking off a series of user analytics that touch all parts of your infrastructure. The advantage is that you can return a speedy response to the user, or, if they are actually waiting on the task results, give them a better loading interface than a white screen and browser loading bar. A Quick Resque Tutorial In Ruby, Resque is a task runner, which by default stores the task descriptions in Redis (though other options are available). Resque jobs are just Ruby classes, with a single mandatory method perform. Resque will call perform with the arguments given in the task description. Let’s look at a minimal task, that takes a single argument and prints it. (Useless, I know.) The @queue variable defines a name that a worker can bind to, in case you want to spread different types of jobs across different machines. To create a task that this worker could run, we just call it from our request: And that’s our job! Maybe not the most interesting job, and probably not prone to performance issues, but we don’t know that yet. So let’s measure it! Tracing a Resque Task Now that we’ve added this to our system, we should have monitoring around it. The easiest way to do this would be to just measure the time each task takes, and log that information: Unfortunately, the data presentation here leaves a bit to be desired, so I’m going to use TraceView to log this information instead. This also has the benefit of logging any SQL queries, cache accesses, or service calls that we might do in a more complex task, as well as reporting errors. To start a trace fresh, we can wrap this call in the start_trace block: That’s a start! We’ve now got some visibility into our Resque jobs, and we can rest easy knowing that this is running smoothly in production. Tracing a Resque Job (with multiple tasks!) For cron-style jobs, the approach of tracing each task individually works fine. For reference, let’s look at the events we’re generating with that code: Pretty straightforward. Now let’s consider a more complicated set of tasks: a document-processing pipeline. That code might look like this: In this case, our first task takes a document, and the second one archives it. If we have multiple tasks, each one gets logged separately, and we can figure out same statistics for each — average, std. dev., percentiles, and the like. But what if you have a job that spans multiple tasks? We can further aggregate the stats, but we might be starting to miss things, like large inputs that cause the entire pipeline to slow down. What we’d really like is to correlate the related tasks, so instead of timing the each task, we’re timing the entire job. Under the hood, TraceView generates a token for each request. If we pass this ID (generally stored in xtrace, after the X-Trace header it’s passed around in) to each task, we can correlate those timings before storing them, and retrieve them all together. To do this, we can modify each task to take this token, and trace using that ID. ProcessDoc then becomes: Now we need to start the trace somewhere, but we’re not doing it in the job. We could start it in the first task, or we could link this one step further up the chain and tie it back to the web request that started it in the first place. In a default rails stack, that request generates the following events: To add in the task queue call to the logged request, we can call the following function: We have to force a fork in the execution path to indicate that we’re running an asynchronous task, possibly in parallel, with the rest of the web request, which is done with the call to fromString. Aside from that, this is the same underlying call as is done by start_trace above — log that we’re entering a named block of code, and start timing it. When we put it all together, we get a secondary execution path attached to the web request, and the logged events look like this: Now we’ve got everything: the original request, all tasks, individual timing information, and a global view of how the process performed. Not that we now have an additional timing measurement here: the delay before starting the task at all. In this case, we waited a full 500ms between queuing the job actually executing it! Once we were in the pipeline, the tasks happened much faster (only 25ms between processing and archiving). Caveats Lest you think that everything was easy, there’s a couple things to keep in mind when you use this in your own application. Because we’re starting the timing in the web request and ending it in a task queue, we’re relying on those two processes to have an identical clock. If they’re on the same machine, it won’t be a problem, but on different machines, any clock skew will effect the timing. I’ve quietly assumed everything in this system is reliable, which is almost certainly wrong. Whatever your error handling is, make sure you always log the exit event for ‘job’, or you may never know that you have errors! As long as I haven’t totally dissuaded you from trying this out, all the code is available in one place in this gist, and you can try in out in your application today with our free version of TraceView! (Source) Related Articles Ruby 2.0 Released: Let The Tracing Begin! AppNeta Rubygems Verified Relieve Event Binding Aches in Backbone.js

After thoroughly exploring CompletableFuture API in Java 8 we are prepared to write a simplistic web crawler. We solved similar problem already using ExecutorCompletionService, Guava ListenableFuture and Scala/Akka. I choose the same problem so that it's easy to compare approaches and implementation techniques. First we shall define a simple, blocking method to download the contents of a single URL private String downloadSite(final String site) { try { log.debug("Downloading {}", site); final String res = IOUtils.toString(new URL("http://" + site), UTF_8); log.debug("Done {}", site); return res; } catch (IOException e) { throw Throwables.propagate(e); } } Nothing fancy. This method will be later invoked for different sites inside thread pool. Another method parses the Stringinto an XML Document (let me leave out the implementation, no one wants to look at it): private Document parse(String xml) //... Finally the core of our algorithm, function computing relevance of each website taking Document as input. Just as above we don't care about the implementation, only the signature is important: private CompletableFuture calculateRelevance(Document doc) //... Let's put all the pieces together. Having a list of websites our crawler shall start downloading the contents of each web site asynchronously and concurrently. Then each downloaded HTML string will be parsed to XML Document and laterrelevance will be computed. As a last step we take all computed relevance metrics and find the biggest one. This sounds pretty straightforward to the moment when you realize that both downloading content and computing relevance is asynchronous (returns CompletableFuture) and we definitely don't want to block or busy wait. Here is the first piece: ExecutorService executor = Executors.newFixedThreadPool(4); List topSites = Arrays.asList( "www.google.com", "www.youtube.com", "www.yahoo.com", "www.msn.com" ); List> relevanceFutures = topSites.stream(). map(site -> CompletableFuture.supplyAsync(() -> downloadSite(site), executor)). map(contentFuture -> contentFuture.thenApply(this::parse)). map(docFuture -> docFuture.thenCompose(this::calculateRelevance)). collect(Collectors.>toList()); There is actually a lot going on here. Defining thread pool and sites to crawl is obvious. But there is this chained expression computing relevanceFutures. The sequence of map() and collect() in the end is quite descriptive. Starting from a list of web sites we transform each site (String) into CompletableFuture by submitting asynchronous task (downloadSite()) into thread pool. So we have a list of CompletableFuture. We continue transforming it, this time applying parse() method on each of them. Remember that thenApply() will invoke supplied lambda when underlying future completes and returnsCompletableFuture immediately. Third and last transformation step composes eachCompletableFuture in the input list with calculateRelevance(). Note that calculateRelevance()returns CompletableFuture instead of Double, thus we use thenCompose() rather than thenApply(). After that many stages we finally collect() a list of CompletableFuture. Now we would like to run some computations on all results. We have a list of futures and we would like to know when all of them (last one) complete. Of course we can register completion callback on each future and use CountDownLatch to block until all callbacks are invoked. I am too lazy for that, let us utilize existing CompletableFuture.allOf(). Unfortunately it has two minor drawbacks - takes vararg instead of Collection and doesn't return a future of aggregated results but Void instead. By aggregated results I mean: if we provide List> such method should return CompletableFuture>, not CompletableFuture! Luckily it's easy to fix with a bit of glue code: private static CompletableFuture> sequence(List> futures) { CompletableFuture allDoneFuture = CompletableFuture.allOf(futures.toArray(new CompletableFuture[futures.size()])); return allDoneFuture.thenApply(v -> futures.stream(). map(future -> future.join()). collect(Collectors.toList()) ); } Watch carefully sequence() argument and return types. The implementation is surprisingly simple, the trick is to use existing allOf() but when allDoneFuture completes (which means all underlying futures are done), simply iterate over all futures and join() (blocking wait) on each. However this call is guaranteed not to block because by now all futures completed! Equipped with such utility method we can finally complete our task: CompletableFuture> allDone = sequence(relevanceFutures); CompletableFuture maxRelevance = allDone.thenApply(relevances -> relevances.stream(). mapToDouble(Double::valueOf). max() ); This one is easy - when allDone completes, apply our function that counts maximal relevance in whole set.maxRelevance is still a future. By the time your JVM reaches this line, probably none of the websites are yet downloaded. But we encapsulated business logic on top of futures, stacking them in an event-driven manner. Code remains readable (version without lambda and with ordinary Futures would be at least twice as long) but avoids blocking main thread. Of course allDone can as well be an intermediate step, we can further transform it, not really having the result yet. Shortcomings CompletableFuture in Java 8 is a huge step forward. From tiny, thin abstraction over asynchronous task to full-blown, functional, feature rich utility. However after few days of playing with it I found few minor disadvantages: CompletableFuture.allOf() returning CompletableFuture discussed earlier. I think it's fair to say that if I pass a collection of futures and want to wait for all of them, I would also like to extract the results when they arrive easily. It's even worse with CompletableFuture.anyOf(). If I am waiting for any of the futures to complete, I can't imagine passing futures of different types, say CompletableFuture andCompletableFuture. If I don't care which one completes first, how am I suppose to handle return type? Typically you will pass a collection of homogeneous futures (e.g. CompletableFuture) and thenanyOf() can simply return future of that type (instead of CompletableFuture again). Mixing settable and listenable abstractions. In Guava there is ListenableFuture and SettableFuture extending it. ListenableFuture allows registering callbacks while SettableFuture adds possibility to set value of the future (resolve it) from arbitrary thread and context. CompletableFuture is equivalent to SettableFuture but there is no limited version equivalent to ListenableFuture. Why is it a problem? If API returns CompletableFuture and then two threads wait for it to complete (nothing wrong with that), one of these threads can resolve this future and wake up other thread, while it's only the API implementation that should do it. But when API tries to resolve the future later, call to complete() is ignored. It can lead to really nasty bugs which are avoided in Guava by separating these two responsibilities. CompletableFuture is ignored in JDK. ExecutorService was not retrofitted to return CompletableFuture. Literally CompletableFuture is not referenced anywhere in JDK. It's a really useful class, backward compatible withFuture, but not really promoted in standard library. Bloated API (?) Fifty methods in total, most in three variants. Splitting settable and listenable (see above) would help. Also some methods like runAfterBoth() or runAfterEither() IMHO do not really belong to anyCompletableFuture. Is there a difference between fast.runAfterBoth(predictable, ...) andpredictable.runAfterBoth(fast, ...)? No, but API favours one or the other. Actually I believerunAfterBoth(fast, predictable, ...) much better expresses my intention. CompletableFuture.getNow(T) should take Supplier instead of raw reference. In the example belowexpensiveAlternative() is always code, irrespective to whether future finished or not: future.getNow(expensiveAlternative()); However we can easily tweak this behaviour (I know, there is a small race condition here, but the original getNow()works this way as well): public static T getNow( CompletableFuture future, Supplier valueIfAbsent) throws ExecutionException, InterruptedException { if (future.isDone()) { return future.get(); } else { return valueIfAbsent.get(); } } With this utility method we can avoid calling expensiveAlternative() when it's not needed: getNow(future, () -> expensiveAlternative()); //or: getNow(future, this::expensiveAlternative); In overall CompletableFuture is a wonderful new tool in our JDK belt. Minor API issues and sometimes too verbose syntax due to limited type inference shouldn't stop you from using it. At least it's a solid foundation for better abstractions and more robust code.