This post comes from Vladimir Šor at the Plumbr blog. Some of you have been there. You have added -Xmx option to your startup scripts and sat back relaxed knowing that there is no way your Java process is going to eat up more memory than your fine-tuned option had permitted. And then you were up for a nasty surprise. Either by yourself by checking a process table in your development / test box or if things got really bad then by operations who calls you in the middle of the night telling that the 4G memory you had asked for the production is exhausted. And that the application just died. So what the heck is happening under the hood? Why is the process consuming more memory than you allocated? Is it a bug or something completely normal? Bear with me and I will guide you through what is happening. First of all, part of it can definitely be a malicious native code leaking memory. But on 99% of the cases it is completely normal behaviour of the JVM. What you have specified via the -Xmx switches is limiting the memory consumed by your application heap. Besides heap there are other regions in memory which your application is using under the hood – namely permgen and stack sizes. So in order to limit those you should also specify the -XX:MaxPermSize and -Xss options respectively. In a short, you can predict your application memory usage with the following formula Max memory = [-Xmx] + [-XX:MaxPermSize] + number_of_threads * [-Xss] But besides the memory consumed by your application, the JVM itself also needs some elbow room. The need for it derives from several different reasons: Garbage collection. As you might recall, Java is a garbage collected language. In order for the garbage collector to know which objects are eligible for collection, it needs to keep track of the object graphs. So this is one part of the memory lost for this internal bookkeeping. Especially G1 is known for its excessive appetite for additional memory, so be aware of this. JIT optimization. Java Virtual Machine optimizes the code during the runtime. Again, to know which parts to optimize it needs to keep track of the execution of certain code parts. So again, you are going to lose memory. Off-heap allocations. If you happen to use off-heap memory, for example while using direct or mapped ByteBuffers yourself or via some clever 3rd party API then voila – you are extending your heap to something you actually cannot control via JVM configuration. JNI code. When you are using native code for example in the format of Type 2 database drivers then again, you are loading code in the native memory. Metaspace. If you are an early adopter of Java 8, you are using metaspace instead of the good old permgen to store class declarations. This is unlimited and in a native part of the JVM. You can end up using memory for other reasons than listed above as well, but I hope I managed to convince you that there is a significant amount of memory eaten up by the JVM internals. But is there a way to predict how much memory is actually going to be needed? Or at least understand where it disappears in order to optimize? As we have found out via painful experience – it is not possible to predict it with a reasonable precision. The JVM overhead can range from anything between just a few percentages to several hundred %. Your best friend is again the good old trial and error. So you need to run your application with loads similar to production environment and measure. Measuring the additional overhead is trivial – just monitor the process with the OS built-in tools (top on Linux, Activity Monitor on OS X, Task Manager on Windows) to find out the real memory consumption. Subtract the heap and permgen sizes from the real consumption and you see the overhead posed. Now if you need to reduce to overhead you would like to understand where it actually disappears. We have found vmmap on Mac OS X and pmap on Linux to be a truly helpful tools in this case. We have not used the vmmap port to Windows by ourselves, but it seems there is a tool for Windows fanboys as well. The following example illustrates this situation. I have launched my Jetty with the following startup parameters: -Xmx168m -Xms168m -XX:PermSize=32m -XX:MaxPermSize=32m -Xss1m Knowing that I have 30 threads launched in my application I might expect that my memory usage does not exceed 230M no matter what. But now when I look at the Activity Monitor on my Mac OS X, I see something different The real memory usage has exceeded 320M. Now digging under the hood how the process with the help of the vmmap output we start to understand where the memory is disappearing. Lets go through some samples: The following says we have lost close to 2MB is lost to memory mapped rt.jar library. mapped file 00000001178b9000-0000000117a88000 [ 1852K] r--/r-x SM=ALI /Library/Java/JavaVirtualMachines/jdk1.7.0_21.jdk/Contents/Home/jre/lib/rt.jar - Next section explains that we are using ~6MB for a particular Dynamic Library loaded __TEXT 0000000104573000-0000000104c00000 [ 6708K] r-x/rwx SM=COW /Library/Java/JavaVirtualMachines/jdk1.7.0_21.jdk/Contents/Home/jre/lib/server/libjvm.dylib - See more at: http://plumbr.eu/blog/why-does-my-java-process-consume-more-memory-than-xmx?utm_source=rss&utm_medium=rss&utm_campaign=rss20130618#sthash.G8fx60eX.dpuf And here we have threads no 25-30 each allocating 1MB for their stacks and stack guards Stack 000000011a5f1000-000000011a6f0000 [ 1020K] rw-/rwx SM=ZER thread 25 Stack 000000011aa8c000-000000011ab8b000 [ 1020K] rw-/rwx SM=ZER thread 27 Stack 000000011ab8f000-000000011ac8e000 [ 1020K] rw-/rwx SM=ZER thread 28 Stack 000000011ac92000-000000011ad91000 [ 1020K] rw-/rwx SM=ZER thread 29 Stack 000000011af0f000-000000011b00e000 [ 1020K] rw-/rwx SM=ZER thread 30 - See more at: http://plumbr.eu/blog/why-does-my-java-process-consume-more-memory-than-xmx?utm_source=rss&utm_medium=rss&utm_campaign=rss20130618#sthash.G8fx60eX.dpuf STACK GUARD 000000011a5ed000-000000011a5ee000 [ 4K] ---/rwx SM=NUL stack guard for thread 25 STACK GUARD 000000011aa88000-000000011aa89000 [ 4K] ---/rwx SM=NUL stack guard for thread 27 STACK GUARD 000000011ab8b000-000000011ab8c000 [ 4K] ---/rwx SM=NUL stack guard for thread 28 STACK GUARD 000000011ac8e000-000000011ac8f000 [ 4K] ---/rwx SM=NUL stack guard for thread 29 STACK GUARD 000000011af0b000-000000011af0c000 [ 4K] ---/rwx SM=NUL stack guard for thread 30 - See more at: http://plumbr.eu/blog/why-does-my-java-process-consume-more-memory-than-xmx?utm_source=rss&utm_medium=rss&utm_campaign=rss20130618#sthash.G8fx60eX.dpuf I hope I managed to shed some light upon the tricky task of predicting and measuring the actual memory consumption. If you enjoyed the content – subscribe to our RSS feed or start following us in Twitter to be notified on future interesting posts.

It’s really nice if you can decouple your external API from the details of application segregation and deployment. In a previous post I explained some of the benefits of using a reverse proxy. On my current project we’ve building a distributed service oriented architecture that also exposes an HTTP API, and we’re using a reverse proxy to route requests addressed to our API to individual components. We have chosen the excellent Nginx web server to serve as our reverse proxy; it’s fast, reliable and easy to configure. We use it to aggregate multiple services exposing HTTP APIs into a single URL space. So, for example, when you type: http://api.example.com/product/pinstripe_suit It gets routed to: http://10.0.1.101:8001/product/pinstripe_suit But when you go to: http://api.example.com/customer/103474783 It gets routed to http://10.0.1.104:8003/customer/103474783 To the consumer of the API it appears that they are exploring a single URL space (http://api.example.com/blah/blah), but behind the scenes the different top level segments of the URL route to different back end servers. /product/… routes to 10.0.1.101:8001, but /customer/… routes to 10.0.1.104:8003. We also want this to be self-configuring. So, say I want to create a new component of the system that records stock levels. Rather than extending an existing component, I want to be able to write a stand-alone executable or service that exposes an HTTP endpoint, have it be automatically deployed to one of the hosts in my cloud infrastructure, and have Nginx automatically route requests addressed http://api.example.com/stock/whatever to my new component. We also want to load balance these back end services. We might want to deploy several instances of our new stock API and have Nginx automatically round robin between them. We call each top level segment ( /stock, /product, /customer ) a claim. A component publishes an ‘AddApiClaim’ message over RabbitMQ when it comes on line. This message has 3 fields: ‘Claim', ‘ipAddress’, and ‘PortNumber’. We have a special component, ProxyAutomation, that subscribes to these messages and rewrites the Nginx configuration as required. It uses SSH and SCP to log into the Nginx server, transfer the various configuration files, and instruct Nginx to reload its configuration. We use the excellent SSH.NET library to automate this. A really nice thing about Nginx configuration is wildcard includes. Take a look at our top level configuration file: ... http { include /etc/nginx/mime.types; default_type application/octet-stream; log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; access_log /var/log/nginx/access.log main; sendfile on; keepalive_timeout 65; include /etc/nginx/conf.d/*.conf; } Line 16 says, take any *.conf file in the conf.d directory and add it here. Inside conf.d is a single file for all api.example.com requests: include /etc/nginx/conf.d/api.example.com.conf.d/upstream.*.conf; server { listen 80; server_name api.example.com; include /etc/nginx/conf.d/api.example.com.conf.d/location.*.conf; location / { root /usr/share/nginx/api.example.com; index index.html index.htm; } } This is basically saying listen on port 80 for any requests with a host header ‘api.example.com’. This has two includes. The first one at line 1, I’ll talk about later. At line 7 it says ‘take any file named location.*.conf in the subdirectory ‘api.example.com.conf.d’ and add it to the configuration. Our proxy automation component adds new components (AKA API claims) by dropping new location.*.conf files in this directory. For example, for our stock component it might create a file, ‘location.stock.conf’, like this: location /stock/ { proxy_pass http://stock; } This simply tells Nginx to proxy all requests addressed to api.example.com/stock/… to the upstream servers defined at ‘stock’. This is where the other include mentioned above comes in, ‘upstream.*.conf’. The proxy automation component also drops in a file named upstream.stock.conf that looks something like this: upstream stock { server 10.0.0.23:8001; server 10.0.0.23:8002; } This tells Nginx to round-robin all requests to api.example.com/stock/ to the given sockets. In this example it’s two components on the same machine (10.0.0.23), one on port 8001 and the other on port 8002. As instances of the stock component get deployed, new entries are added to upstream.stock.conf. Similarly, when components get uninstalled, the entry is removed. When the last entry is removed, the whole file is also deleted. This infrastructure allows us to decouple infrastructure configuration from component deployment. We can scale the application up and down by simply adding new component instances as required. As a component developer, I don’t need to do any proxy configuration, just make sure my component publishes add and remove API claim messages and I’m good to go.

NHibernate has a lot of interesting and specific option for mapping entities that can really cover every scenario you have in mind, but you need to be aware of every implication each advanced option has on performances. If you are in a legacy-database scenario where entity A reference Entity B, but someone outside the control of NHibernate can delete record from table used by Entity B, without setting the corresponding referencing field on Entity A. We will end with a Database with broken reference, where rows from Table A references with a field id a record in Table B that no longer exists. When this happens, if you load an Entity of type A that reference an Entity of type B that was deleted, it will throw an exception if you try to access navigation property, because NHibernate cannot find related entity in the Database. If you know NHibernate you can use the not-found=”Ignore” mapping option, that basically tells NHibernate to ignore a broken reference key, if EntityA references an Entity B that was already deleted from database, the reference will be ignored, navigation property will be set to Null, and no exception occurs. This kind of solution is not without side effects, first of all you will find that Every time you load an Entity of Type A another query is issued to the database to verify if related Entity B is really there. This actually disable lazy load, because related entity is always selected. This is not an optimum scenario, because you will end with a lot of extra query and this happens because not-found=”ignore” is only a way to avoid a real problem: you have broken foreign-key in your database. My suggestion is, fix data in database, keep the database clean without broken foreign-keys and remove all not-found=”ignore” mapping option unless you really have no other solution. Please remember that even if you are using NHibernate, you should not forget SQL capabilities. As an example SQL Server (and quite all of the relational database in the market) has the ability to setup rules for foreign-key, es ON DELETE SET NULL that automatically set to null a foreign key on a table, when related record is deleted. Such a feature will prevent you from having broken foreign key, even if some legacy process manipulates the database deleting records without corresponding update in related foreign-key. - See more at: http://www.codewrecks.com/blog/index.php/2013/06/18/relations-with-not-foundignore-disable-lazy-load-and-impact-on-performances/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+AlkampferEng+%28Alkampfer%27s+Place%29#sthash.93db7RQX.dpuf

This is a classic DevOps presentation from a former Velocity conference that I thought people could take another look at or see this for the first time. It's definitely worth it. Velocity 09: John Allspaw, "10+ Deploys Per Day: Dev and Ops Cooperation at Flickr" John Allspaw (Flickr/Yahoo!) and Paul Hammond (Flickr), "10+ Deploys Per Day: Dev and Ops Cooperation at Flickr"

We’ve covered the controversy of a DevOps Team on this blog before. DevOps Teams are dangerous in that many organizations realize that their Dev and Ops groups are so far apart, that they need a neutral, expert group that can bring them together. At the same time, there are increasing reports of DevOps teams becoming yet another silo – and one that is often arrogant and disliked. More silos being exactly the opposite of what we are going for, this is a frightening result. The dangers in a DevOps Team seem to be that they will: End up owning a lot of things, and be a silo onto themselves Be over aggressive in dictating how teams should work A little over a week ago at the IBM Innovate conference in Orlando an attendee shared her successful experience creating a DevOps Liaison Team. I am very intrigued by the naming here. When the team is formed with the name and charter to bring the other groups together, it is hard for them to either own systems or dictate. The other pattern that I liked was what WebMD did in their project to select and implement a deployment automation tool. They went to manages in various traditional silos (Dev, QA, Ops, etc) and asked for a techie who: Did real work Had the respect of their peers The manager would delegate authority to compromise to They ended with a team full of skilled engineers who would work together, but then go back to their own teams once the project was done. However, they had formed solid working relationships cross-silo and could become a liaison between their group and others. Both of these approaches seem to form a DevOps Team of sorts, without creating something evil. They also take chisel to walls between groups rather than trying to reorganize radically all at once. I’m encouraged that our industry is beginning to find some healthy patterns for enterprise DevOps adoption. For more on non-evil DevOps teams, check out our recorded webinar: Building a DevOps Team that Isn’t Evil. What about you? Have you had success forming a dedicated team that helps the rest of the organization grok DevOps? Have you failed? Leave your tips in the comments area below.

Axis2 uses CommonsHTTPTransportSender by default, which is based on commons-httpclient-3.1. At transport level, there’re two types of timeouts that can be set: 1. Socket Timeout 2. Connection Timeout Here’s how you can configure the above ones: Way #1: Configuring timeouts in axis2.xml Socket Timeout some_integer_value Connection timeout some_integer_value Way #2: Configuring timeouts in code … Options options = new Options(); options.setProperty(HTTPConstants.SO_TIMEOUT, new Integer(timeOutInMilliSeconds)); options.setProperty(HTTPConstants.CONNECTION_TIMEOUT, new Integer(timeOutInMilliSeconds)); // or options.setTimeOutInMilliSeconds(timeOutInMilliSeconds); … Real-life code: How to set timeout for a Axis2 Stub? long timeout = 2 * 60 * 1000; // Two minutes Stub stub = new TestStub(); stub._getServiceClient().getOptions().setTimeOutInMilliSeconds(soTimeout); //or long timeout = 2 * 60 * 1000; // Two minutes Stub stub = new TestStub(); stub._getServiceClient().getOptions().setProperty( HTTPConstants.SO_TIMEOUT, new Integer(timeOutInMilliSeconds)); stub._getServiceClient().getOptions().setProperty( HTTPConstants.CONNECTION_TIMEOUT, new Integer(timeOutInMilliSeconds));

Andres and I recently found ourselves wanting to delete a remote branch which had the same name as a tag and therefore the normal way of doing that wasn’t worked out as well as we’d hoped. I created a dummy repository to recreate the state we’d got ourselves into: $ echo "mark" > README $ git commit -am "readme" $ echo "for the branch" >> README $ git commit -am "for the branch" $ git checkout -b same Switched to a new branch 'same' $ git push origin same Counting objects: 5, done. Writing objects: 100% (3/3), 263 bytes, done. Total 3 (delta 0), reused 0 (delta 0) To ssh://[email protected]/markhneedham/branch-tag-test.git * [new branch] same -> same $ git checkout master $ echo "for the tag" >> README $ git commit -am "for the tag" $ git tag same $ git push origin refs/tags/same Counting objects: 5, done. Writing objects: 100% (3/3), 266 bytes, done. Total 3 (delta 0), reused 0 (delta 0) To ssh://[email protected]/markhneedham/branch-tag-test.git * [new tag] same -> same We wanted to delete the remote ‘same’ branch and the following command would work if we hadn’t created a tag with the same name. Instead it throws an error: $ git push origin :same error: dst refspec same matches more than one. error: failed to push some refs to 'ssh://[email protected]/markhneedham/branch-tag-test.git' We learnt that what we needed to do was refer to the full path for the branch when trying to delete it remotely: $ git push origin :refs/heads/same To ssh://[email protected]/markhneedham/branch-tag-test.git - [deleted] same To delete the tag we could do the same thing: $ git push origin :refs/tags/same remote: warning: Deleting a non-existent ref. To ssh://[email protected]/markhneedham/branch-tag-test.git - [deleted] same Of course the tag and branch still exist locally: $ ls -alh .git/refs/heads/ total 16 drwxr-xr-x 4 markhneedham wheel 136B 13 Jun 23:09 . drwxr-xr-x 5 markhneedham wheel 170B 13 Jun 22:39 .. -rw-r--r-- 1 markhneedham wheel 41B 13 Jun 23:08 master -rw-r--r-- 1 markhneedham wheel 41B 13 Jun 23:08 same $ ls -alh .git/refs/tags/ total 8 drwxr-xr-x 3 markhneedham wheel 102B 13 Jun 23:08 . drwxr-xr-x 5 markhneedham wheel 170B 13 Jun 22:39 .. -rw-r--r-- 1 markhneedham wheel 41B 13 Jun 23:08 same So we got rid of them as well: $ git checkout master Switched to branch 'master' $ git branch -d same Deleted branch same (was 08ad88c). $ git tag -d same Deleted tag 'same' (was 1187891) And now they are gone: $ ls -alh .git/refs/heads/ total 8 drwxr-xr-x 3 markhneedham wheel 102B 13 Jun 23:16 . drwxr-xr-x 5 markhneedham wheel 170B 13 Jun 22:39 .. -rw-r--r-- 1 markhneedham wheel 41B 13 Jun 23:08 master $ ls -alh .git/refs/tags/ total 0 drwxr-xr-x 2 markhneedham wheel 68B 13 Jun 23:16 . drwxr-xr-x 5 markhneedham wheel 170B 13 Jun 22:39 .. Out of interest we’d ended up with this situation by mistake rather than by design but it was still fun to do a little bit of git digging to figure out how to solve the problem we’d created for ourselves.

There are two ways to speedup UNIONs in a MySQL database. First use UNION ALL if at all possible, and second try to push down your conditions. 1. UNION ALL is much faster than UNION How does a UNION work? Imagine you have two tables for shirts. The short_sleeve table looks like this: blue green gray black And long_sleeve another that looks like this: red green yellow blue Related: Why Generalists are Better at Scaling the Web If you UNION those two tables, first MySQL will sort the combined set into a temp table like this: black blue blue gray green green red yellow Once it’s done this sort, it can easily remove the duplicate blue & duplicate green for this resulting set: black blue gray green red yellow See also: Mythical MySQL DBA – the talent drought. Why does it do this? UNION is defined that way in SQL. Duplicates must be removed and this is an efficient way for the MySQL engine to remove them. Combine results, sort, remove duplicates and return the set. Queries with UNION can be accelerated in two ways. Switch to UNION ALL or try to push ORDER BY, LIMIT and WHERE conditions inside each subquery. You’ll be glad you did! What if we did UNION ALL? The result would look like this: blue green gray black red green yellow blue Read this: MySQL DBA Interview & Hiring Guide. It doesn’t have to sort, and doesn’t have to remove duplicates. If you imagine combining two 10 million row tables, and don’t have to sort, this speedup can be HUGE. 2. Use Push-down Conditions to speedup UNION in MySQL Imagine with our example above the shirts have a design date, the year they were released. Yes we’re keeping this example very simple to illustrate the concept. Here is the short_sleeve table: blue 2013 green 2013 green 2012 gray 2011 black 2009 black 2011 And long_sleeve table looks like this: red 2012 red 2013 green 2011 yellow 2010 blue 2011 For 2013 designs could combine them like this: (SELECT type, release FROM short_sleeve) UNION (SELECT type, release FROM long_sleeve); WHERE release >=2013; See also: 5 More Things Deadly to Scalability and the original 5 Things Toxic to Scalability.. Here the WHERE clause works on this 11 record temp table: black 2009 black 2011 blue 2011 blue 2013 gray 2011 green 2013 green 2012 green 2011 red 2012 red 2013 yellow 2010 But it would be much faster to move the WHERE inside each subquery like this: (SELECT type, release FROM short_sleeve WHERE release >=2013) UNION (SELECT type, release FROM long_sleeve WHERE release >=2013); That would be operating on a combined 3 record table. Faster to sort & remove duplicates. Smaller result sets cache better too, providing a pay forward dividend. That’s what performance optimization is all about! Read this: RDS or MySQL – 10 Use Cases. Remember multi-million row sets in each part of this query will quickly illustrate the optimization. We’re using very small results to make visualizing easier. You can also use this optimization for ORDER BY and for LIMIT conditions. By reducing the number of records returned by EACH PART of the UNION, you reduce the work that happens at the stage where they are all combined. If you’re seeing some UNION queries in your slow query log, I suggest you try this optimization out and see if you can tweak it.

What is an Object? An object is a collection of data and actions. An object is an instance of a class. Objects have states and behaviors. In the real-world, we can find so many objects around us, for example Cars, Birds, Humans etc. All these objects have a state and behavior. If we consider a Car then it have some data speed, lights on, direction, etc. and have some actions turn right, accelerate, turn lights on, etc. If you compare the java object with a real world object, both of them have similar characteristics. Java objects also have a state and behavior. A Java object's state is stored in fields and behavior is shown via methods. Technically speaking Car, Bird and Human are considered as Class in Java. Brian Christopher is an object of human and Vehicle XKMV-669 is the object of car. Creating Object Using new keyword is the most common way to create an object in java. Syntax:- ClassName Obj.Name = new ClassName(); // Human brianChristopher= new Human(); // Car vehicleXKMV_669 = new Car(); The first statement creates a new Human object and second statement creates Car object. This single statement performs three actions, Declaration, Instantiation, and Initialization. Here, Human brianChristopher is a variable declaration which simply declares to the compiler that the name brianChristopher will be used to refer to an object whose type is Human, the new operator instantiates the Human class (thereby creating a new Human object), and Human initializes the object. Object Lifecycle In Java, it has seven states in Object lifecycle. They are, Created In use Invisible Unreachable Collected Finalized De-allocated Created The following are the some actions performed when an object is created,New memory is allocated for an object. Once the object has been created, assuming that it is assigned to some variable and then it directly moves to the In Use state. In use Objects that are held by at least one strong reference are considered to be “In Use”. Invisible An object is in the “Invisible” state when there are no longer any strong references that are accessible to the program, even though there might still be references. Unreachable An object enters an “unreachable” state when no more strong references to it exist. When an object is unreachable then it is a state for collection. It is important to note that not just any strong reference will hold an object in memory. These must be references that chain from a garbage collection root. Garbage collection roots are a special class of variable that includes,Temporary variables on the stack Collected An object is in the “collected” state when the garbage collector has recognized an object as unreachable and readies it for final processing as a precursor to de-allocation. If the object has a finalize method, then it is marked for finalization. Finalized An object is in the “finalized” state if it is still unreachable after it’s finalize method, if any, has been run. A finalized object is awaiting de-allocation. If you are considering using a finalizer to ensure that important resources are freed in a timely manner, you might want to reconsider. To lengthening object lifetimes, finalize methods can increase object size. De-allocated The de-allocated state is the final step in garbage collection. If an object is still unreachable after all the above work has done, then this is the state for de-allocation. For more detailed discussion about Object Lifecycle with real-world examples download OCAJP 7 Training Lab from EPractize Labs.

I had to read a log file containing a few hundred lines today at work. My first instinct was to open it using my trusty Notepad++.

In the code I have quite recently came across a really bad piece of code that based on class casting in terms of performing some actions on objects. Of course the code needed to be refactored but sometimes you can't do it / or don't want to do it (and it should be understandable) if first you don't have unit tests of that functionality. In the following post I will show how to test such code, how to refactor it and in fact what I think about such code ;) Let's take a look at the project structure: As presented in the post regarding Mocktio RETURNS_DEEP_STUBS Answer for JAXB yet again we have the JAXB generated classes by the JAXB compiler in thecom.blogspot.toomuchcoding.model package. Let's ommit the discussion over the pom.xml file since it's exactly the same as in the previous post. In the com.blogspot.toomuchcoding.adapter package we have adapters over the JAXB PlayerDetails class that provides access to the Player interface. There is the CommonPlayerAdapter.java package com.blogspot.toomuchcoding.adapter; import com.blogspot.toomuchcoding.model.Player; import com.blogspot.toomuchcoding.model.PlayerDetails; /** * User: mgrzejszczak * Date: 09.06.13 * Time: 15:42 */ public class CommonPlayerAdapter implements Player { private final PlayerDetails playerDetails; public CommonPlayerAdapter(PlayerDetails playerDetails){ this.playerDetails = playerDetails; } @Override public void run() { System.out.printf("Run %s. Run!%n", playerDetails.getName()); } public PlayerDetails getPlayerDetails() { return playerDetails; } } DefencePlayerAdapter.java package com.blogspot.toomuchcoding.adapter; import com.blogspot.toomuchcoding.model.DJ; import com.blogspot.toomuchcoding.model.DefensivePlayer; import com.blogspot.toomuchcoding.model.JavaDeveloper; import com.blogspot.toomuchcoding.model.PlayerDetails; /** * User: mgrzejszczak * Date: 09.06.13 * Time: 15:42 */ public class DefencePlayerAdapter extends CommonPlayerAdapter implements DefensivePlayer, DJ, JavaDeveloper { public DefencePlayerAdapter(PlayerDetails playerDetails){ super(playerDetails); } @Override public void defend(){ System.out.printf("Defence! %s. Defence!%n", getPlayerDetails().getName()); } @Override public void playSomeMusic() { System.out.println("Oops I did it again...!"); } @Override public void doSomeSeriousCoding() { System.out.println("System.out.println(\"Hello world\");"); } } OffensivePlayerAdapter.java package com.blogspot.toomuchcoding.adapter; import com.blogspot.toomuchcoding.model.OffensivePlayer; import com.blogspot.toomuchcoding.model.PlayerDetails; /** * User: mgrzejszczak * Date: 09.06.13 * Time: 15:42 */ public class OffensivePlayerAdapter extends CommonPlayerAdapter implements OffensivePlayer { public OffensivePlayerAdapter(PlayerDetails playerDetails){ super(playerDetails); } @Override public void shoot(){ System.out.printf("%s Shooooot!.%n", getPlayerDetails().getName()); } } Ok, now let's go to the more interesting part. Let us assume that we have a very simple factory of players: PlayerFactoryImpl.java package com.blogspot.toomuchcoding.factory; import com.blogspot.toomuchcoding.adapter.CommonPlayerAdapter; import com.blogspot.toomuchcoding.adapter.DefencePlayerAdapter; import com.blogspot.toomuchcoding.adapter.OffensivePlayerAdapter; import com.blogspot.toomuchcoding.model.Player; import com.blogspot.toomuchcoding.model.PlayerDetails; import com.blogspot.toomuchcoding.model.PositionType; /** * User: mgrzejszczak * Date: 09.06.13 * Time: 15:53 */ public class PlayerFactoryImpl implements PlayerFactory { @Override public Player createPlayer(PositionType positionType) { PlayerDetails player = createCommonPlayer(positionType); switch (positionType){ case ATT: return new OffensivePlayerAdapter(player); case MID: return new OffensivePlayerAdapter(player); case DEF: return new DefencePlayerAdapter(player); case GK: return new DefencePlayerAdapter(player); default: return new CommonPlayerAdapter(player); } } private PlayerDetails createCommonPlayer(PositionType positionType){ PlayerDetails playerDetails = new PlayerDetails(); playerDetails.setPosition(positionType); return playerDetails; } } Ok so we have the factory that builds Players. Let's take a look at the Service that uses the factory: PlayerServiceImpl.java package com.blogspot.toomuchcoding.service; import com.blogspot.toomuchcoding.factory.PlayerFactory; import com.blogspot.toomuchcoding.model.*; /** * User: mgrzejszczak * Date: 08.06.13 * Time: 19:02 */ public class PlayerServiceImpl implements PlayerService { private PlayerFactory playerFactory; @Override public Player playAGameWithAPlayerOfPosition(PositionType positionType) { Player player = playerFactory.createPlayer(positionType); player.run(); performAdditionalActions(player); return player; } private void performAdditionalActions(Player player) { if(player instanceof OffensivePlayer){ OffensivePlayer offensivePlayer = (OffensivePlayer) player; performAdditionalActionsForTheOffensivePlayer(offensivePlayer); }else if(player instanceof DefensivePlayer){ DefensivePlayer defensivePlayer = (DefensivePlayer) player; performAdditionalActionsForTheDefensivePlayer(defensivePlayer); } } private void performAdditionalActionsForTheOffensivePlayer(OffensivePlayer offensivePlayer){ offensivePlayer.shoot(); } private void performAdditionalActionsForTheDefensivePlayer(DefensivePlayer defensivePlayer){ defensivePlayer.defend(); try{ DJ dj = (DJ)defensivePlayer; dj.playSomeMusic(); JavaDeveloper javaDeveloper = (JavaDeveloper)defensivePlayer; javaDeveloper.doSomeSeriousCoding(); }catch(ClassCastException exception){ System.err.println("Sorry, I can't do more than just play football..."); } } public PlayerFactory getPlayerFactory() { return playerFactory; } public void setPlayerFactory(PlayerFactory playerFactory) { this.playerFactory = playerFactory; } } Let's admit it... this code is bad. Internally when you look at it (regardless of the fact whether it used instance of operator or not) you feel that it is evil :) As you can see in the code we have some class casts going on... How on earth can we test it? In the majority of testing frameworks you can't do such class casts on mocks since they are built with the CGLIB library and there can be some ClassCastExceptions thrown. You could still not return mocks and real implementations (assuming that those will not perform any ugly stuff in the construction process) and it could actually work but still - this is bad code :P Mockito comes to the rescue (although you shouldn't overuse this feature - in fact if you need to use it please consider refactoring it) with its extraInterfaces feature: extraInterfaces MockSettings extraInterfaces(java.lang.Class... interfaces) Specifies extra interfaces the mock should implement. Might be useful for legacy code or some corner cases. For background, see issue 51 hereThis mysterious feature should be used very occasionally. The object under test should know exactly its collaborators & dependencies. If you happen to use it often than please make sure you are really producing simple, clean & readable code. Examples: Foo foo = mock(Foo.class, withSettings().extraInterfaces(Bar.class, Baz.class)); //now, the mock implements extra interfaces, so following casting is possible: Bar bar = (Bar) foo; Baz baz = (Baz) foo; Parameters:interfaces - extra interfaces the should implement. Returns:settings instance so that you can fluently specify other settings Now let's take a look at the test: PlayerServiceImplTest.java package com.blogspot.toomuchcoding.service; import com.blogspot.toomuchcoding.factory.PlayerFactory; import com.blogspot.toomuchcoding.model.*; import org.junit.Test; import org.junit.runner.RunWith; import org.mockito.InjectMocks; import org.mockito.Mock; import org.mockito.invocation.InvocationOnMock; import org.mockito.runners.MockitoJUnitRunner; import org.mockito.stubbing.Answer; import static org.hamcrest.CoreMatchers.is; import static org.junit.Assert.assertThat; import static org.mockito.BDDMockito.*; /** * User: mgrzejszczak * Date: 08.06.13 * Time: 19:26 */ @RunWith(MockitoJUnitRunner.class) public class PlayerServiceImplTest { @Mock PlayerFactory playerFactory; @InjectMocks PlayerServiceImpl objectUnderTest; @Mock(extraInterfaces = {DJ.class, JavaDeveloper.class}) DefensivePlayer defensivePlayerWithDjAndJavaDevSkills; @Mock DefensivePlayer defensivePlayer; @Mock OffensivePlayer offensivePlayer; @Mock Player commonPlayer; @Test public void shouldReturnOffensivePlayerThatRan() throws Exception { //given given(playerFactory.createPlayer(PositionType.ATT)).willReturn(offensivePlayer); //when Player createdPlayer = objectUnderTest.playAGameWithAPlayerOfPosition(PositionType.ATT); //then assertThat(createdPlayer == offensivePlayer, is(true)); verify(offensivePlayer).run(); } @Test public void shouldReturnDefensivePlayerButHeWontBeADjNorAJavaDev() throws Exception { //given given(playerFactory.createPlayer(PositionType.GK)).willReturn(defensivePlayer); //when Player createdPlayer = objectUnderTest.playAGameWithAPlayerOfPosition(PositionType.GK); //then assertThat(createdPlayer == defensivePlayer, is(true)); verify(defensivePlayer).run(); verify(defensivePlayer).defend(); verifyNoMoreInteractions(defensivePlayer); } @Test public void shouldReturnDefensivePlayerBeingADjAndAJavaDev() throws Exception { //given given(playerFactory.createPlayer(PositionType.GK)).willReturn(defensivePlayerWithDjAndJavaDevSkills); doAnswer(new Answer

NetBeans IDE 7.3.1 is an update to NetBeans IDE 7.3 and includes the following highlights: Support for Java EE 7 development Deployment to GlassFish 4 Support for major Java EE 7 specifications: JSF 2.2, JPA 2.1, JAX-RS 2.0, WebSocket 1.0 and more Support for WebLogic 12.1.2 and JBoss 7.x Integration of recent patches There are two ways to get the recent changes: To use the new Java EE 7 support, it is recommended to download and install NetBeans IDE 7.3.1. To get only the integration of recent patches: Launch your current installation of NetBeans IDE 7.3. An update notification will appear in the IDE. Click the notification box to install the updates. OR to perform the update manually, in the IDE select Help-->Check for Updates. NetBeans IDE 7.3.1 is available in English, Brazilian Portuguese, Japanese, Russian, and Simplified Chinese.

RPC/encoded is a vestige from before SOAP objects were defined with XML Schema. It’s not widely supported anymore. You will need to generate the stubs using Apache Axis 1.0, which is from the same era. java org.apache.axis.wsdl.WSDL2Java http://someurl?WSDL You will need the following jars or equivalents in the -cp classpath param: axis-1.4.jar commons-logging-1.1.ja commons-discovery-0.2.jar jaxrpc-1.1.jar saaj-1.1.jar wsdl4j-1.4.jar activation-1.1.jar mail-1.4.jar This will generate similar stubs to wsimport. Alternatively, if you are not using the parts of the schema that require rpc/encoded, you can download a copy of the WSDL and comment out those bits. Then run wsimport against the local file. If you look at the WSDL, the following bits are using rpc/encoded: Sources 1. http://bitkickers.blogspot.com/2008/12/rpcencoded-web-services-on-java-16.html 2. http://stackoverflow.com/questions/412772/java-rpc-encoded-wsdls-are-not-supported-in-jaxws-2-0

Sorry for not having written for some time but I was busy with writing the JBoss Drools Refcard for DZone and I am in the middle of writing a book about Mockito so I don't have too much time left for blogging... Anyway quite recently on my current project I had an interesting situation regarding unit testing with Mockito and JAXB structures. We have very deeply nested JAXB structures generated from schemas that are provided for us which means that we can't change it in anyway. Let's take a look at the project structure: The project structure is pretty simple - there is a Player.xsd schema file that thanks to using the jaxb2-maven-plugin produces the generated JAXB Java classes corresponding to the schema in the target/jaxb/ folder in the appropriate package that is defined in the pom.xml. Speaking of which let's take a look at the pom.xml file. The pom.xml : 4.0.0 com.blogspot.toomuchcoding mockito-deep_stubs 0.0.1-SNAPSHOT UTF-8 1.6 1.6 spring-release http://maven.springframework.org/release maven-us-nuxeo https://maven-us.nuxeo.org/nexus/content/groups/public junit junit 4.10 org.mockito mockito-all 1.9.5 test org.apache.maven.plugins maven-compiler-plugin 2.5.1 org.codehaus.mojo jaxb2-maven-plugin 1.5 xjc xjc com.blogspot.toomuchcoding.model ${project.basedir}/src/main/resources/xsd Apart from the previously defined project dependencies, as mentioned previously in the jaxb2-maven-plugin in the configuration node you can define the packageName value that defines to which package should the JAXB classes be generated basing on the schemaDirectory value where the plugin can find the proper schema files. Speaking of which let's check the Player.xsd schema file (simillar to the one that was present in the Spring JMS automatic message conversion article of mine): As you can see I'm defining some complex types that even though might have no business sense but you can find such examples in the real life :) Let's find out how the method that we would like to test looks like. Here we have the PlayerServiceImpl that implements the PlayerService interface: package com.blogspot.toomuchcoding.service; import com.blogspot.toomuchcoding.model.PlayerDetails; /** * User: mgrzejszczak * Date: 08.06.13 * Time: 19:02 */ public class PlayerServiceImpl implements PlayerService { @Override public boolean isPlayerOfGivenCountry(PlayerDetails playerDetails, String country) { String countryValue = playerDetails.getClubDetails().getCountry().getCountryCode().getCountryCode().value(); return countryValue.equalsIgnoreCase(country); } } We are getting the nested elements from the JAXB generated classes. Although it violates the Law of Demeter it is quite common to call methods of structures because JAXB generated classes are in fact structures so in fact I fully agree with Martin Fowler that it should be called the Suggestion of Demeter. Anyway let's see how you could test the method: @Test public void shouldReturnTrueIfCountryCodeIsTheSame() throws Exception { //given PlayerDetails playerDetails = new PlayerDetails(); ClubDetails clubDetails = new ClubDetails(); CountryDetails countryDetails = new CountryDetails(); CountryCodeDetails countryCodeDetails = new CountryCodeDetails(); playerDetails.setClubDetails(clubDetails); clubDetails.setCountry(countryDetails); countryDetails.setCountryCode(countryCodeDetails); countryCodeDetails.setCountryCode(CountryCodeType.ENG); //when boolean playerOfGivenCountry = objectUnderTest.isPlayerOfGivenCountry(playerDetails, COUNTRY_CODE_ENG); //then assertThat(playerOfGivenCountry, is(true)); } The function checks if, once you have the same Country Code, you get a true boolean from the method. The only problem is the amount of sets and instantiations that take place when you want to create the input message. In our projects we have twice as many nested elements so you can only imagine the number of code that we would have to produce to create the input object... So what can be done to improve this code? Mockito comes to the rescue to together with the RETURN_DEEP_STUBS default answer to the Mockito.mock(...) method: @Test public void shouldReturnTrueIfCountryCodeIsTheSameUsingMockitoReturnDeepStubs() throws Exception { //given PlayerDetails playerDetailsMock = mock(PlayerDetails.class, RETURNS_DEEP_STUBS); CountryCodeType countryCodeType = CountryCodeType.ENG; when(playerDetailsMock.getClubDetails().getCountry().getCountryCode().getCountryCode()).thenReturn(countryCodeType); //when boolean playerOfGivenCountry = objectUnderTest.isPlayerOfGivenCountry(playerDetailsMock, COUNTRY_CODE_ENG); //then assertThat(playerOfGivenCountry, is(true)); } So what happened here is that you use the Mockito.mock(...) method and provide the RETURNS_DEEP_STUBS answer that will create mocks automatically for you. Mind you that Enums can't be mocked that's why you can't write in the Mockito.when(...) functionplayerDetailsMock.getClubDetails().getCountry().getCountryCode().getCountryCode().getValue(). Summing it up you can compare the readability of both tests and see how clearer it is to work with JAXB structures by using Mockito RETURNS_DEEP_STUBS default answer. Naturally sources for this example are available at BitBucket and GitHub.

Apache XML-RPC is a Java implementation of XML-RPC, a popular protocol that uses XML over HTTP to implement remote procedure calls. Here’s how timeouts can be configured for an XML-RPC client: XmlRpcClient client = new XmlRpcClient(); XmlRpcClientConfigImpl config = new XmlRpcClientConfigImpl(); // Setting timeouts for xmlrpc calls made using XmlRpcSunHttpTransportFactory, the default connection factory int xmlrpcConnTimeout = 10000; // Connection timeout int xmlrpcReplyTimeOut = 60000; // Reply timeout XmlRpcClientConfigImpl config = new XmlRpcClientConfigImpl(); config.setServerURL(new URL(serverURL)); config.setConnectionTimeout(xmlrpcConnTimeout); config.setReplyTimeout(xmlrpcReplyTimeOut); client.setConfig(config);

Elasticsearch is a flexible and powerful open source, distributed real-time search and analytics engine for the cloud based on Apache Lucene which provides full text search capabilities. It is document oriented and schema free. Asciidoctor is a pure Ruby processor for converting AsciiDoc source files and strings into HTML 5, DocBook 4.5 and other formats. Apart of Asciidoctor Ruby part, there is an Asciidoctor-java-integration project which let us call Asciidoctor functions from Java without noticing that Ruby code is being executed. In this post we are going to see how we can use Elasticsearch over AsciiDocdocuments to make them searchable by their header information or by their content. Let's add required dependencies: junit junit 4.11 test com.googlecode.lambdaj lambdaj 2.3.3 org.elasticsearch elasticsearch 0.90.1 org.asciidoctor asciidoctor-java-integration 0.1.3 Lambdaj library is used to convert AsciiDoc files to a json documents. Now we can start an Elasticsearch instance which in our case it is going to be an embedded instance. node = nodeBuilder().local(true).node(); Next step is parse AsciiDoc document header, read its content and convert them into a json document. An example of json document stored in Elasticsearch can be: { "title":"Asciidoctor Maven plugin 0.1.2 released!", "authors":[ { "author":"Jason Porter", "email":"[email protected]" } ], "version":null, "content":"= Asciidoctor Maven plugin 0.1.2 released!.....", "tags":[ "release", "plugin" ] } And for converting an AsciiDoc File to a json document we are going to useXContentBuilder class which is provided by ElasticsearchJava API to create jsondocuments programmatically. package com.lordofthejars.asciidoctor; import static org.elasticsearch.common.xcontent.XContentFactory.*; import java.io.File; import java.io.FileInputStream; import java.io.FileNotFoundException; import java.io.IOException; import java.util.List; import org.asciidoctor.Asciidoctor; import org.asciidoctor.Author; import org.asciidoctor.DocumentHeader; import org.asciidoctor.internal.IOUtils; import org.elasticsearch.common.xcontent.XContentBuilder; import ch.lambdaj.function.convert.Converter; public class AsciidoctorFileJsonConverter implements Converter { private Asciidoctor asciidoctor; public AsciidoctorFileJsonConverter() { this.asciidoctor = Asciidoctor.Factory.create(); } public XContentBuilder convert(File asciidoctor) { DocumentHeader documentHeader = this.asciidoctor.readDocumentHeader(asciidoctor); XContentBuilder jsonContent = null; try { jsonContent = jsonBuilder() .startObject() .field("title", documentHeader.getDocumentTitle()) .startArray("authors"); Author mainAuthor = documentHeader.getAuthor(); jsonContent.startObject() .field("author", mainAuthor.getFullName()) .field("email", mainAuthor.getEmail()) .endObject(); List authors = documentHeader.getAuthors(); for (Author author : authors) { jsonContent.startObject() .field("author", author.getFullName()) .field("email", author.getEmail()) .endObject(); } jsonContent.endArray() .field("version", documentHeader.getRevisionInfo().getNumber()) .field("content", readContent(asciidoctor)) .array("tags", parseTags((String)documentHeader.getAttributes().get("tags"))) .endObject(); } catch (IOException e) { throw new IllegalArgumentException(e); } return jsonContent; } private String[] parseTags(String tags) { tags = tags.substring(1, tags.length()-1); return tags.split(", "); } private String readContent(File content) throws FileNotFoundException { return IOUtils.readFull(new FileInputStream(content)); } } Basically we are building the json document by calling startObject methods to start a new object, field method to add new fields, and startArray to start an array. Then this builder will be used to render the equivalent object in json format. Notice that we are using readDocumentHeader method from Asciidoctor class which returns header attributes from AsciiDoc file without reading and rendering the whole document. And finally content field is set with all document content. And now we are ready to start indexing documents. Note that populateData method receives as parameter a Client object. This object is from Elasticsearch Java APIand represents a connection to Elasticsearch database. import static ch.lambdaj.Lambda.convert; //.... private void populateData(Client client) throws IOException { List asciidoctorFiles = new ArrayList() {{ add(new File("target/test-classes/java_release.adoc")); add(new File("target/test-classes/maven_release.adoc")); }; List jsonDocuments = convertAsciidoctorFilesToJson(asciidoctorFiles); for (int i=0; i < jsonDocuments.size(); i++) { client.prepareIndex("docs", "asciidoctor", Integer.toString(i)).setSource(jsonDocuments.get(i)).execute().actionGet(); } client.admin().indices().refresh(new RefreshRequest("docs")).actionGet(); } private List convertAsciidoctorFilesToJson(List asciidoctorFiles) { return convert(asciidoctorFiles, new AsciidoctorFileJsonConverter()); } It is important to note that the first part of the algorithm is converting all our AsciiDocfiles (in our case two) to XContentBuilder instances by using previous converter class and the method convert of Lambdaj project. If you want you can take a look to both documents used in this example in https://github.com/asciidoctor/asciidoctor.github.com/blob/develop/news/asciidoctor-java-integration-0-1-3-released.adoc and https://github.com/asciidoctor/asciidoctor.github.com/blob/develop/news/asciidoctor-maven-plugin-0-1-2-released.adoc. Next part is inserting documents inside one index. This is done by using prepareIndexmethod, which requires an index name (docs), an index type (asciidoctor), and the idof the document being inserted. Then we call setSource method which transforms theXContentBuilder object to json, and finally by calling execute().actionGet(), data is sent to database. The final step is only required because we are using an embedded instance ofElasticsearch (in production this part should not be required), which refresh the indexes by calling refresh method. After that point we can start querying Elasticsearch for retrieving information from our AsciiDoc documents. Let's start with very simple example, which returns all documents inserted: SearchResponse response = client.prepareSearch().execute().actionGet(); Next we are going to search for all documents that has been written by Alex Sotowhich in our case is one. import static org.elasticsearch.index.query.QueryBuilders.matchQuery; //.... QueryBuilder matchQuery = matchQuery("author", "Alex Soto"); QueryBuilder matchQuery = matchQuery("author", "Alexander Soto"); Note that I am searching for field author the string Alex Soto, which returns only one. The other document is written by Jason. But it is interesting to say that if you search for Alexander Soto, the same document will be returned; Elasticsearch is smart enough to know that Alex and Alexander are very similar names so it returns the document too. More queries, how about finding documents written by someone who is called Alex, but not Soto. import static org.elasticsearch.index.query.QueryBuilders.fieldQuery; //.... QueryBuilder matchQuery = fieldQuery("author", "+Alex -Soto"); And of course no results are returned in this case. See that in this case we are using afield query instead of a term query, and we use +, and - symbols to exclude and include words. Also you can find all documents which contains the word released on title. import static org.elasticsearch.index.query.QueryBuilders.matchQuery; //.... QueryBuilder matchQuery = matchQuery("title", "released"); And finally let's find all documents that talks about 0.1.2 release, in this case only one document talks about it, the other one talks about 0.1.3. QueryBuilder matchQuery = matchQuery("content", "0.1.2"); Now we only have to send the query to Elasticsearch database, which is done by using prepareSearch method. SearchResponse response = client.prepareSearch("docs") .setTypes("asciidoctor") .setQuery(matchQuery) .execute() .actionGet(); SearchHits hits = response.getHits(); for (SearchHit searchHit : hits) { System.out.println(searchHit.getSource().get("content")); } Note that in this case we are printing the AsciiDoc content through console, but you could use asciidoctor.render(String content, Options options) method to render the content into required format. So in this post we have seen how to index documents using Elasticsearch, how to get some important information from AsciiDoc files using Asciidoctor-java-integration project, and finally how to execute some queries to inserted documents. Of course there are more kind of queries in Elasticsearch, but the intend of this post wasn't to explore all possibilities of Elasticsearch. Also as corollary, note how important it is using AsciiDoc format for writing your documents. Without much effort you can build a search engine for your documentation. On the other side, imagine all code that would be required to implement the same using any proprietary binary format like Microsoft Word. So we have shown another reason to use AsciiDoc instead of other formats.

i’ve already dealt on how to index documents with solr and tika and in this article i’ll explain how you can not only search for documents that match your query, but returns even some text extract that shows where the document match the query . to achieve this, you should store the full content of the document inside your index, usually i create a couple of fields, one called content that will contain the content of the file, and with a copyfield directive ( ) automatically copy that value inside the catch all field called text. text field is multivalued and not stored, it is only indexed to permit search inside various field of the document . content field store the extracted text from tikaand it is useful both for highlighting and to troubleshoot extraction problems, because it contains the exact text extracted by tika. now suppose you want to search for the term branch and want also to highlight the part of the text where you find that term , you can simply issue a query that ask for highlighting, it is really simple. http://localhost:8080/testinstance/tikacrawl/select?q=text%3abranch&fl=id%2ctitle%2cauthor&wt=xml&hl=true&hl.snippets=20&hl.fl=content&hl.usephrasehighlighter=true this simple query ask for document with text that contains word branch, i want to extract (fl=) only title and author fields, want xml format and with hl=true i’m asking for snippet of matching text, hl.snippets=20 instruct solr to search for a maximum of 20 snippet and hl.usephrasehighlighter=true use a specific highlighter that try to extract a single phrase from the text. the most important parameter is the hl.fl=content that specify the field of the document that contains the text used for highlight. in the results, after all matching documents there is a new section that contains all the highlights for each document figure 1: hilight for the tfs branching guid – scenarios 2.0.pdf file the name of the element match the id of the document (in my configuration full path of the file), and there a list of highlights that follows. but the true power of solr comes out if you start to use languages specific fields . i’ve just changed in schema.xml the type of content and text from general_text to text_en and this simple modification enables a more specific tokenizer, capable of doing full-text searches. suppose you want to know all documents that deal with branching strategies, here is a possible query http://localhost:8080/testinstance/tikacrawl/select?q=text%3a% 22branch+strategy%22~3 &fl=id%2ctitle&wt=xml&hl=true&hl.snippets=5&hl.fl=content&hl.usephrasehighlighter=true&hl.fragsize=300 the key is in the search query text:”branch strategy”~3 that states i’m interested in documents containing both branch and strategy terms and with a relative distance of no more than three words. since text was indexed with text_en field type i got full-text search, and i have a confirmation looking at the highlights. figure 2: highlights for a proximity query with full text, as you can see word branching matches even if i searched for branch and voilà!! you have full-text searching inside file content with minimal amount of work and a simple rest interface for querying the index

I’ve recently had the need to automate configuration of Nginx on an Ubuntu server. Of course, in UNIX land we like to use SSH (Secure Shell) to log into our servers and manage them remotely. Wouldn’t it be nice, I thought, if there was a managed SSH library somewhere so that I could automate logging onto my Ubuntu server, run various commands and transfer files. A short Google turned up SSH.NET by the somewhat mysterious Olegkap (at least I couldn’t find out anything else about them) which turned out to be just what I wanted. Here’s the blurb on the CodePlex site: “This project was inspired by Sharp.SSH library which was ported from java and it seems like was not supported for quite some time. This library is complete rewrite using .NET 4.0, without any third party dependencies and to utilize the parallelism as much as possible to allow best performance I can get.” It does exactly what it says on the tin. It’s on NuGet, so you can grab it with: PM> Install-Package SSH.NET Here’s how you run a remote command. First you need to build a ConnectionInfo object: public ConnectionInfo CreateConnectionInfo() { const string privateKeyFilePath = @"C:\some\private\key.pem"; ConnectionInfo connectionInfo; using (var stream = new FileStream(privateKeyFilePath, FileMode.Open, FileAccess.Read)) { var privateKeyFile = new PrivateKeyFile(stream); AuthenticationMethod authenticationMethod = new PrivateKeyAuthenticationMethod("ubuntu", privateKeyFile); connectionInfo = new ConnectionInfo( "my.server.com", "ubuntu", authenticationMethod); } return connectionInfo; } Then you simply create an SshClient instance and run commands: public void Connect() { using (var ssh = new SshClient(CreateConnectionInfo())) { ssh.Connect(); var command = ssh.CreateCommand("uptime"); var result = command.Execute(); Console.Out.WriteLine(result); ssh.Disconnect(); } } Here I’m running the ‘uptime’ command which output this when I ran it just now: 14:37:46 up 22 days, 3:59, 0 users, load average: 0.08, 0.03, 0.05 To transfer a file, just use the ScpClient: public void GetConfigurationFiles() { using (var scp = new ScpClient(CreateNginxServerConnectionInfo())) { scp.Connect(); scp.Download("/etc/nginx/", new DirectoryInfo(@"D:\Temp\ScpDownloadTest")); scp.Disconnect(); } } Which grabs all my Nginx configuration and transfers it to a directory tree on my windows machine. All in all a very nice little library that’s been working well for me so far. Give it a try if you need to interact with a UNIX-like machine from .NET code.

about an hour ago i gave a presentation on indexeddb. one of the attendees asked about dates and being able to filter based on a date range. i told him that my assumption was that you would need to convert the dates into numbers and use a number-based range. turns out i was wrong. here is an example. i began by creating an objectstore that used an index on the created field. since our intent is to search via a date field, i decided "created" would be a good name. i also named my objectstore as "data". boring, but it works. var openrequest = indexeddb.open("idbpreso_date1",1); openrequest.onupgradeneeded = function(e) { var thisdb = e.target.result; if(!thisdb.objectstorenames.contains("data")) { var os = thisdb.createobjectstore("data", {autoincrement:true}); os.createindex("created", "created", {unique:false}); } } next - i built a simple way to seed data. i based on a button click event to add 10 objects. each object will have one property, created, and the date object will be based on a random date from now till 7 days in the future. function doseed() { var now = new date(); for(var i=0; i<10; i++) { var daydiff = getrandomint(1, 7); var thisdate = new date(); thisdate.setdate(now.getdate() + daydiff); db.transaction(["data"],"readwrite").objectstore("data").add({created:thisdate}); } } //credit: mozilla developer center function getrandomint (min, max) { return math.floor(math.random() * (max - min + 1)) + min; } note that since indexeddb calls are asynchronous, my code should handle updating the user to let them know when the operation is done. since this is just a quick demo though, and since that add operation will complete incredibly fast, i decided to not worry about it. so at this point we'd have an application that lets us add data containing a created property with a valid javascript date. note i didn't change it to milliseconds. i just passed it in as is. for the final portion i added two date fields on my page. in chrome this is rendered nicely: based on these, i can then create an indexeddb range of either bounds, lowerbounds, or upperbounds. i.e., give me crap either after a date, before a date, or inside a date range. function dosearch() { var fromdate = document.queryselector("#fromdate").value; var todate = document.queryselector("#todate").value; var range; if(fromdate == "" && todate == "") return; var transaction = db.transaction(["data"],"readonly"); var store = transaction.objectstore("data"); var index = store.index("created"); if(fromdate != "") fromdate = new date(fromdate); if(todate != "") todate = new date(todate); if(fromdate != "" && todate != "") { range = idbkeyrange.bound(fromdate, todate); } else if(fromdate == "") { range = idbkeyrange.upperbound(todate); } else { range = idbkeyrange.lowerbound(fromdate); } var s = ""; index.opencursor(range).onsuccess = function(e) { var cursor = e.target.result; if(cursor) { s += "key "+cursor.key+""; for(var field in cursor.value) { s+= field+"="+cursor.value[field]+""; } s+=""; cursor.continue(); } document.queryselector("#status").innerhtml = s; } } the only conversion required here was to take the user input and turn it into "real" date objects. once done, everything works great: you can run the full demo below.

My team and I are creating a services platform based on a set of RESTful JSON services where each service contributes to the platform by providing distinct feature(s) and/or data. With logs being generated all over the place, we thought it was a good idea to centralize logging and perhaps also provide a rudimentary log viewer that allowed us to view, filter, sort and search our logs. We also wanted our logging to be asynchronous as we didn’t want our services to be held up while trying to write logs say maybe directly to a database. The strategy for achieving this was straight forward. Setup ActiveMQ Create a log4j appender that writes logs to the queue (log4j ships with one such appender but lets write our own. Write a message listener that reads logs from a JMS queue setup on an MQ server and persists them Let’s take a look one by one. Setup ActiveMQ Setting up an external ActiveMQ server is simple enough. A great tutorial is available at http://servicebus.blogspot.com/2011/02/installing-apache-active-mq-on-ubuntu.html to set it up on Ubuntu. You can also choose to embed a message broker within your application. Spring makes this easy. We will see how later. Creating a Lo4j JMS appender First, we create a log4j JMS appender. log4j ships with one such appender (that writes to a JMS topic instead of a queue) import javax.jms.DeliveryMode; import javax.jms.Destination; import javax.jms.MessageProducer; import javax.jms.ObjectMessage; import javax.jms.Session; import org.apache.activemq.ActiveMQConnectionFactory; import org.apache.log4j.Appender; import org.apache.log4j.AppenderSkeleton; import org.apache.log4j.Logger; import org.apache.log4j.PatternLayout; import org.apache.log4j.spi.LoggingEvent; /** * JMSQueue appender is a log4j appender that writes LoggingEvent to a queue. * @author faheem * */ public class JMSQueueAppender extends AppenderSkeleton implements Appender{ private static Logger logger = Logger.getLogger("JMSQueueAppender"); private String brokerUri; private String queueName; @Override public void close() { } @Override public boolean requiresLayout() { return false; } @Override protected synchronized void append(LoggingEvent event) { try { ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory( this.brokerUri); // Create a Connection javax.jms.Connection connection = connectionFactory.createConnection(); connection.start();np // Create a Session Session session = connection.createSession(false,Session.AUTO_ACKNOWLEDGE); // Create the destination (Topic or Queue) Destination destination = session.createQueue(this.queueName); // Create a MessageProducer from the Session to the Topic or Queue MessageProducer producer = session.createProducer(destination); producer.setDeliveryMode(DeliveryMode.NON_PERSISTENT); ObjectMessage message = session.createObjectMessage(new LoggingEventWrapper(event)); // Tell the producer to send the message producer.send(message); // Clean up session.close(); connection.close(); } catch (Exception e) { e.printStackTrace(); } } public void setBrokerUri(String brokerUri) { this.brokerUri = brokerUri; } public String getBrokerUri() { return brokerUri; } public void setQueueName(String queueName) { this.queueName = queueName; } public String getQueueName() { return queueName; } } Lets see whats happening here. Line 19: We implement the Log4J appender interface that asks us to implement three methods. requiresLayout, close and append. We will keep things simple for the moment and implement the append method which gets called whenever a method call to the logger is made. Line 37: log4j calls the append method and passes a LoggingEvent object as a parameter which represents a call to a logger. A LoggingEvent object encapsulates all information about every log item. Line 41 & 42: Create a new connection factory by providing it with a uri of a JMS, in our case activemq, server Line 45, 46 and 49: We establish a connection and a session to the JMS server. A Session can be opened in several modes. An Auto_Acknowledge session is one in which the acknowledgment of message happens automatically. Other modes include Client_Acknowledge in which a client has to explicitly acknowledge receipt and/or processing of a message and two other modes. For details, refer to the docs at http://download.oracle.com/javaee/1.4/api/javax/jms/Session.html Line 52: Create a queue. Send the queue name to connect to as a parameter. Line 56: We set the delivery mode to Non_Persistent. The other option is Persistent where the message is persisted to a persistent store. Persistent mode slows down but adds reliability to the message transfer. Line 58: We are doing multiple things. First of all I am wrapping the LoggingEvent object into a LoggingEventWrapper. This is because there are some properties within the LoggingEvent object that are not serializeable and also because I want to capture some additional information such as IP address and host name. Next, using the JMS session object, I prepare an object (the wrapper) for transport. Line 61: I send the object to the queue. Below is the code for the wrapper. import java.io.Serializable; import java.net.InetAddress; import java.net.UnknownHostException; import org.apache.log4j.EnhancedPatternLayout; import org.apache.log4j.spi.LoggingEvent; /** * Logging Event Wraps a log4j LoggingEvent object. Wrapping is required by some information is lost * when the LoggingEvent is serialized. The idea is to extract all information required from the LoggingEvent * object, place it in the wrapper and then serialize the LoggingEventWrapper. This way all required data remains * available to us. * @author faheem * */ public class LoggingEventWrapper implements Serializable{ private static final String ENHANCED_PATTERN_LAYOUT = "%throwable"; private static final long serialVersionUID = 3281981073249085474L; private LoggingEvent loggingEvent; private Long timeStamp; private String level; private String logger; private String message; private String detail; private String ipAddress; private String hostName; public LoggingEventWrapper(LoggingEvent loggingEvent){ this.loggingEvent = loggingEvent; //Format event and set detail field EnhancedPatternLayout layout = new EnhancedPatternLayout(); layout.setConversionPattern(ENHANCED_PATTERN_LAYOUT); this.detail = layout.format(this.loggingEvent); } public Long getTimeStamp() { return this.loggingEvent.timeStamp; } public String getLevel() { return this.loggingEvent.getLevel().toString(); } public String getLogger() { return this.loggingEvent.getLoggerName(); } public String getMessage() { return this.loggingEvent.getRenderedMessage(); } public String getDetail() { return this.detail; } public LoggingEvent getLoggingEvent() { return loggingEvent; } public String getIpAddress() { try { return InetAddress.getLocalHost().getHostAddress(); } catch (UnknownHostException e) { return "Could not determine IP"; } } public String getHostName() { try { return InetAddress.getLocalHost().getHostName(); } catch (UnknownHostException e) { return "Could not determine Host Name"; } } } The Message Listener The message listener “listens” to the queue (or topic). Whenever a new message is added to the queue, the onMessage method is called. import javax.jms.JMSException; import javax.jms.Message; import javax.jms.MessageListener; import javax.jms.ObjectMessage; import org.apache.log4j.Logger; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.stereotype.Component; @Component public class LogQueueListener implements MessageListener { public static Logger logger = Logger.getLogger(LogQueueListener.class); @Autowired private ILoggingService loggingService; public void onMessage( final Message message ) { if ( message instanceof ObjectMessage ) { try{ final LoggingEventWrapper loggingEventWrapper = (LoggingEventWrapper)((ObjectMessage) message).getObject(); loggingService.saveLog(loggingEventWrapper); } catch (final JMSException e) { logger.error(e.getMessage(), e); } catch (Exception e) { logger.error(e.getMessage(),e); } } } } Line 23: Checking if the object being picked off the queue is an instance of ObjectMessage Line 26: Extracting LoggingEventWrapper from the Message Line 27: Call a service method to persist the log Wiring up in Spring Lines 5-9: Use the broker tag to setup an embedded message broker. Since I am using an external one, I don’t need it. Line 12: Mention the name of the queue you want to connect to. Line 14: URI of the Broker Server. Line 15-19: Connection Factory setup Line 26-28: Message Listener Setup where we specify the number of concurrent threads that can consume messages off the queue. Of course, the above example will not work out of the box. You still have to include all JMS dependencies and implement the service that persists logs. But I hope it gives you a decent idea.