Introduction When using Spring to asynchronously execute pieces of your code you typically use the @Async annotation on your Spring @Components methods. When using the @Async annotation Spring will use AOP (Aspect Oriented Programming) to wrap your call in a Runnable. This Runnable will then be scheduled for execution on a TaskExecutor. Exception Handling Spring comes with a number of pre-packaged TaskExecutors which are documented in the Spring documentation here. In practice you will likely use the ThreadPoolTaskExecutor or the SimpleAsyncTaskExecutor. When doing so you might at times wonder why a task has not been executed. This happens when an exception occurs inside the method you are trying to execute asynchronous. The aforementioned task executors do not handle the exceptions so the execution fails silently. This problem can be solved by implementing your own AsyncTaskExecutor which handles the exceptions by logging them (or in any other way you wish). Bellow you'll find an example implementation of such a AsyncTaskExecutor. package nl.jborsje.blog.examples; import java.util.concurrent.Callable; import java.util.concurrent.Future; import org.springframework.core.task.AsyncTaskExecutor; import org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor; public class ExceptionHandlingAsyncTaskExecutor implements AsyncTaskExecutor { private AsyncTaskExecutor executor; public ExceptionHandlingAsyncTaskExecutor(AsyncTaskExecutor executor) { this.executor = executor; } public void execute(Runnable task) { executor.execute(createWrappedRunnable(task)); } public void execute(Runnable task, long startTimeout) { executor.execute(createWrappedRunnable(task), startTimeout); } public Future submit(Runnable task) { return executor.submit(createWrappedRunnable(task)); } public Future submit(final Callable task) { return executor.submit(createCallable(task)); } private Callable createCallable(final Callable task) { return new Callable() { public T call() throws Exception { try { return task.call(); } catch (Exception ex) { handle(ex); throw ex; } } }; } private Runnable createWrappedRunnable(final Runnable task) { return new Runnable() { public void run() { try { task.run(); } catch (Exception ex) { handle(ex); } } }; } private void handle(Exception ex) { System.err.println("Error during @Async execution: " + ex); } } This task executor decorates another task executor and wraps all Runnables and Callables in versions which handle the exception. In order to be able to use this task executor it has to be configured in the Spring application context as a managed bean and it has to be passed the real executor (the one that does the actual work) in its constructor. An example of such a configuration can be found bellow. JUnit testing In the application context used for JUnit testing Spring beans the asynchronous execution of methods can be cumbersome. You typically want to validate the inner workings of your methods not Springs asynchronicity. To achieve this you can use the SyncTaskExecutor provided by Spring. This task executor runs the tasks it is given immediately on the thread that adds the task instead of using a thread pool and running the tasks asynchronous. This task executor is configured in the test scope application context as follows.

This technical tip shows how .NET developers can export Microsoft Visio diagram to XML inside their own applications using Aspose.Diagram for .NET. Aspose.Diagram for .NET lets you export diagrams to a variety of formats: image formats, HTML, SVG, SWF and XML formats: VDX defines an XML diagram. VTX defines an XML template. VSX defines an XML stencil. The Diagram class' constructors read a diagram and the Save method is used to save, or export, a diagram in a different file format. The code snippets in this article show how to use the Save method to save a Visio file to VDX, VTX and VSX. Exporting VSD to VDX VDX is a schema-based XML file format that lets you save diagrams in a format that products other than Microsoft Visio can read. It's a useful format for transferring diagrams between software applications and retaining editable data. To export a VSD diagram to VDX first create an instance of the Diagram class and call the Diagram class' Save method to write the Visio drawing file to VDX. Exporting from VSD to VSX VSX is an XML format for defining stencils, the basic objects from which a diagram is built up. When a Visio file is converted to VSX, only the stencils are exported. To export a VSD diagram to VSX first you need to create an instance of the Diagram class and then call the Diagram class' Save method to write the Visio drawing file to VSX. //The Sample code shows how to export VSD to VDX //[C# Sample] //Call the diagram constructor to load diagram from a VSD file Diagram diagram = new Diagram("D:\\Drawing1.vsd"); this.Response.Clear(); this.Response.ClearHeaders(); this.Response.ContentType = "application/vnd.ms-visio"; this.Response.AppendHeader("Content-Disposition", "attachment; filename=Diagram.vdx"); this.Response.Flush(); System.IO.Stream vdxStream = this.Response.OutputStream; //Save input VSD as VDX diagram.Save(vdxStream, SaveFileFormat.VDX); this.Response.End(); //[VB.NET Code Sample] 'Call the diagram constructor to load diagram from a VSD file Dim diagram As New Diagram("D:\Drawing1.vsd") Me.Response.Clear() Me.Response.ClearHeaders() Me.Response.ContentType = "application/vnd.ms-visio" Me.Response.AppendHeader("Content-Disposition", "attachment; filename=Diagram.vdx") Me.Response.Flush() Dim vdxStream As System.IO.Stream = Me.Response.OutputStream 'Save inpupt VSD as VDX diagram.Save(vdxStream, SaveFileFormat.VDX) Me.Response.End() //The Sample code shows how to export VSD to VSX format. [C# Code Sample] // Call the diagram constructor to load diagram from a VSD file Diagram diagram = new Diagram("D:\\Drawing1.vsd"); this.Response.Clear(); this.Response.ClearHeaders(); this.Response.ContentType = "application/vnd.ms-visio"; this.Response.AppendHeader("Content-Disposition", "attachment; filename=Diagram.vsx"); this.Response.Flush(); System.IO.Stream vsxStream = this.Response.OutputStream; //Save input VSD as VSX diagram.Save(vsxStream, SaveFileFormat.VSX); this.Response.End() //[VB.NET Code Sample] 'Call the diagram constructor to load diagram from a VSD file Dim diagram As New Diagram("D:\Drawing1.vsd") Me.Response.Clear() Me.Response.ClearHeaders() Me.Response.ContentType = "application/vnd.ms-visio" Me.Response.AppendHeader("Content-Disposition", "attachment; filename=Diagram.vsx") Me.Response.Flush() Dim vsxStream As System.IO.Stream = Me.Response.OutputStream 'Save input VSD as VSX diagram.Save(vsxStream, SaveFileFormat.VSX) Me.Response.End()

Learn about using Mockito to create autowired fields.

Java SE 7 type inference I taught an introductory Java session on generics, and of course demonstrated the shorthand introduced in Java SE 7 for instantiating an instance of a generic type: // Java SE 6 List l = new ArrayList(); // Java SE 7 List l = new ArrayList<>(); This inference is very friendly, especially when we get into more complex collections: // This Map> m = new HashMap>(); // Becomes Map> m = new HashMap<>(); Not only the key and value type of the map, but the type of object stored in the collection used for the value type can be inferred. Of course, sometimes this inference breaks down. It so happens I ran across an interesting example of this. Imagine populating a set from a list, so as to speed up random access and remove duplicates. Something like this will work: List list = ...; // From somewhere Set nodup = new HashSet<>(list); However, this runs into trouble if the list could be null. The HashSetconstructor will not just return an empty set but will throwNullPointerException. So we need to guard against null here. Of course, like all good programmers, we seize the chance to use a ternary operator because ternary operators are cool. List list = ...; // From somewhere Set nodup = (null == list) ? new HashSet<>() : new HashSet<>(list); And here’s where inference breaks down. Because this is no longer a simple assignment, the statement new HashSet<>() can no longer use the left hand side in order to infer the type. As a result, we get that friendly error message, “Type mismatch: cannot convert from HashSet to Set”. What’s especially interesting is that inference breaks down even though the compiler knows that an object of typeSet is what is needed in order to gain agreement of types. The rules for inference are written to be conservative by doing nothing when an invalid inference might cause issues, while the compiler’s type checking is also conservative in what it considers to be matching types. Also interesting is that we only get that error message for the new HashSet<>(). The statement new HashSet<>(list) that uses the list to populate the set works just fine. This is because the inference is completed using the listparameter. Here’s the constructor: public class HashSet extends ... implements ... { ... public HashSet(Collection c) { ... } ... } The List that we pass in gets captured as Collection and this means that E is bound to String, so all is well. As a result, we wind up with the perfectly valid, if a little funny looking: List list = ...; // From somewhere Set nodup = (null == list) ? new HashSet() : new HashSet<>(list); Of course, I imagine most Java programmers do what I do, which is try to use the shortcut and then add the type parameter when the compiler complains. Following the rule about not meddling in the affairs of compilers (subtle; quick to anger), normally I would just fix it without trying very hard to understand why the compiler liked or didn’t like things done in a certain way. But this one was such a strange case I figured it was worth a longer look.

Note: the following article is purely theoretical. I don’t know if it fits a real-life use-case, but the point is just too good to miss Java’s List sorting has two flavors: one follows the natural ordering of collection objects, the other requires an external comparator. In the first case, Java assumes objects are naturally ordered. From a code point of view, this means types of objects in the list must implement the Comparable interface. For example, such is the case for String and Date objects. If this is not the case, or if objects cannot be compared to one another (because perhapsthey belong to incompatible type as both String and Date). The second case happens when the natural order is not relevant and a comparator has to be implemented. For example, strings are sorted according to the character value, meaning case is relevant. When the use-case requires a case-insensitive sort, the following code will do (using Java 8 enhanced syntax): Collections.sort(strings, (s1, s2) -> s1.compareToIgnoreCase(s2)); The Comparable approach is intrinsic, the Comparator extrinsic; the former case rigid, the latter adaptable to the required context. What applies to lists, however, cannot be applied to Java sets. Objects added to sets have to define equals() and hashCode() and both properties (one could say that it’s only one since they are so coupled together) are intrinsic. There is no way to define an equality that can change depending on the context in the JDK. Enters Trove: The Trove library provide primitive collections with similar APIs to the above. This gap in the JDK is often addressed by using the “wrapper” classes (java.lang.Integer, java.lang.Float, etc.) with Object-based collections. For most applications, however, collections which store primitives directly will require less space and yield significant performance gains. Let’s be frank, Trove is under-documented. However, it offers what is missing regarding extrinsic equality: it provides a dedicated set implementation, that accepts its own extrinsic equality abstraction. A sample code would look like that: HashingStrategy strategy = new MyCustomStrategy(); Set dates = new TCustomHashSet(strategy); A big bonus for using Trove is performance, though: It probably is the first argument to use Trove I never tested that in any context To go further, just have a look at Trove for yourself.

As follow up to previous posts covering JMS, this post will delve into more depth on Spring's CachingConnectionFactory. Spring provides two implementations of the javax.jms.ConnectionFactory interface, namely, the SingleConnectionFactory and the CachingConnectionFactory. The SingleConnectionFactory returns as you might expect the same single connection upon all calls to the createConnection() method. This is fine for certain scenarios and applications but the CachingConnectionFactory provides a more performant and scalable solution. By default, a single session is cached so for a multi threaded application you would set the sessionCacheSize to be a more suitable number although this number wouldn't reflect the true number of sessions cached as this figure refers to the size of cache per session acknowledgement type eg AUTO_ACKNOWLEDGE, CLIENT_ACKNOWLEDGE, DUPS_OK_ACKNOWLEDGE and SESSION_TRANSACTED. By default, the CachingConnectionFactory will cache the Message Producers and Message Consumers for every session. As an aside the Message Consumers are cached using keys which include the JMS selector so the more fine grained the message filter the more Message Consumers there would be, and Message Consumers aren't closed until the session is closed and removed from the pool. An alternative is to use a Listener Container for consuming messages. Also to be noted is that on creating a CachingConnectionFactory instance, the reconnect on exception flag is set to be true. This should mean that the onException method on the default ExceptionListener class gets called which will reset the connections. You can also override the default exception listener with your own implementation. The below snippet of XML shows a simple configuration of a CachingConnectionFactory:

gaurav gupta is a senior software engineer working on the bpmn workflow engine. he's created several plugins for netbeans ide, notably the js css minify compress plugin ( here ) and the jbpmn plugin, which is the topic of this interview. the plugin provides netbeans ide tools for working with business process modeling notation (bpmn). the plugin can be accessed here, including for netbeans ide 7.4 and 8.0 beta, together with screencasts and tutorials: http://plugins.netbeans.org/plugin/50735/jbpmn hi, gaurav, why did you create the bpmn plugin? business process modeling notation is an increasingly important standard for process modelling and has enjoyed high levels of adoption, so the specific intent of the jbpmn plugin is to create a bpmn modeler that supports the complete bpmn specification and can be integrated by multiple bpmn engine vendors into netbeans ide. as a result, different bpmn engine vendors will not need to create separate bpmn modelers for netbeans ide. another reason i created this plugin was to bring the netbeans community closer to the bpm community because bpmn has become the de-facto standard for business process modeling. also, when we talk about "netbeans vs eclipse", the basic conclusion is that netbeans is much more intuitive and easy to use, while eclipse has a wider range of third-party plugins support from more companies. in that context, i have taken this initiative to create the bpm plugin to help the netbeans community. what would you consider to be the best features of the bpmn plugin? the jbpmn nb modeler is a graphical modelling tool which allows the creation and editing of bpmn process diagrams. it provides debugging supports for jbpm 5.0. it is bpmn engine vendor neutral, can be used by any vendor or generated xml, and can run on any bpmn engine which adopts the bpmn 2.0 standard. here's another screenshot: what are the future development plans for this plugin? to cover the complete bpmn ssecification with user friendly gui and properties support. jbpmn currently supports only the bpmn process model, it will also support the bpmn conversation model. it will also provide support to extends the modeller with your own palette elements, properties, and generated xml tags. it will provides debugging supports for jbpm 6.0, activities, and also extensions so that any bpmn engine vendor can integrate debugging functionality within jbpmn. it will also provide a netbeans modeler platform api, which any business modeller can use to easily build solutions such as a bpmn conversation model.

Learn how to set up a four node Hadoop cluster using AWS EC2, PuTTy(gen), and WinSCP.

i’ve been pushing this branching model for something like 14 years now. it’s nice to see facebook say a little more about their trunk based development . of course they’re not doing it because they read anything i wrote, as the practice isn’t mine, it’s been hanging around in the industry for many years, but always as bridesmaid so to speak. if not trunk, what? mainline? mainline as popularized by clearcase is what we’re trying to kill. at least historically. it’s very different to trunk based development, and even having vastly improved merge tools doesn’t make it better – you still risk regressions, and huge nerves around ordering of releases. clearcase’s best-practices also foisted a ‘many repos’ (vobs) on teams using it, and that courted the whole conway’s law prophesy. i mentioned conway’s law before in scaling trunk based development and it concerns undue self-importance of teams around arbitrary separations. multiple small repos for a dvcs ? there is a great statement by a reddit user in the programming section of reddit, in conjunction with the facebook announcement: comment ref or all comments this redditor is right, there’s a lack of atomicity around a many-repos design, that stymies bisect. it could be that git subtrees (not submodules) are a way of getting that back (thanks @chris_stevenson on a back channel). there’s also a real problem moving code easily between repos (with history) though @offbytwo (back channel again) points out that subtrees carefully used can help do that. trunk at google vs facebook tuesday’s announcement was from facebook, and to give some balance, there’s deeper info on google’s trunk design in: google’s scaled trunk based development . subsetting the trunk for checkouts tl;dr: different google have many thousands of buildable and deployable things, which have very different release schedules. facebook don’t as they substantially have the php web-app, and apps for ios and android in different repos. well at least the main php web-app is in the mercurial trunk they talked about on tuesday. i’m not sure how the ios and android apps are managed, but at least the android one is outside the main trunk. google subset their trunk. i posted about that on monday . in that article i pointed out that the checkout can grow (or shrink) depending on the nature of the change being undertaken. it’s very different to a multiple-small-repos design. facebook don’t subset their trunk on checkout, as they do not need to; the head revisions of everything in that trunk are not big enough for a c: drive or ide to buckle. there’s also no compile stage for php , for regular development work. maximized sharing of code tl;dr: the same code is shared using globbed directories within the source tree. it’s shared as source files, in situ, rather than classes in a jar (or equivalent). refactoring tl;dr: the same developers take on refactorings where appropriate. sure it means a bigger atomic commit, but knowing all the affected source is in front of you as you do the refactoring is comforting. at least, knowing that if intellij (or eclipse, etc) completes the refactoring there’s a very strong possibility that the build will stay green, and that you’re only going to have a slight impact on other people’s working copy, and only if they are concurrently editing the same files. bigger refactoring probably still require a warning email. super tooling of the build phase tl;dr: the same google have what amounts to a super-computer doing the compilation for them (all languages that are compiled). all developers and all ci daemons leverage it. and by effective super-computer, i mean previous-compiled bits and pieces are pulled out of an internal cloud-map-thing for source permutations that have been compiled before. the distributed hashmap is possibly lru centric rather that everything forever. facebook don’t have that big hashmap of recently compiled bits and pieces, but they do have hiphop in the toolchain (originally a php to c++ compiler) which is interesting because at face value php is an interpreted language and ‘compile’ makes no sense. hiphop was created to reduce the server footprint and requirements for production deployments, while still being 100% functionally identical to the interpreted php app. it’s also faster in production. more recently hiphop became a virtual machine. it continues to be incrementally improved. like google, facebook can measure cost-benefit of continued work on it (prod rack space & prod electricity vs developer salaries). source-control weapons of choice tl;dr: different google use perforce for their trunk (with additional tooling), and many (but not all) developers use git on their local workstation to gain local-branching with an inhouse developed bridge for interop with perforce. facebook uses mercurial with additional tooling for the central server/repo. it’s unclear whether developers, by habit, exist with the mercurial client software or use git which can interop with mercurial backends. both google and facebook do trunk based development of course. branches & merge pain tl;dr: the same they don’t have merge pain, because as a rule developers are not merging to/from branches. at least up to the central repo’s server they are not. on workstations, developers may be merging to/from local branches, and rebasing when the push something that’s “done” back to the central repo. release engineers might cherry-pick defect fixes from time to time, but regular developers are not merging (you should not count to-working-copy merges) eating own dog-food tl;dr: mostly different all staff at facebook use a not-live-yet version of the web-app for all of their communication, documentation, management etc. if there’s a bug everyone feels it – though selenium2 functional tests and zillions of unit-tests guard against that happening too often. google has too many different apps for the team making each to be said to be a daily user of it. for example the adsense developer may use a dog-food version of gmail, but they are making adsense, so are hardly hurting themselves as they are not minute by minute using the interface as part of their regular existence at google. code review tl;dr: same both google and facebook insist on code reviews before the commit is accepted into the remote repo’s trunk for all others to use. there’s no mechanism of code review that’s more efficient or effective. google back in 2009 were pivoting incoming changes to the trunk around the code-review process managed by mondrian. i wrote about that in “continuous review #1” in december . i think they are unchanged in that respect: developers actively push their commit after a code review has been completed. facebook have just flipped to mercurial (from subversion). in the article linked to at the top of the page, facebook have not mentioned “pull request” or “patch queue”, or indeed “code review”. the article was mostly about speed, robustness and scale. i suspect they are sitting within the semantics of mercurials patch-queue processing though, although assigning a bot to it rather than a human. update: simon stewart pinged me and reminded me that they use (and made) phabricator. he spoke about it in a mobile@scale presentation, and that video is here . in the video he says the review is queue based now, but that they experimenting with landing the change sets into the master now. the video is from november, and was for the android + ios platforms, but it is likely to be used today for the main trunk for the php web-app. automated testing tl;dr: same heavy reliance on unit tests (not necessarily made in a tdd style). later in an build pipeline, selenium2 tests (for web-apps at least) kick in to guard the functional quality of deployed app. manual qa tl;dr: mostly the same both companies have progressively moved way from manual qa and dedicated testing professionals, towards developers testing their own stuff at discrete moments (note the dog-food item above too). prod release frequency tl;dr: it varies. facebook for the main web app, are twice a day presently (at least on weekdays). i published info on that at the start of last year. google have many apps with different release schedules, and some are “many times a day”, while others are “planned releases every few weeks”. many are in between. prod db deployment tl;dr: mostly the same database (or equivalent) table shapes (or equivalent) are designed to be forwards/backwards compatible as far as possible. pull requests as part of workflow tl;dr: same etsy, github, and other high throughput organizations are trunking by some definition, but using pull-requests to merge in things being done. it has different obligations if done, but google and facebook are not doing this in their trunks – they both essentially push (after review). refer the ‘code review’ section above. common code ownership tl;dr: the same you can commit to any part of the source tree, provided it passed a fair code review. notional owners of directories within the source tree take a boy-scout pledge to do their best with unsolicited incoming change-lists. there are strong permissions in the google perforce implementation, but the pledge means that contributions are not often rejected if the merit is there. build is ever broken tl;dr: the same almost never. directionality of merge for prod bug fixes tl;dr: the same trunk receives the defect fix, it gets cherry picked to the release branch. the release branch might have been made from a tag, if it didn’t exist before. binary dependencies tl;dr: the same checked into source-control without version suffixing (harmonized versions across all apps). e.g. – log4j.jar rather than log4j-1.2.8.jar.

One of the great things about git is how fast it is. You can create a new branch, or switch to another branch, almost as fast as you can type the command. This tends to lower the impedance of branching. As a result, many individuals and teams will naturally converge on a process where they create many, many branches. If you’re like me, you may have 30 branches at any given time. This can make viewing all the branches unwieldy. Once I week or so, I would go on a branch deletion spree by manually copying and pasting multiple branch names into a git branch -D statement. The basic use case is that you want to delete any branches that are already merged into master. Here is a python script that automated just that. from subprocess import check_output import sys def get_merged_branches(): ''' a list of merged branches, not couting the current branch or master ''' raw_results = check_output('git branch --merged upstream/master', shell=True) return [b.strip() for b in raw_results.split('\n') if b.strip() and not b.startswith('*') and b.strip() != 'master'] def delete_branch(branch): return check_output('git branch -D %s' % branch, shell=True).strip() if __name__ == '__main__': dry_run = '--confirm' not in sys.argv for branch in get_merged_branches(): if dry_run: print branch else: print delete_branch(branch) if dry_run: print '*****************************************************************' print 'Did not actually delete anything yet, pass in --confirm to delete' print '*****************************************************************' To print the branches that would be deleted, just execute python delete_merged_branches.py. To actually delete the branches, execute python delete_merged_branches.py --confirm.

Nginx-Clojure is a Nginx module for embedding Clojure or Java programs, typically those Ring based handlers.

Caching allows you to significantly speed up applications with only little effort. Two great cache implementations for the Java platform are the Guava caches and Ehcache. While Ehcache is much richer in features (such as its Searchable API, the possibility of persisting caches to disk or overflowing to big memory), it also comes with quite an overhead compared to Guava. In a recent project, I found a need to overflow a comprehensive cache to disk but at the same time, I regularly needed to invalidate particular values of this cache. Because Ehcache's Searchable API is only accessible to in-memory caches, this put me in quite a dilemma. However, it was quite easy to extend a Guava cache to allow overflowing to disk in a structured manner. This allowed me both overflowing to disk and the required invalidation feature. In this article, I want to show how this can be achieved. I will implement this file persisting cache FilePersistingCache in form of a wrapper to an actual Guava Cache instance. This is of course not the most elegant solution (more elegant would to implement an actual Guava Cache with this behavior), but I will do for most cases. To begin with, I will define a protected method that creates the backing cache I mentioned before: private LoadingCache makeCache() { return customCacheBuild() .removalListener(new PersistingRemovalListener()) .build(new PersistedStateCacheLoader()); } protected CacheBuilder customCacheBuild(CacheBuilder cacheBuilder) { return CacheBuilder.newBuilder(); } The first method will be used internally to build the necessary cache. The second method is supposed to be overridden in order to implement any custom requirement to the cache as for example an expiration strategy. This could for example be a maximum value of entries or soft references. This cache will be used just as any other Guava cache. The key to the cache's functionality are the RemovalListener and the CacheLoader that are used for this cache. We will define these two implementation as inner classes of the FilePersistingCache: private class PersistingRemovalListener implements RemovalListener { @Override public void onRemoval(RemovalNotification notification) { if (notification.getCause() != RemovalCause.COLLECTED) { try { persistValue(notification.getKey(), notification.getValue()); } catch (IOException e) { LOGGER.error(String.format("Could not persist key-value: %s, %s", notification.getKey(), notification.getValue()), e); } } } } public class PersistedStateCacheLoader extends CacheLoader { @Override public V load(K key) { V value = null; try { value = findValueOnDisk(key); } catch (Exception e) { LOGGER.error(String.format("Error on finding disk value to key: %s", key), e); } if (value != null) { return value; } else { return makeValue(key); } } } As obvious from the code, these inner classes call methods of FilePersistingCache we did not yet define. This allows us to define custom serialization behavior by overriding this class. The removal listener will check the reasons for a cache entry being evicted. If the RemovalCause is COLLECTED, the cache entry was not manually removed by the user but it was removed as a consequence of the cache's eviction strategy. We will therefore only try to persist a cache entry if the user did not wish the entries removal. The CacheLoader will first attempt to restore an existent value from disk and create a new value only if such a value could not be restored. The missing methods are defined as follows: private V findValueOnDisk(K key) throws IOException { if (!isPersist(key)) return null; File persistenceFile = makePathToFile(persistenceDirectory, directoryFor(key)); (!persistenceFile.exists()) return null; FileInputStream fileInputStream = new FileInputStream(persistenceFile); try { FileLock fileLock = fileInputStream.getChannel().lock(); try { return readPersisted(key, fileInputStream); } finally { fileLock.release(); } } finally { fileInputStream.close(); } } private void persistValue(K key, V value) throws IOException { if (!isPersist(key)) return; File persistenceFile = makePathToFile(persistenceDirectory, directoryFor(key)); persistenceFile.createNewFile(); FileOutputStream fileOutputStream = new FileOutputStream(persistenceFile); try { FileLock fileLock = fileOutputStream.getChannel().lock(); try { persist(key, value, fileOutputStream); } finally { fileLock.release(); } } finally { fileOutputStream.close(); } } private File makePathToFile(@Nonnull File rootDir, List pathSegments) { File persistenceFile = rootDir; for (String pathSegment : pathSegments) { persistenceFile = new File(persistenceFile, pathSegment); } if (rootDir.equals(persistenceFile) || persistenceFile.isDirectory()) { throw new IllegalArgumentException(); } return persistenceFile; } protected abstract List directoryFor(K key); protected abstract void persist(K key, V value, OutputStream outputStream) throws IOException; protected abstract V readPersisted(K key, InputStream inputStream) throws IOException; protected abstract boolean isPersist(K key); The implemented methods take care of serializing and deserializing values while synchronizing file access and guaranteeing that streams are closed appropriately. The last four methods remain abstract and are up to the cache's user to implement. The directoryFor(K) method should identify a unique file name for each key. In the easiest case, the toString method of the key's K class is implemented in such a way. Additionally, I made the persist, readPersisted and isPersist methods abstract in order to allow for a custom serialization strategy such as using Kryo. In the easiest scenario, you would use the built in Java functionality which uses ObjectInputStream and ObjectOutputStream. For isPersist, you would return true, assuming that you would only use this implementation if you need serialization. I added this feature to support mixed caches where you can only serialize values to some keys. Be sure not to close the streams within the persist and readPersisted methods since the file system locks rely on the streams to be open. The above implementation will take care of closing the stream for you. Finally, I added some service methods to access the cache. Implementing Guava's Cache interface would of course be a more elegant solution: public V get(K key) { return underlyingCache.getUnchecked(key); } public void put(K key, V value) { underlyingCache.put(key, value); } public void remove(K key) { underlyingCache.invalidate(key); } protected Cache getUnderlyingCache() { return underlyingCache; } Of course, this solution can be further improved. If you use the cache in a concurrent scenario, be further aware that the RemovalListener is, other than most Guava cache method's executed asynchronously. As obvious from the code, I added file locks to avoid read/write conflicts on the file system. This asynchronicity does however imply that there is a small chance that a value entry gets recreated even though there is still a value in memory. If you need to avoid this, be sure to call the underlying cache's cleanUp method within the wrapper's get method. Finally, remember to clean up the file system when you expire your cache. Optimally, you will use a temporary folder of your system for storing your cache entries in order to avoid this problem at all. In the example code, the directory is represented by an instance field named persistenceDirectory which could for example be initialized in the constructor. Update: I wrote a clean implementation of what I described above which you can find on my Git Hub page and on Maven Central. Feel free to use it, if you need to store your cache objects on disk.

This post was originally written by Brett Lawson. So, recently I added support to our Node.js client for executing N1QL queries against your cluster, providing you are running an instance of the N1QL engine (to get a hold of the updated version of the Node.js client with this support, point npm to our github master branch at https://github.com/couchbase/couchnode). When I implemented it, I didn’t have very much to test against at the time, so I figured it would be a interesting endeavor to see how nice the Node.js’s beer-sample example would look if we used entirely N1QL queries rather than using any views. I first started by converting over the basic queries which simply selected all beers or breweries from the sample data, and then moved on to converting the live-search querying to use N1QL as well. I figured I would write a little blog post on the conversions and make some remarks about what I noticed along the way. Here is our first query: var q = { limit : ENTRIES_PER_PAGE, stale : false }; db.view( "beer", "by_name", q).query(function(err, values) { var keys = _.pluck(values, 'id'); db.getMulti( keys, null, function(err, results) { var beers = _.map(results, function(v, k) { v.value.id = k; return v.value; }); res.render('beer/index', {'beers':beers}); }) }); and the converted version: db.query( "SELECT META().id AS id, * FROM beer-sample WHERE type='beer' LIMIT " + ENTRIES_PER_PAGE, function(err, beers) { res.render('beer/index', {'beers':beers}); }); As you can see, we no longer need to do two separate operations to retrieve the list. We can execute our N1QL query which will returns all the information that we need, and formats it appropriately; rather than needing to reformat the data and add our id values, we can simply select it as part of the result set. I find the N1QL version here is much more concise and appreciate how simple it was to construct the query. I then converted the brewery listing function following a similar path, and here is what I ended up with, as you can see, it is similarly beautiful and concise: db.query( "SELECT META().id AS id, name FROM beer-sample WHERE type='brewery' LIMIT " + ENTRIES_PER_PAGE, function(err, breweries) { res.render('brewery/index', {'breweries':breweries}); }); Next I converted the searching methods. These were a bit more of a challenge as looking at the original code directly, without thinking about what it was trying to achieve, the semantics were not immediately obvious, here is a look at what it looked like: var q = { startkey : value, endkey : value + JSON.parse('"\u0FFF"'), stale : false, limit : ENTRIES_PER_PAGE } db.view( "beer", "by_name", q).query(function(err, values) { var keys = _.pluck(values, 'id'); db.getMulti( keys, null, function(err, results) { var beers = []; for(var k in results) { beers.push({ 'id': k, 'name': results[k].value.name, 'brewery_id': results[k].value.brewery_id }); } res.send(beers); }); }); Again, we have quite a bit of code to achieve something which you should expect to be quite simple. In case you can’t tell, the map/reduce query above retrieves a listing of beers whose names begin with the value entered by the user. We are going to convert this to a N1QL LIKE clause, and as an added bonus, we will allow the search term to appear anywhere in the string, instead of requiring it at the beginning: db.query( "SELECT META().id, name, brewery_id FROM beer-sample WHERE type='beer' AND LOWER(name) LIKE '%" + term + "%' LIMIT " + ENTRIES_PER_PAGE, function(err, beers) { res.send(beers); }); We have again collapsed a large amount of vaguely understandable code down to a simple and concise query. I believe this begins to show the power of N1QL and why I am personally so excited to see N1QL. There is however one caveat I noticed while doing this, and this is that similar to SQL, you need to be careful about what kind of user-data you are passing into your queries. I wrote a simple cleaning function to try and prevent any malicious intent (though N1QL is currently read-only anyways), but my cleaning code is by no means extensive. Another issue I noticed is that our second query with the LIKE clause executed significantly slower as a N1QL query then it did when using map/reduce. I believe this is simply a result of N1QL still being developer preview, and there is lots of optimizations left to be done by the N1QL team. If you want to see the fully converted source code, take a look at the n1ql branch of the beersample-node repository available here, https://github.com/couchbaselabs/beersample-node/tree/n1ql. Thanks! Brett

Introduction There are cases in which it is necessary to execute pieces of code asynchronous. An example is the sending of a (JMS) message from your system to another system. If it is not important for the database transaction whether or not the message has been sent successfully you can send the message asynchronous. The advantage is that the user does not have to wait in the front-end while the message is being send. Another example of possible asynchronous execution is the case where messages have a clear ordering. In order to try to prevent message A from being overtaken by message B you might want to schedule the sending of message B asynchronous with a delay. Spring Framework Spring supports the asynchronous execution of methods on your @Components by means of the @Async annotation. When using this annotation on a method of your @Component Spring will always execute this method asynchronous. The Spring framework will use AOP (Aspect Oriented Programming) on the Spring proxy of the @Component to wrap calls to this method in a Runnable and schedule this Runnable on a task executor. This task executor has a thread pool and when a thread in the pool becomes available the Runnable will be executed on this thread. The caller of the method annotated with @Async does not wait untill the Runnable has been executed but terminates after the Runnable has been scheduled. Return values If the method to be executed asynchronously has a return value it should return a Future in its signature. This Future can be used by the caller to track the progress of the asynchronous task and to retrieve the result of the task once it has been completely executed. The @Async method should return an ASyncResult (which implements the Future interface) containing the actual result. Transaction management If the @Async annotation is being used extra care should be taken with respect to transactions. In normal circumstances (without @Async) a transaction gets propagated through the call hierarchy from one Spring @Component to the other. However, when a @Transactional Spring @Component calls a method annotated with @Async this does not happen. The call to the asynchronous method is being scheduled and executed at a later time by a task executor and is thus handled as a 'fresh' call, i.e. without a transactional context. If the @Async method (or the @Component in which it is declared) is not @Transactional by itself Spring will not manage any needed transactions. In case the method sends a message to an ESB (Enterprise Service Bus) using JMS (or another protocol) this can lead to problems, because you probably want to use the transaction manager declared in your Spring application context (typically a JTA transaction manager). In order to make Spring manage the transaction of the @Async method either the @Component or the method itself should declare the @Transactional annotation, this way Spring will manage the transaction even if a method is being executed asynchronous. Caveats A final reminder: both @Transactional and @Async work with AOP an proxying. The methods to which these annotation are applied should be public, otherwise the annotations are not picked up by Spring. When JUnit testing methods annotated with @Async in combination with transactions and automatic rollbacks extra care should be taken, this will be the topic of another blog post in the near future.

I'm passionate about front end optimization and have been for years. My original inspiration was Steve Souders and his Even Faster Web Sites talk at OSCON 2008. Since then, I've optimized this blog, made it even faster with a new design, doubled the speed of several apps for clients and showed how to make AppFuse faster. As part of my Devoxx 2013 presentation, I showed how to do page speed optimization in a Java webapp. I developed a couple AngularJS apps last year. To concat and minify their stylesheets and scripts, I used mechanisms that already existed in the projects. On one project, it was Ant and its concat task. On the other, it was part of a Grails application, so I used the resources and yui-minify-resources plugins. The Angular project I'm working on now will be published on a web server, as well as bundled in an iOS native app. Therefore, I turned to Grunt to do the optimization this time. I found it to be quite simple, once I figured out how to make it work with Angular. Based on my findings, I submitted a pull request to add Grunt to angular-seed. Below are the steps I used to add Grunt to my Angular project. Install Grunt's command line interface with "sudo npm install -g grunt-cli". Edit package.json to include a version number (e.g. "version": "1.0.0"). Add Grunt plugins in package.json to do concat/minify/asset versioning: "grunt": "~0.4.1", "grunt-contrib-concat": "~0.3.0", "grunt-contrib-uglify": "~0.2.7", "grunt-contrib-cssmin": "~0.7.0", "grunt-usemin": "~2.0.2", "grunt-contrib-copy": "~0.5.0", "grunt-rev": "~0.1.0", "grunt-contrib-clean": "~0.5.0" Create a Gruntfile.js that runs all the plugins. module.exports = function (grunt) { grunt.initConfig({ pkg: grunt.file.readJSON('package.json'), clean: ["dist", '.tmp'], copy: { main: { expand: true, cwd: 'app/', src: ['**', '!js/**', '!lib/**', '!**/*.css'], dest: 'dist/' }, shims: { expand: true, cwd: 'app/lib/webshim/shims', src: ['**'], dest: 'dist/js/shims' } }, rev: { files: { src: ['dist/**/*.{js,css}', '!dist/js/shims/**'] } }, useminPrepare: { html: 'app/index.html' }, usemin: { html: ['dist/index.html'] }, uglify: { options: { report: 'min', mangle: false } } }); grunt.loadNpmTasks('grunt-contrib-clean'); grunt.loadNpmTasks('grunt-contrib-copy'); grunt.loadNpmTasks('grunt-contrib-concat'); grunt.loadNpmTasks('grunt-contrib-cssmin'); grunt.loadNpmTasks('grunt-contrib-uglify'); grunt.loadNpmTasks('grunt-rev'); grunt.loadNpmTasks('grunt-usemin'); // Tell Grunt what to do when we type "grunt" into the terminal grunt.registerTask('default', [ 'copy', 'useminPrepare', 'concat', 'uglify', 'cssmin', 'rev', 'usemin' ]); }; Add comments to app/index.html so usemin knows what files to process. The comments are the important part, your files will likely be different. ... A couple of things to note: 1) the copy task copies the "shims" directory from Webshims lib because it loads files dynamically and 2) setting "mangle: false" on the uglify task is necessary for Angular's dependency injection to work. I tried to use grunt-ngmin with uglify and had no luck. After making these changes, I'm able to run "grunt" and get an optimized version of my app in the "dist" folder of my project. For development, I continue to run the app from my "app" folder, so I don't currently have a need for watching and processing assets on-the-fly. That could change if I start using LESS or CoffeeScript. The results speak for themselves: from 27 requests to 5 on initial load, and only 3 requests for less than 2K after that. YSlow Page Speed No optimization 75 27 HTTP requests / 464K 55/100 Apache optimization (gzip and expires headers) 89 initial load: 26 requests / 166K primed cache: 4 requests / 40K 88/100 Apache + concat/minified/versioned files 98 initial load: 5 requests / 136K primed cache: 3 requests / 1.4K 93/100

This is a simple example of how to transfer a file using Groovy, JSch, and SFTP. @Grab(group='com.jcraft', module='jsch', version='0.1.46') import com.jcraft.jsch.* java.util.Properties config = new java.util.Properties() config.put "StrictHostKeyChecking", "no" JSch ssh = new JSch() Session sess = ssh.getSession "user", "server.domain.com", 22 sess.with { setConfig config setPassword "somecomplicatedpassword" connect() Channel chan = openChannel "sftp" chan.connect() ChannelSftp sftp = (ChannelSftp) chan; def sessionsFile = new File('c:/important_document.doc') sessionsFile.withInputStream { istream -> sftp.put(istream, "/home/brippe/Documents/important_document.doc") } chan.disconnect() disconnect() }

The biggest competitor to the Java virtual machine might be Microsoft's CLR that hosts languages such as C#. The CLR allows to write unsafe code as an entry gate for low level programming, something that is hard to achieve on the JVM. If you need such advanced functionality in Java, you might be forced to use the JNI which requires you to know some C and will quickly lead to code that is tightly coupled to a specific platform. With sun.misc.Unsafe, there is however another alternative to low-level programming on the Java plarform using a Java API, even though this alternative is discouraged. Nevertheless, several applications rely on sun.misc.Unsafe such for example objenesis and therewith all libraries that build on the latter such for example kryo which is again used in for example Twitter's Storm. Therefore, it is time to have a look, especially since the functionality of sun.misc.Unsafe is considered to become part of Java's public API in Java 9. Getting hold of an instance of sun.misc.Unsafe The sun.misc.Unsafe class is intended to be only used by core Java classes which is why its authors made its only constructor private and only added an equally private singleton instance. The public getter for this instances performs a security check in order to avoid its public use: public static Unsafe getUnsafe() { Class cc = sun.reflect.Reflection.getCallerClass(2); if (cc.getClassLoader() != null) throw new SecurityException("Unsafe"); return theUnsafe; } This method first looks up the calling Class from the current thread’s method stack. This lookup is implemented by another internal class named sun.reflection.Reflection which is basically browsing down the given number of call stack frames and then returns this method’s defining class. This security check is however likely to change in future version. When browsing the stack, the first found class (index 0) will obviously be the Reflection class itself, and the second (index 1) class will be the Unsafe class such that index 2 will hold your application class that was calling Unsafe#getUnsafe(). This looked-up class is then checked for its ClassLoader where a null reference is used to represent the bootstrap class loader on a HotSpot virtual machine. (This is documented in Class#getClassLoader() where it says that “some implementations may use null to represent the bootstrap class loader”.) Since no non-core Java class is normally ever loaded with this class loader, you will therefore never be able to call this method directly but receive a thrown SecurityException as an answer. (Technically, you could force the VM to load your application classes using the bootstrap class loader by adding it to the –Xbootclasspath, but this would require some setup outside of your application code which you might want to avoid.) Thus, the following test will succeed: @Test(expected = SecurityException.class) public void testSingletonGetter() throws Exception { Unsafe.getUnsafe(); } However, the security check is poorly designed and should be seen as a warning against the singleton anti-pattern. As long as the use of reflection is not prohibited (which is hard since it is so widely used in many frameworks), you can always get hold of an instance by inspecting the private members of the class. From the Unsafe class's source code, you can learn that the singleton instance is stored in a private static field called theUnsafe. This is at least true for the HotSpot virtual machine. Unfortunately for us, other virtual machine implementations sometimes use other names for this field. Android’s Unsafe class is for example storing its singleton instance in a field called THE_ONE. This makes it hard to provide a “compatible” way of receiving the instance. However, since we already left the save territory of compatibility by using the Unsafe class, we should not worry about this more than we should worry about using the class at all. For getting hold of the singleton instance, you simply read the singleton field's value: Field theUnsafe = Unsafe.class.getDeclaredField("theUnsafe"); theUnsafe.setAccessible(true); Unsafe unsafe = (Unsafe) theUnsafe.get(null); Alternatively, you can invoke the private instructor. I do personally prefer this way since it works for example with Android while extracting the field does not: Constructor unsafeConstructor = Unsafe.class.getDeclaredConstructor(); unsafeConstructor.setAccessible(true); Unsafe unsafe = unsafeConstructor.newInstance(); The price you pay for this minor compatibility advantage is a minimal amount of heap space. The security checks performed when using reflection on fields or constructors are however similar. Create an Instance of a Class Without Calling a Constructor The first time I made use of the Unsafe class was for creating an instance of a class without calling any of the class's constructors. I needed to proxy an entire class which only had a rather noisy constructor but I only wanted to delegate all method invocations to a real instance which I did however not know at the time of construction. Creating a subclass was easy and if the class had been represented by an interface, creating a proxy would have been a straight-forward task. With the expensive constructor, I was however stuck. By using the Unsafe class, I was however able to work my way around it. Consider a class with an artificially expensive constructor: class ClassWithExpensiveConstructor { private final int value; private ClassWithExpensiveConstructor() { value = doExpensiveLookup(); } private int doExpensiveLookup() { try { Thread.sleep(2000); } catch (InterruptedException e) { e.printStackTrace(); } return 1; } public int getValue() { return value; } } Using the Unsafe, we can create an instance of ClassWithExpensiveConstructor (or any of its subclasses) without having to invoke the above constructor, simply by allocating an instance directly on the heap: @Test public void testObjectCreation() throws Exception { ClassWithExpensiveConstructor instance = (ClassWithExpensiveConstructor) unsafe.allocateInstance(ClassWithExpensiveConstructor.class); assertEquals(0, instance.getValue()); } Note that final field remained uninitialized by the constructor but is set with its type's default value. Other than that, the constructed instance behaves like a normal Java object. It will for example be garbage collected when it becomes unreachable. The Java run time itself creates objects without calling a constructor when for example creating objects for deserialization. Therefore, the ReflectionFactory offers even more access to individual object creation: @Test public void testReflectionFactory() throws Exception { @SuppressWarnings("unchecked") Constructor silentConstructor = ReflectionFactory.getReflectionFactory() .newConstructorForSerialization(ClassWithExpensiveConstructor.class, Object.class.getConstructor()); silentConstructor.setAccessible(true); assertEquals(10, silentConstructor.newInstance().getValue()); } Note that the ReflectionFactory class only requires a RuntimePermission called reflectionFactoryAccess for receiving its singleton instance and no reflection is therefore required here. The received instance of ReflectionFactory allows you to define any constructor to become a constructor for the given type. In the example above, I used the default constructor of java.lang.Object for this purpose. You can however use any constructor: class OtherClass { private final int value; private final int unknownValue; private OtherClass() { System.out.println("test"); this.value = 10; this.unknownValue = 20; } } @Test public void testStrangeReflectionFactory() throws Exception { @SuppressWarnings("unchecked") Constructor silentConstructor = ReflectionFactory.getReflectionFactory() .newConstructorForSerialization(ClassWithExpensiveConstructor.class, OtherClass.class.getDeclaredConstructor()); silentConstructor.setAccessible(true); ClassWithExpensiveConstructor instance = silentConstructor.newInstance(); assertEquals(10, instance.getValue()); assertEquals(ClassWithExpensiveConstructor.class, instance.getClass()); assertEquals(Object.class, instance.getClass().getSuperclass()); } Note that value was set in this constructor even though the constructor of a completely different class was invoked. Non-existing fields in the target class are however ignored as also obvious from the above example. Note that OtherClass does not become part of the constructed instances type hierarchy, the OtherClass's constructor is simply borrowed for the "serialized" type. Not mentioned in this blog entry are other methods such as Unsafe#defineClass, Unsafe#defineAnonymousClass or Unsafe#ensureClassInitialized. Similar functionality is however also defined in the public API's ClassLoader. Native Memory Allocation Did you ever want to allocate an array in Java that should have had more than Integer.MAX_VALUE entries? Probably not because this is not a common task, but if you once need this functionality, it is possible. You can create such an array by allocating native memory. Native memory allocation is used by for example direct byte buffers that are offered in Java's NIO packages. Other than heap memory, native memory is not part of the heap area and can be used non-exclusively for example for communicating with other processes. As a result, Java's heap space is in competition with the native space: the more memory you assign to the JVM, the less native memory is left. Let us look at an example for using native (off-heap) memory in Java with creating the mentioned oversized array: class DirectIntArray { private final static long INT_SIZE_IN_BYTES = 4; private final long startIndex; public DirectIntArray(long size) { startIndex = unsafe.allocateMemory(size * INT_SIZE_IN_BYTES); unsafe.setMemory(startIndex, size * INT_SIZE_IN_BYTES, (byte) 0); } } public void setValue(long index, int value) { unsafe.putInt(index(index), value); } public int getValue(long index) { return unsafe.getInt(index(index)); } private long index(long offset) { return startIndex + offset * INT_SIZE_IN_BYTES; } public void destroy() { unsafe.freeMemory(startIndex); } } @Test public void testDirectIntArray() throws Exception { long maximum = Integer.MAX_VALUE + 1L; DirectIntArray directIntArray = new DirectIntArray(maximum); directIntArray.setValue(0L, 10); directIntArray.setValue(maximum, 20); assertEquals(10, directIntArray.getValue(0L)); assertEquals(20, directIntArray.getValue(maximum)); directIntArray.destroy(); } First, make sure that your machine has sufficient memory for running this example! You need at least (2147483647 + 1) * 4 byte = 8192 MB of native memory for running the code. If you have worked with other programming languages as for example C, direct memory allocation is something you do every day. By calling Unsafe#allocateMemory(long), the virtual machine allocates the requested amount of native memory for you. After that, it will be your responsibility to handle this memory correctly. The amount of memory that is required for storing a specific value is dependent on the type's size. In the above example, I used an int type which represents a 32-bit integer. Consequently a single int value consumes 4 byte. For primitive types, size is well-documented. It is however more complex to compute the size of object types since they are dependent on the number of non-static fields that are declared anywhere in the type hierarchy. The most canonical way of computing an object's size is using the Instrumented class from Java's attach API which offers a dedicated method for this purpose called getObjectSize. I will however evaluate another (hacky) way of dealing with objects in the end of this section. Be aware that directly allocated memory is always native memory and therefore not garbage collected. You therefore have to free memory explicitly as demonstrated in the above example by a call to Unsafe#freeMemory(long). Otherwise you reserved some memory that can never be used for something else as long as the JVM instance is running what is a memory leak and a common problem in non-garbage collected languages. Alternatively, you can also directly reallocate memory at a certain address by calling Unsafe#reallocateMemory(long, long) where the second argument describes the new amount of bytes to be reserved by the JVM at the given address. Also, note that the directly allocated memory is not initialized with a certain value. In general, you will find garbage from old usages of this memory area such that you have to explicitly initialize your allocated memory if you require a default value. This is something that is normally done for you when you let the Java run time allocate the memory for you. In the above example, the entire area is overriden with zeros with help of the Unsafe#setMemory method. When using directly allocated memory, the JVM will neither do range checks for you. It is therefore possible to corrupt your memory as this example shows: @Test public void testMallaciousAllocation() throws Exception { long address = unsafe.allocateMemory(2L * 4); unsafe.setMemory(address, 8L, (byte) 0); assertEquals(0, unsafe.getInt(address)); assertEquals(0, unsafe.getInt(address + 4)); unsafe.putInt(address + 1, 0xffffffff); assertEquals(0xffffff00, unsafe.getInt(address)); assertEquals(0x000000ff, unsafe.getInt(address + 4)); } Note that we wrote a value into the space that was each partly reserved for the first and for the second number. This picture might clear things up. Be aware that the values in the memory run from the "right to the left" (but this might be machine dependent). The first row shows the initial state after writing zeros to the entire allocated native memory area. Then we override 4 byte with an offset of a single byte using 32 ones. The last row shows the result after this writing operation. Finally, we want to write an entire object into native memory. As mentioned above, this is a difficult task since we first need to compute the size of the object in order to know the amount of size we need to reserve. The Unsafe class does however not offer such functionality. At least not directly since we can at least use the Unsafe class to find the offset of an instance's field which is used by the JVM when itself allocates objects on the heap. This allows us to find the approximate size of an object: public long sizeOf(Class clazz) long maximumOffset = 0; do { for (Field f : clazz.getDeclaredFields()) { if (!Modifier.isStatic(f.getModifiers())) { maximumOffset = Math.max(maximumOffset, unsafe.objectFieldOffset(f)); } } } while ((clazz = clazz.getSuperclass()) != null); return maximumOffset + 8; } This might at first look cryptic, but there is no big secret behind this code. We simply iterate over all non-static fields that are declared in the class itself or in any of its super classes. We do not have to worry about interfaces since those cannot define fields and will therefore never alter an object's memory layout. Any of these fields has an offset which represents the first byte that is occupied by this field's value when the JVM stores an instance of this type in memory, relative to a first byte that is used for this object. We simply have to find the maximum offset in order to find the space that is required for all fields but the last field. Since a field will never occupy more than 64 bit (8 byte) for a long or double value or for an object reference when run on a 64 bit machine, we have at least found an upper bound for the space that is used to store an object. Therefore, we simply add these 8 byte to the maximum index and we will not run into danger of having reserved to little space. This idea is of course wasting some byte and a better algorithm should be used for production code. In this context, it is best to think of a class definition as a form of heterogeneous array. Note that the minimum field offset is not 0 but a positive value. The first few byte contain meta information. The graphic below visualizes this principle for an example object with an int and a long field where both fields have an offset. Note that we do not normally write meta information when writing a copy of an object into native memory so we could further reduce the amount of used native memoy. Also note that this memory layout might be highly dependent on an implementation of the Java virtual machine. With this overly careful estimate, we can now implement some stub methods for writing shallow copies of objects directly into native memory. Note that native memory does not really know the concept of an object. We are basically just setting a given amount of byte to values that reflect an object's current values. As long as we remember the memory layout for this type, these byte contain however enough information to reconstruct this object. public void place(Object o, long address) throws Exception { Class clazz = o.getClass(); do { for (Field f : clazz.getDeclaredFields()) { if (!Modifier.isStatic(f.getModifiers())) { long offset = unsafe.objectFieldOffset(f); if (f.getType() == long.class) { unsafe.putLong(address + offset, unsafe.getLong(o, offset)); } else if (f.getType() == int.class) { unsafe.putInt(address + offset, unsafe.getInt(o, offset)); } else { throw new UnsupportedOperationException(); } } } } while ((clazz = clazz.getSuperclass()) != null); } public Object read(Class clazz, long address) throws Exception { Object instance = unsafe.allocateInstance(clazz); do { for (Field f : clazz.getDeclaredFields()) { if (!Modifier.isStatic(f.getModifiers())) { long offset = unsafe.objectFieldOffset(f); if (f.getType() == long.class) { unsafe.putLong(instance, offset, unsafe.getLong(address + offset)); } else if (f.getType() == int.class) { unsafe.putLong(instance, offset, unsafe.getInt(address + offset)); } else { throw new UnsupportedOperationException(); } } } } while ((clazz = clazz.getSuperclass()) != null); return instance; } @Test public void testObjectAllocation() throws Exception { long containerSize = sizeOf(Container.class); long address = unsafe.allocateMemory(containerSize); Container c1 = new Container(10, 1000L); Container c2 = new Container(5, -10L); place(c1, address); place(c2, address + containerSize); Container newC1 = (Container) read(Container.class, address); Container newC2 = (Container) read(Container.class, address + containerSize); assertEquals(c1, newC1); assertEquals(c2, newC2); } Note that these stub methods for writing and reading objects in native memory only support int and long field values. Of course, Unsafe supports all primitive values and can even write values without hitting thread-local caches by using the volatile forms of the methods. The stubs were only used to keep the examples concise. Be aware that these "instances" would never get garbage collected since their memory was allocated directly. (But maybe this is what you want.) Also, be careful when precalculating size since an object's memory layout might be VM dependent and also alter if a 64-bit machine runs your code compared to a 32-bit machine. The offsets might even change between JVM restarts. For reading and writing primitives or object references, Unsafe provides the following type-dependent methods: getXXX(Object target, long offset): Will read a value of type XXX from target's address at the specified offset. putXXX(Object target, long offset, XXX value): Will place value at target's address at the specified offset. getXXXVolatile(Object target, long offset): Will read a value of type XXX from target's address at the specified offset and not hit any thread local caches. putXXXVolatile(Object target, long offset, XXX value): Will place value at target's address at the specified offset and not hit any thread local caches. putOrderedXXX(Object target, long offset, XXX value): Will place value at target's address at the specified offet and might not hit all thread local caches. putXXX(long address, XXX value): Will place the specified value of type XXX directly at the specified address. getXXX(long address): Will read a value of type XXX from the specified address. compareAndSwapXXX(Object target, long offset, long expectedValue, long value): Will atomicly read a value of type XXX from target's address at the specified offset and set the given value if the current value at this offset equals the expected value. Be aware that you are copying references when writing or reading object copies in native memory by using the getObject(Object, long) method family. You are therefore only creating shallow copies of instances when applying the above method. You could however always read object sizes and offsets recursively and create deep copies. Pay however attention for cyclic object references which would cause infinitive loops when applying this principle carelessly. Not mentioned here are existing utilities in the Unsafe class that allow manipulation of static field values sucht as staticFieldOffset and for handling array types. Finally, both methods named Unsafe#copyMemory allow to instruct a direct copy of memory, either relative to a specific object offset or at an absolute address as the following example shows: @Test public void testCopy() throws Exception { long address = unsafe.allocateMemory(4L); unsafe.putInt(address, 100); long otherAddress = unsafe.allocateMemory(4L); unsafe.copyMemory(address, otherAddress, 4L); assertEquals(100, unsafe.getInt(otherAddress)); } Throwing Checked Exceptions Without Declaration There are some other interesting methods to find in Unsafe. Did you ever want to throw a specific exception to be handled in a lower layer but you high layer interface type did not declare this checked exception? Unsafe#throwException allows to do so: @Test(expected = Exception.class) public void testThrowChecked() throws Exception { throwChecked(); } public void throwChecked() { unsafe.throwException(new Exception()); } Native Concurrency The park and unpark methods allow you to pause a thread for a certain amount of time and to resume it: @Test public void testPark() throws Exception { final boolean[] run = new boolean[1]; Thread thread = new Thread() { @Override public void run() { unsafe.park(true, 100000L); run[0] = true; } }; thread.start(); unsafe.unpark(thread); thread.join(100L); assertTrue(run[0]); } Also, monitors can be acquired directly by using Unsafe using monitorEnter(Object), monitorExit(Object) and tryMonitorEnter(Object). A file containing all the examples of this blog entry is available as a gist.

Our best option is to create some pre-processing tool that will first split the big file in multiple smaller chunks before they are processed by the middle-ware.

Get started with Spring IDE and the Spring Tool Suite – a set of plugins to simplify the development of Spring-based applications in Eclipse.

Introduction Some time ago we decided to upgrade our application from JBoss 5 to 7 (technically 7.2). In this article I going to describe several things which we found problematic. At the end I also provided a short list of benefits we gained in retrospect. First some general information about our application. It was built using EJB 3.0 technology. We have 2 interfaces for communicating with other components – JMS and JAX-WS. We use JBoss AS 5 as our messaging broker which is started as a separate JVM process. This part of the system we were not allowed to change. Finally – we use JPA to store processing results to Oracle DB. Step #1 – Convince your Product Owner Although our application was rather small and built on JEE5 standard it took us 4 weeks to migrate it to JEE6 and JBoss 7. So you can't do it as a maintenance ticket – it's simply too big. There is always problem with providing Business Value of such migration for Product Owners as well as for key Stakeholders. There are several aspects which might help you convincing them. One of the biggest benefits is processing time. JBoss 7 is simply faster and has better caching (Infinispan over Ehcache). Another one is startup time (our server is ready to go in 5-6 seconds opposed to 1 minute in JBoss 5). Finally – development is much faster (EJB 3.1 is much better then 3.0). The last one might be translated to “time to market”. Having above arguments I'm pretty sure you'll convince them. Step #2 – Do some reading Here is a list on interesting links which are worth reading before the migration: JBoss 5 -> 7 migration guide: https://docs.jboss.org/author/display/AS7/How+do+I+migrate+my+application+from+AS5+or+AS6+to+AS7 JBoss 7 vs EAP libraries: https://access.redhat.com/site/articles/112673 JBoss EAP Faq: http://www.jboss.org/jbossas/faq Cache implementation benchmarks: http://sourceforge.net/p/nitrocache/blog/2012/05/performance-benchmark-nitrocache--ehcache--infinispan--jcs--cach4j/ JBoss 7 performence tuning: http://www.mastertheboss.com/jboss-performance/jboss-as-7-performance-tuning JBoss caching: http://www.mastertheboss.com/hibernate-howto/using-hibernate-second-level-cache-with-jboss-as-5-6-7 Step #3 – Off you go – change Maven dependencies JBoss 5 isn't packaged very well, so I suppose you many dependencies included in your classpath (either directly or by transitive dependencies). This is the first big change in JBoss 7. Now I strongly advice you to use this artifact in your dependency management section: org.jboss.as jboss-as-parent 7.2.0.Final pom import We also decided to stick only to JEE6 spec and configure all additional JBoss 7 options with proper XML files. If it sounds good for your project too, just add this dependency and you're done with this step: org.jboss.spec jboss-javaee-6.0 1.0.0.Final pom provided After cleaning up dependencies your code probably won't compile for a couple of days or even weeks. It takes time to clean this up. Step #4 – EJB 3.0 to 3.1 migration Dependency Injection is a heart of the application, so it is worth to start with it. Almost all of your code should work, but you'll have some problems with beans annotated with @Service (these are singletons with JBoss 5 EJB Extended API). You just need to replace them with @Singleton annotations and put @PostConstruct annotation on your init method. One last thing – remember to use proper concurrency strategy. We decided to use @ConcurrencyManagement(BEAN) and leave the implementation as is. Step #5 – Upgrade to JPA 2.0 If you used JPA 1.0 with Hibernate, I'm pretty sure you have a lot of non standard annotations defining caching or cascading. All of them might be successfully replaced with JPA 2.0 annotations and finally you might get rid of Hibernate from compile classpath and depend only on JPA 2.0. Here are several standard things to do: Get rid of Hibernate's Session.evict and switch to EntityManager.detach Get rid of Hibernate's @Cache annotation and replace it with @Cachable Fix Cascades (now delete orphan is a part of @XXXToYYY annotations) Remove Hibernate dependency and stick with JEE6 spec Step #6 – Fix Hibernate's sequencer Migrating Hibernate 3 to 4 is a bit tricky because of the way it uses sequences (fields annotated with @Id). Hibernate by default uses a pool of ids instead of incrementing sequence. An example will be more descriptive: Some_DB_Sequence.nextval -> 1 Hibernate 3: 1*50 = 50; IDs to be used = 50, 51, 52.…, 99 Some_DB_Sequence.nextval -> 2 Hibernate 3: 2*50 = 100; IDs to be used = 100, 101, 102.…, 149 In Hibernate 4.x there is a new sequence generator that uses new IDs that are 1:1 related to DB sequence. Typically it's disabled by default... but not in JBoss 7.1. So after migration, Hibernate tries to insert entities using IDs read from sequence (using new sequence generator) that were already used which causes constraint violation. The fastest solution is to switch Hibernate to the old method of sequence generation (described in example above), that requires following change in persistence.xml: Step #7 – Caching Infinispan is shipped with JBoss 7 and does not require much configuration. There is only one setting in persistence.xml which needs to be set and the others might be removed: Infinispan itself might require some extra configuration – just use standalone-full-ha.xml as guide. Step #8 – RMI with JBoss 5 If you're using a lot of RMI communicating with other JBoss 5 servers – I have bad information for you – JBoss 5 and 7 are totally different and this kind of comminication will not work. I strongly recommend to switch to some other technology like JAX-WS. In the retrospect we are very glad we decided to do it. Step #9 – JMS migration We thought it would be really hard to connect with JMS server based on JBoss 5. It turned out that you have 2 options and both work fine: Start HornetQ server on your own instance and create a bridge to JBoss 5 instance Use Generic JMS adapter: https://github.com/jms-ra/generic-jms-ra Step #10 – Fix EAR layout In JBoss 5 it does not matter where all jars are being placed. All EJBs are being started. It does not work with JBoss 7 anymore. All EJB which should start must be added as modules. Step #11 – JMX console Bad information – it's not present in JBoss 7. We liked it very much, but we had to switch to jvisualvm to invoke our JMX operations. There is a ticket in WildFly Jira opened for that: https://issues.jboss.org/browse/WFLY-1197. Unfortunately at moment of writing this article it is not resolved. Some thoughts in retrospect It is really time consuming task to migrate from JBoss 5 to 7. Although in my opinion it is worth it. Now we have better caching for cluster solutions (Infinispan), better DI (EJB 3.1) and better Web Services (CXF instead of JBoss WS). Processing time decreased by 25% without any code change. Development speed increased in my opinion (it is really hard to measure it) by 50% and we are much more productive (faster server restarts). Memory footprint lowered from 1GB to 512MB. Finally automatic application redeployment finally works! However there is always a price to pay – the migration took us 4 weeks (2 sprints). We didn't write any code for our business in that period. So make sure you prepare well for such migration and my last advice – invest some time to write good automatic functional tests (we use Arquillian for that). Once they're green again – you're almost crossing finishing line.