Here's a great JDK 8 command-line static dependency checker.

A Mule application which uses the Jboss TX transaction manager needs a persistent Object Store to hold the objects and states of the transactions being processed (further information about different object stores can be found in the following page). By default Mule uses the ShadowNoFileLockStorem, which uses the file system to store the objects. As one can guess, if an application does not have permission to write the object store to the file system, the Jboss Transaction Manager will not be able to work properly and will throw an exception similar to the following: com.arjuna.ats.arjuna: ARJUNA12218: cant create new instance of {0} java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ... Caused by: com.arjuna.ats.arjuna.exceptions.ObjectStoreException: ARJUNA12225: FileSystemStore::setupStore - cannot access root of object store: //ObjectStore/ShadowNoFileLockStore/defaultStore/ at com.arjuna.ats.internal.arjuna.objectstore.FileSystemStore.(FileSystemStore.java:482) at com.arjuna.ats.internal.arjuna.objectstore.ShadowingStore.(ShadowingStore.java:619) at com.arjuna.ats.internal.arjuna.objectstore.ShadowNoFileLockStore.(ShadowNoFileLockStore.java:53) ... 36 more Since the object store is not created, the XA Transaction Manager is not initialised properly. This will throw a ‘Could not initialize class’ exception whenever the transaction manager is invoked. org.mule.exception.DefaultSystemExceptionStrategy: Caught exception in Exception Strategy: errorCode: 0 javax.resource.spi.work.WorkCompletedException: errorCode: 0 at org.mule.work.WorkerContext.run(WorkerContext.java:335) at java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:2025) ... Caused by: java.lang.NoClassDefFoundError: Could not initialize class com.arjuna.ats.arjuna.coordinator.TxControl at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.begin(BaseTransaction.java:87) at org.mule.transaction.XaTransaction.doBegin(XaTransaction.java:63) at org.mule.transaction.AbstractTransaction.begin(AbstractTransaction.java:66) at org.mule.transaction.XaTransactionFactory.beginTransaction(XaTransactionFactory.java:32) at org.mule.execution.BeginAndResolveTransactionInterceptor.execute(BeginAndResolveTransactionInterceptor.java:51) at org.mule.execution.ResolvePreviousTransactionInterceptor.execute(ResolvePreviousTransactionInterceptor.java:48) at org.mule.execution.SuspendXaTransactionInterceptor.execute(SuspendXaTransactionInterceptor.java:54) at org.mule.execution.ValidateTransactionalStateInterceptor.execute(ValidateTransactionalStateInterceptor.java:44) at org.mule.execution.IsolateCurrentTransactionInterceptor.execute(IsolateCurrentTransactionInterceptor.java:44) at org.mule.execution.ExternalTransactionInterceptor.execute(ExternalTransactionInterceptor.java:52) at org.mule.execution.RethrowExceptionInterceptor.execute(RethrowExceptionInterceptor.java:32) at org.mule.execution.RethrowExceptionInterceptor.execute(RethrowExceptionInterceptor.java:17) at org.mule.execution.TransactionalErrorHandlingExecutionTemplate.execute(TransactionalErrorHandlingExecutionTemplate.java:113) at org.mule.execution.TransactionalErrorHandlingExecutionTemplate.execute(TransactionalErrorHandlingExecutionTemplate.java:34) at org.mule.transport.jms.XaTransactedJmsMessageReceiver.poll(XaTransactedJmsMessageReceiver.java:214) at org.mule.transport.AbstractPollingMessageReceiver.performPoll(AbstractPollingMessageReceiver.java:219) at org.mule.transport.PollingReceiverWorker.poll(PollingReceiverWorker.java:84) at org.mule.transport.PollingReceiverWorker.run(PollingReceiverWorker.java:53) at org.mule.work.WorkerContext.run(WorkerContext.java:311) ... 15 more Mule computes the default directory where to write the object store as follows : muleInternalDir = config.getWorkingDirectory(); (see the code for further analysis). If Mule is started from a directory where the user does not have write permissions, the problems mentioned above will be faced. The easiest way to fix this issue is to make sure that the user running Mule as full write permission to the working directory. If that cannot be achieved, fear not, there is a solution. On first analysis, one would be tempted to set the Object Store Directory by using Spring properties as follows: Unfortunately this will not work since the Jboss Transaction Manager is a Singleton and this property is used in the constructor of the object. Hence a behaviour similar to the following will be experienced: Caused by: com.arjuna.ats.arjuna.exceptions.ObjectStoreException: ARJUNA12225: FileSystemStore::setupStore - cannot access root of object store: PutObjectStoreDirHere/ShadowNoFileLockStore/defaultStore/ (Please note that “PutObjectStoreDirHere” is the default directory assigned by the JBoss TX transaction manager). One way to go around this issue is to be sure that these properties are set before the object is initialised. There are at least two ways to be sure that this is achieved: 1. Set the properties on start up as follows: ./mule -M-Dcom.arjuna.ats.arjuna.objectstore.objectStoreDir=/path/to/objectstoreDir -M-DObjectStoreEnvironmentBean.objectStoreDir=/path/to/objectstoreDir 2. Set the properties in the wrapper.config as follow: wrapper.java.additional.x=-Dcom.arjuna.ats.arjuna.objectstore.objectStoreDir=/path/to/objectstoreDir wrapper.java.additional.x=-DObjectStoreEnvironmentBean.objectStoreDir=/path/to/objectstoreDir (x is the next number available in the wrapper.config by default this is 4). Otherwise, take the easiest route and make sure that Mule can write to the start up directory.

Postgres and Oracle compatibility with Hibernate There are situations your JEE application needs to support Postgres and Oracle as a Database. Hibernate should do the job here, however, there are some specifics worth mentioning. While enabling Postgres for application already running Oracle I came across following tricky parts: BLOBs support, CLOBs support, Oracle not knowing Boolean type (using Integer) instead and DUAL table. These were the tricks I had to apply to make the @Entity classes running on both of these. Please note I’ve used Postgres 9.3 with Hibernate 4.2.1.SP1. BLOBs support The problem with Postgres is that it offers 2 types of BLOB storage: bytea - data stored in table oid - table holds just identifier to data stored elsewhere I guess in the most of the situations you can live with the bytea as well as I did. The other one as far as I’ve read is to be used for some huge data (in gigabytes) as it supports streams for IO operations. Well, it sounds nice there is such a support, however using Hibernate in this case can make things quite problematic (due to need to use the specific annotations), especially if you try to achieve compatibility with Oracle. To see the trouble here, see StackOverflow: proper hibernate annotation for byte[] All- the combinations are described there: annotation postgres oracle works on ------------------------------------------------------------- byte[] + @Lob oid blob oracle byte[] bytea raw(255) postgresql byte[] + @Type(PBA) oid blob oracle byte[] + @Type(BT) bytea blob postgresql where @Type(PBA) stands for: @Type(type="org.hibernate.type.PrimitiveByteArrayBlobType") and @Type(BT) stands for: @Type(type="org.hibernate.type.BinaryType"). These result in all sorts of Postgres errors, like: ERROR: column “foo” is of type oid but expression is of type bytea or ERROR: column “foo” is of type bytea but expression is of type oid Well, there seems to be a solution, still it includes patching of Hibernate library (something I see as the last option when playing with 3.rd party library). There is also a reference to official blog post from the Hibernate guys on the topic: PostgreSQL and BLOBs. Still solution described in blog post seems not working for me and based on the comments, seems to be invalid for more people. BLOBs solved OK, so now the optimistic part. After quite some debugging I ended up with the Entity definition like this : @Lob private byte[] foo; Oracle has no trouble with that, moreover I had to customize the Postgres dialect in a way: public class PostgreSQLDialectCustom extends PostgreSQL82Dialect { @Override public SqlTypeDescriptor remapSqlTypeDescriptor(SqlTypeDescriptor sqlTypeDescriptor) { if (sqlTypeDescriptor.getSqlType() == java.sql.Types.BLOB) { return BinaryTypeDescriptor.INSTANCE; } return super.remapSqlTypeDescriptor(sqlTypeDescriptor); } } That’s it! Quite simple right? That works for persisting to bytea typed columns in Postgres (as that fits my usecase). CLOBs support The errors in misconfiguration looked something like this: org.postgresql.util.PSQLException: Bad value for type long : ... So first I’ve found (on String LOBs on PostgreSQL with Hibernate 3.6) following solution: @Lob @Type(type = "org.hibernate.type.TextType") private String foo; Well, that works, but for Postgres only. Then there was a suggestion (on StackOverflow: Postgres UTF-8 clobs with JDBC) from to go for: @Lob @Type(type="org.hibernate.type.StringClobType") private String foo; That pointed me the right direction (the funny part was that it was just a comment to some answers). It was quite close, but didn’t work for me in all cases, still resulted in errors in my tests. CLOBs solved The important was @deprecation javadocs in the org.hibernate.type.StringClobType that brought me to working one: @Lob @Type(type="org.hibernate.type.MaterializedClobType") private String foo; That works for both Postgres and Oracle, without any further hacking (on Hibernate side) needed. Boolean type Oracle knows no Boolean type and the trouble is that Postgres does. As there was also some plain SQL present, I ended up In Postgres with error: ERROR: column “foo” is of type boolean but expression is of type integer I decided to enable cast from Integer to Boolean in Postgres rather than fixing all the plain SQL places (in a way found in Forum: Automatically Casting From Integer to Boolean): update pg_cast set castcontext = 'i' where oid in ( select c.oid from pg_cast c inner join pg_type src on src.oid = c.castsource inner join pg_type tgt on tgt.oid = c.casttarget where src.typname like 'int%' and tgt.typname like 'bool%'); Please note you should run the SQL update by user with provileges to update catalogs (probably not your postgres user used for DB connection from your application), as I’ve learned on Stackoverflow: Postgres - permission denied on updating pg_catalog.pg_cast. DUAL table There is one more specific in the Oracle I came across. If you have plain SQL, in Oracle there is DUAL table provied (see more info on Wikipedia on that) that might harm you in Postgres. Still the solution is simple. In Postgres create a view that would fill the similar purpose. It can be created like this: create or replace view dual as select 1; Conclusion Well that should be it. Enjoy your cross DB compatible JEE apps.

Want to learn more about interface default methods in Java 8? Check out this tutorial to learn how using this new feature.

If you need to embed and display Google Maps in your Java Desktop Swing application, then JxBrowser Java library is what you need.

Sometimes we want to use Hibernate native SQL in our code. For example we might need to invoke a selectable stored procedure, we cannot invoke in another way. To invoke a native SQL query we use the method createSQLQuery() which is available from the Hibernate session object. In our Grails code we must then first get access to the current Hibernate session. Luckily we only have to inject the sessionFactory bean in our Grails service or controller. To get the current session we invoke the getCurrentSession() method and we are ready to execute a native SQL query. The query itself is defined as a String value and we can use placeholders for variables, just like with other Hibernate queries. In the following sample we create a new Grails service and use a Hibernate native SQL query to execute a selectable stored procedure with the nameorganisation_breadcrumbs. This stored procedure takes one argument startId and will return a list of results with an id, name and level column. // File: grails-app/services/com/mrhaki/grails/OrganisationService.groovy package com.mrhaki.grails import com.mrhaki.grails.Organisation class OrganisationService { // Auto inject SessionFactory we can use // to get the current Hibernate session. def sessionFactory List breadcrumbs(final Long startOrganisationId) { // Get the current Hiberante session. final session = sessionFactory.currentSession // Query string with :startId as parameter placeholder. final String query = 'select id, name, level from organisation_breadcrumbs(:startId) order by level desc' // Create native SQL query. final sqlQuery = session.createSQLQuery(query) // Use Groovy with() method to invoke multiple methods // on the sqlQuery object. final results = sqlQuery.with { // Set domain class as entity. // Properties in domain class id, name, level will // be automatically filled. addEntity(Organisation) // Set value for parameter startId. setLong('startId', startOrganisationId) // Get all results. list() } results } } In the sample code we use the addEntity() method to map the query results to the domain class Organisation. To transform the results from a query to other objects we can use the setResultTransformer() method. Hibernate (and therefore Grails if we use the Hibernate plugin) already has a set of transformers we can use. For example with the org.hibernate.transform.AliasToEntityMapResultTransformer each result row is transformed into a Map where the column aliases are the keys of the map. // File: grails-app/services/com/mrhaki/grails/OrganisationService.groovy package com.mrhaki.grails import org.hibernate.transform.AliasToEntityMapResultTransformer class OrganisationService { def sessionFactory List> breadcrumbs(final Long startOrganisationId) { final session = sessionFactory.currentSession final String query = 'select id, name, level from organisation_breadcrumbs(:startId) order by level desc' final sqlQuery = session.createSQLQuery(query) final results = sqlQuery.with { // Assign result transformer. // This transformer will map columns to keys in a map for each row. resultTransformer = AliasToEntityMapResultTransformer.INSTANCE setLong('startId', startOrganisationId) list() } results } } Finally we can execute a native SQL query and handle the raw results ourselves using the Groovy Collection API enhancements. The result of thelist() method is a List of Object[] objects. In the following sample we use Groovy syntax to handle the results: // File: grails-app/services/com/mrhaki/grails/OrganisationService.groovy package com.mrhaki.grails class OrganisationService { def sessionFactory List> breadcrumbs(final Long startOrganisationId) { final session = sessionFactory.currentSession final String query = 'select id, name, level from organisation_breadcrumbs(:startId) order by level desc' final sqlQuery = session.createSQLQuery(query) final queryResults = sqlQuery.with { setLong('startId', startOrganisationId) list() } // Transform resulting rows to a map with key organisationName. final results = queryResults.collect { resultRow -> [organisationName: resultRow[1]] } // Or to only get a list of names. //final List names = queryResults.collect { it[1] } results } } Code written with Grails 2.3.7.

As businesses accelerate their move toward making B2E applications available to employees on mobile devices, the subject of mobile application security is getting more attention. Mobile Device Management (MDM) solutions are being deployed in the largest enterprises - but there are still application-level security issues that are important to consider. Furthermore, medium size businesses are moving to mobilize their applications prior to having a formalized MDM solution or policy in place. A key element of a mobile app strategy is whether to go Native, Hybrid, or pure HTML5. As an early proponent of HTML5 platforms, Gizmox has been thinking about the security angle of HTML5 applications for a long time. In a recent webinar, we discussed 4 ways that HTML5 - done right - can be more secure than native apps. 1. Applications should leverage HTML5's basic security model HTML5 represents a revolutionary step for HTML-based browsers as the first truly cross-platform technology for rich, interactive applications. It has earned endorsements by all the major IT vendors (e.g. Google, Microsoft, IBM, Oracle, etc...). Security of applications and websites has been a consideration from the start of HTML5 development. The first element of the security model is that HTML5 applications live within the secure shell of the browser sandbox. Application code is to a large degree insulated from the device. The browser's interaction with the device and any other application on the device is highly limited. This makes it difficult for HTML5 application code to influence other applications/data on the device or for other applications to interact with the application running on the browser. The second element is that, built correctly, HTML5 thin clients are "secure by design." Application logic running on the server insultates sensitive intellectual property from the client. Proper design strategies would include minimal or no data caching; keeping tokens, passwords, credentials, and security profiles on the server; minimizing logic on the client - focusing on pure UI interaction with the server. Finally, HTML5 apps should be architected to ensure that no data is left behind in cache. 2. HTML5 apps can be containerized within secure browsers Secure browsers are just one element of MDM that can be deployed on their own to enhance application security. HTML5 application security can be extended with the use of secure browsers that restrict access to enterprise-approved URLs, prevent cross-site scripting, and integrate with company VPNs. Furthermore, secure browsers further harden the interaction between HTML5 applications and the device, the device OS and other applciations on the device. 3. Integration with Mobile Device Management MDM solutions play a variety of security roles including application inventory management (i.e. who gets access to what on which device), application distribution (i.e. through enterprise app store), implementation of security standards (e.g. passwords, encryption, VPN, authentication, etc...), and implemetation of enterprise access control policies. While MDM was in part conceived to enable secure distribution and control of native applications, HTML5 apps can be managed and further secured as well. While full MDM solutions are not required for HTML5 security, HTML5 apps can be integrated into a broader mobile security strategy that incorporates MDM. 4. HTML5 was conceived for the BYOD world The complexity of managing security for native apps gets multiplied as application variants are created for different mobile device form factors and operating systems. With cross-platform HTML5 applications that run on any desktop, tablet, or smartphone, security strategy is implemented and controlled centrally. Updates and security fixes are implemented on the server and there are no concerns with users not applying updates to the apps on their devices. There are many reasons to evaluate HTML5 as the platform for mobile business applications. Security of HTML5 apps (built with good practices and leveraging a full platform like Visual WebGui) is a particularly compelling reason to consider. Check out this slide share from recent webinar on HTML5 security strategies. Security strategies for html5 enterprise mobile apps from Gizmox

Digital signing is a widely used mechanism to make digital contents authentic. By producing a digital signature for some content, we can let another party capable of validating that content. It can provide a guarantee that, is not altered after we signed it, with this validation. With this sample I am to share how to generate the a signature for SOAP envelope. But of course this is valid for any other content signing as well. Here, I will sign The SOAP envelope itself An attachment Place the signature inside SOAP header With the placement of signature inside the SOAP header which is also signed by the signature, this becomes a demonstration of enveloped signature. I am using Apache Santuario library for signing. Following is the code segment I used. I have shared the complete sample here to to be downloaded. public static void main(String unused[]) throws Exception { String keystoreType = "JKS"; String keystoreFile = "src/main/resources/PushpalankaKeystore.jks"; String keystorePass = "pushpalanka"; String privateKeyAlias = "pushpalanka"; String privateKeyPass = "pushpalanka"; String certificateAlias = "pushpalanka"; File signatureFile = new File("src/main/resources/signature.xml"); Element element = null; String BaseURI = signatureFile.toURI().toURL().toString(); //SOAP envelope to be signed File attachmentFile = new File("src/main/resources/sample.xml"); //get the private key used to sign, from the keystore KeyStore ks = KeyStore.getInstance(keystoreType); FileInputStream fis = new FileInputStream(keystoreFile); ks.load(fis, keystorePass.toCharArray()); PrivateKey privateKey = (PrivateKey) ks.getKey(privateKeyAlias, privateKeyPass.toCharArray()); //create basic structure of signature javax.xml.parsers.DocumentBuilderFactory dbf = javax.xml.parsers.DocumentBuilderFactory.newInstance(); dbf.setNamespaceAware(true); DocumentBuilderFactory dbFactory = DocumentBuilderFactory.newInstance(); DocumentBuilder dBuilder = dbFactory.newDocumentBuilder(); Document doc = dBuilder.parse(attachmentFile); XMLSignature sig = new XMLSignature(doc, BaseURI, XMLSignature.ALGO_ID_SIGNATURE_RSA_SHA1); //optional, but better element = doc.getDocumentElement(); element.normalize(); element.getElementsByTagName("soap:Header").item(0).appendChild(sig.getElement()); { Transforms transforms = new Transforms(doc); transforms.addTransform(Transforms.TRANSFORM_C14N_OMIT_COMMENTS); //Sign the content of SOAP Envelope sig.addDocument("", transforms, Constants.ALGO_ID_DIGEST_SHA1); //Adding the attachment to be signed sig.addDocument("../resources/attachment.xml", transforms, Constants.ALGO_ID_DIGEST_SHA1); } //Signing procedure { X509Certificate cert = (X509Certificate) ks.getCertificate(certificateAlias); sig.addKeyInfo(cert); sig.addKeyInfo(cert.getPublicKey()); sig.sign(privateKey); } //write signature to file FileOutputStream f = new FileOutputStream(signatureFile); XMLUtils.outputDOMc14nWithComments(doc, f); f.close(); } At first it reads in the private key which is to be used in signing. To create a key pair for your own, this post will be helpful. Then it has created the signature and added the SOAP message and the attachment as the documents to be signed. Finally it performs signing and write the signed document to a file. The signed SOAP message looks as follows. FUN PARTY uri:www.pjxml.org/socialService/Ping FUN PARTY FUN 59c64t0087fg3kfs000003n9 uri:www.pjxml.org/socialService/ Ping FUN 59c64t0087fg3kfs000003n9 2013-10-22T17:12:20 uri:www.pjxml.org/socialService/ Ping 9RXY9kp/Klx36gd4BULvST4qffI= 3JcccO8+0bCUUR3EJxGJKJ+Wrbc= d0hBQLIvZ4fwUZlrsDLDZojvwK2DVaznrvSoA/JTjnS7XZ5oMplN9 THX4xzZap3+WhXwI2xMr3GKO................x7u+PQz1UepcbKY3BsO8jB3dxWN6r+F4qTyWa+xwOFxqLj546WX35f8zT4GLdiJI5oiYeo1YPLFFqTrwg== MIIDjTCCAnWgAwIBAgIEeotzFjANBgkqhkiG9w0BAQsFADB3MQswCQYDVQQGEwJMSzEQMA4GA1UE...............qXfD/eY+XeIDyMQocRqTpcJIm8OneZ8vbMNQrxsRInxq+DsG+C92b k5y0amGgOQ2O/St0Kc2/xye80tX2fDEKs2YOlM/zCknL8VgK0CbAKVAwvJoycQL9mGRkPDmbitHe............StGofmsoKURzo8hofYEn41rGsq5wCuqJhhHYGDrPpFcuJiuI3SeXgcMtBnMwsIaKv2uHaPRbNX31WEuabuv6Q== AQAB 1.90 In a next post lets see how to verify this signature, so that we can guarantee signed documents are not changed. Cheers!

We all know what XML is right? Just in case not, no problem here is what it is all about. 5 Now, what the computer really needs is the number five and some context around it. In XML you (human and computer) can see how it represents context to five. Now lets say instead you have a business XML document like FPML 32.00 150000 1.00 EUR 405000 2001-07-17Z NONE EUR 2.70 ISDA2002 ISDA2002Equity TODO GBEN Party A Party B That is a lot of extra unnecessary data points. Now lets look at this using Apache Avro. With Avro, the context and the values are separated. This means the schema/structure of what the information is does not get stored or streamed over and over and over and over (and over) again. The Avro schema is hashed. So the data structure only holds the value and the computer understands the fingerprint (the hash) of the schema and can retrieve the schema using the fingerprint. 0x d7a8fbb307d7809469ca9abcb0082e4f8d5651e46d3cdb762d02d0bf37c9e592 This type of implementation is pretty typical in the data space. When you do this you can reduce your data between 20%-80%. When I tell folks this they immediately ask, “why such a large gap of unknowns”. The answer is because not every XML is created the same. But that is the problem because you are duplicating the information the computer needs to understand the data. XML is nice for humans to read, sure … but that is not optimized for the computer. Here is a converter we are working on https://github.com/stealthly/xml-avro to help get folks off of XML and onto lower cost, open source systems. This allows you to keep parts of your systems (specifically the domain business code) using the XML and not having to be changed (risk mitigation) but store and stream the data with less overhead (optimize budget).

Here is a short snippet of Ansible playbook that installs R and any required packages to any nodes of the cluster: - name: Making sure R is installed apt: pkg=r-base state=installed - name: adding a few R packages command: /usr/bin/Rscript --slave --no-save --no-restore-history -e "if (! ('{{item}' %in% installed.packages()[,'Package'])) install.packages(pkgs={{item}, repos=c('http://www.freestatistics.org/cran/'))" with_items: - rjson - rPython - plyr - psych - reshape2 You should replace the repos with one chosen from the list of Cran mirrors. Note that the command above installs each package only if it is not already present, but messes up the “changed” status of Ansible’s PLAY RECAP by incorrectly reporting a change per R package at every run. Find more big data technical posts on my blog.

NoSQL has been a hot topic a pretty long time (well, it's not only a buzz anymore). However, when should we really use it instead of an RDBMS?

A while back, I wrote an article showing how to Live Migrate Your VMs in One Line of Powershell between non-clustered Windows Server 2012 Hyper-V hosts using Shared Nothing Live Migration. Since then, I’ve been asked a few times for how this type of parallel Live Migration would be performed for highly available virtual machines between Hyper-V hosts within a cluster. In this article, we’ll walk through the steps of doing exactly that … via Windows PowerShell on Windows Server 2012 or 2012 R2 or our FREE Hyper-V Server 2012 R2 bare-metal, enterprise-grade hypervisor in a clustered configuration. Wait! Do I need PowerShell to Live Migrate multiple VMs within a Cluster? Well, actually … No. You could certainly use the Failover Cluster Manager GUI tool to select multiple highly available virtual machines, right-click and select Move | Live Migration … Failover Cluster Manager – Performing Multi-VM Live Migration But, you may wish to script this process for other reasons … perhaps to efficiently drain all VM’s from a host as part of a maintenance script that will be performing other tasks. Can I use the same PowerShell cmdlets for Live Migrating within a Cluster? Well, actually … No again. When VMs are made highly available resources within a cluster, they’re managed as cluster group resources instead of being standalone VM resources. As a result, we have a different set of Cluster-aware PowerShell cmdlets that we use when managing these cluster groups. To perform a scripted multi-VM Live Migration, we’ll be leveraging three of these cmdlets: Get-ClusterNode, Get-ClusterGroup and Move-ClusterVirtualMachineRole Now, let’s see that one line of PowerShell! Before getting to the point of actually performing the multi-VM Live Migration in a single PowerShell command line, we first need to setup a few variables to handle the "what" and "where" of moving these VMs. First, let’s specify the name of the cluster with which we’ll be working. We’ll store it in a $clusterName variable. $clusterName = read-host -Prompt "Cluster name" Next, we’ll need to select the cluster node to which we’ll be Live Migrating the VMs. Lets use the Get-ClusterNode and Out-GridView cmdlets together to prompt for the cluster node and store the value in a $targetClusterNode variable. $targetClusterNode = Get-ClusterNode -Cluster $clusterName | Out-GridView -Title "Select Target Cluster Node" ` -OutputMode Single And then, we’ll need to create a list of all the VMs currently running in the cluster. We can use the Get-ClusterGroup cmdlet to retrieve this list. Below, we have an example where we are combining this cmdlet with a Where-Object cmdlet to return only the virtual machine cluster groups that are running on any node except the selected target cluster node. After all, it really doesn’t make any sense to Live Migrate a VM to the same node on which it’s currently running! $haVMs = Get-ClusterGroup -Cluster $clusterName | Where-Object {($_.GroupType -eq "VirtualMachine") ` -and ($_.OwnerNode -ne $targetClusterNode.Name)} We’ve stored the resulting list of VMs in a $haVMs variable. Ready to Live Migrate! OK … Now we have all of our variables defined for the cluster, the target cluster node and the list of VMs from which to choose. Here’s our single line of PowerShell to do the magic … $haVMs | Out-GridView -Title "Select VMs to Move" –PassThru | Move-ClusterVirtualMachineRole -MigrationType Live ` -Node $targetClusterNode.Name -Wait 0 Proceed with care: Keep in mind that your target cluster node will need to have sufficient available resources to run the VM's that you select for Live Migration. Of course, it's best to initially test tasks like this in your lab environment first. Here’s what is happening in this single PowerShell command line: We’re passing the list of VMs stored in the $haVMs variable to the Out-GridView cmdlet. Out-GridView prompts for which VMs to Live Migrate and then passes the selected VMs down the PowerShell object pipeline to the Move-ClusterVirtualMachineRole cmdlet. This cmdlet initiates the Live Migration for each selected VM, and because it’s using a –Wait 0 parameter, it initiates each Live Migration one-after-another without waiting for the prior task to finish. As a result, all of the selected VMs will Live Migrate in parallel, up to the maximum number of concurrent Live Migrations that you’ve configured on these cluster nodes. The VMs selected beyond this maximum will simply queue up and wait their turn. Unlike some competing hypervisors, Hyper-V doesn't impose an artificial hard-coded limit on how many VMs for you can Live Migrate concurrently. Instead, it's up to you to set the maximum to a sensible value based on your hardware and network capacity. Do you have your own PowerShell automation ideas for Hyper-V? Feel free to share your ideas in the Comments section below. See you in the Clouds! - Keith

If you used earlier versions of Neo4j via its Java API with Java 6 you probably have code similar to the following to ensure write operations happen within a transaction: public class StylesOfTx { public static void main( String[] args ) throws IOException { String path = "/tmp/tx-style-test"; FileUtils.deleteRecursively(new File(path)); GraphDatabaseService db = new GraphDatabaseFactory().newEmbeddedDatabase( path ); Transaction tx = db.beginTx(); try { db.createNode(); tx.success(); } finally { tx.close(); } } } In Neo4j 2.0 Transaction started extending AutoCloseable which meant that you could use ‘try with resources’ and the ‘close’ method would be automatically called when the block finished: public class StylesOfTx { public static void main( String[] args ) throws IOException { String path = "/tmp/tx-style-test"; FileUtils.deleteRecursively(new File(path)); GraphDatabaseService db = new GraphDatabaseFactory().newEmbeddedDatabase( path ); try ( Transaction tx = db.beginTx() ) { Node node = db.createNode(); tx.success(); } } } This works quite well although it’s still possible to have transactions hanging around in an application when people don’t use this syntax – the old style is still permissible. In Venkat Subramaniam’s Java 8 book he suggests an alternative approach where we use a lambda based approach: public class StylesOfTx { public static void main( String[] args ) throws IOException { String path = "/tmp/tx-style-test"; FileUtils.deleteRecursively(new File(path)); GraphDatabaseService db = new GraphDatabaseFactory().newEmbeddedDatabase( path ); Db.withinTransaction(db, neo4jDb -> { Node node = neo4jDb.createNode(); }); } static class Db { public static void withinTransaction(GraphDatabaseService db, Consumer fn) { try ( Transaction tx = db.beginTx() ) { fn.accept(db); tx.success(); } } } } The ‘withinTransaction’ function would actually go on GraphDatabaseService or similar rather than being on that Db class but it was easier to put it on there for this example. A disadvantage of this style is that you don’t have explicit control over the transaction for handling the failure case – it’s assumed that if ‘tx.success()’ isn’t called then the transaction failed and it’s rolled back. I’m not sure what % of use cases actually need such fine grained control though. Brian Hurt refers to this as the ‘hole in the middle pattern‘ and I imagine we’ll start seeing more code of this ilk once Java 8 is released and becomes more widely used.

Learn to parse CSV (Comma Separated Values) files with Python examples using the csv module's reader function and DictReader class.

Sometimes it is useful to “backcast” a time series — that is, forecast in reverse time. Although there are no in-built R functions to do this, it is very easy to implement. Suppose x is our time series and we want to backcast for periods. Here is some code that should work for most univariate time series. The example is non-seasonal, but the code will also work with seasonal data. library(forecast) x <- WWWusage h <- 20 f <- frequency(x) # Reverse time revx <- ts(rev(x), frequency=f) # Forecast fc <- forecast(auto.arima(revx), h) plot(fc) # Reverse time again fc$mean <- ts(rev(fc$mean),end=tsp(x)[1] - 1/f, frequency=f) fc$upper <- fc$upper[h:1,] fc$lower <- fc$lower[h:1,] fc$x <- x # Plot result plot(fc, xlim=c(tsp(x)[1]-h/f, tsp(x)[2]))

Within this post we will have a deeper look into the new Date/Time API we get with Java 8 (JSR 310). Please note that this post is mainly driven by code examples that show the new API functionality. I think the examples are self-explanatory so I did not spent much time writing text around them :-) Let's get started! Working with Date and Time Objects All classes of the Java 8 Date/Time API are located within the java.time package. The first class we want to look at is java.time.LocalDate. A LocalDate represents a year-month-day date without time. We start with creating new LocalDate instances: // the current date LocalDate currentDate = LocalDate.now(); // 2014-02-10 LocalDate tenthFeb2014 = LocalDate.of(2014, Month.FEBRUARY, 10); // months values start at 1 (2014-08-01) LocalDate firstAug2014 = LocalDate.of(2014, 8, 1); // the 65th day of 2010 (2010-03-06) LocalDate sixtyFifthDayOf2010 = LocalDate.ofYearDay(2010, 65); LocalTime and LocalDateTime are the next classes we look at. Both work similar to LocalDate. ALocalTime works with time (without dates) while LocalDateTime combines date and time in one class: LocalTime currentTime = LocalTime.now(); // current time LocalTime midday = LocalTime.of(12, 0); // 12:00 LocalTime afterMidday = LocalTime.of(13, 30, 15); // 13:30:15 // 12345th second of day (03:25:45) LocalTime fromSecondsOfDay = LocalTime.ofSecondOfDay(12345); // dates with times, e.g. 2014-02-18 19:08:37.950 LocalDateTime currentDateTime = LocalDateTime.now(); // 2014-10-02 12:30 LocalDateTime secondAug2014 = LocalDateTime.of(2014, 10, 2, 12, 30); // 2014-12-24 12:00 LocalDateTime christmas2014 = LocalDateTime.of(2014, Month.DECEMBER, 24, 12, 0); By default LocalDate/Time classes will use the system clock in the default time zone. We can change this by providing a time zone or an alternative Clock implementation: // current (local) time in Los Angeles LocalTime currentTimeInLosAngeles = LocalTime.now(ZoneId.of("America/Los_Angeles")); // current time in UTC time zone LocalTime nowInUtc = LocalTime.now(Clock.systemUTC()); From LocalDate/Time objects we can get all sorts of useful information we might need. Some examples: LocalDate date = LocalDate.of(2014, 2, 15); // 2014-06-15 boolean isBefore = LocalDate.now().isBefore(date); // false // information about the month Month february = date.getMonth(); // FEBRUARY int februaryIntValue = february.getValue(); // 2 int minLength = february.minLength(); // 28 int maxLength = february.maxLength(); // 29 Month firstMonthOfQuarter = february.firstMonthOfQuarter(); // JANUARY // information about the year int year = date.getYear(); // 2014 int dayOfYear = date.getDayOfYear(); // 46 int lengthOfYear = date.lengthOfYear(); // 365 boolean isLeapYear = date.isLeapYear(); // false DayOfWeek dayOfWeek = date.getDayOfWeek(); int dayOfWeekIntValue = dayOfWeek.getValue(); // 6 String dayOfWeekName = dayOfWeek.name(); // SATURDAY int dayOfMonth = date.getDayOfMonth(); // 15 LocalDateTime startOfDay = date.atStartOfDay(); // 2014-02-15 00:00 // time information LocalTime time = LocalTime.of(15, 30); // 15:30:00 int hour = time.getHour(); // 15 int second = time.getSecond(); // 0 int minute = time.getMinute(); // 30 int secondOfDay = time.toSecondOfDay(); // 55800 Some information can be obtained without providing a specific date. For example, we can use the Year class if we need information about a specific year: Year currentYear = Year.now(); Year twoThousand = Year.of(2000); boolean isLeap = currentYear.isLeap(); // false int length = currentYear.length(); // 365 // sixtyFourth day of 2014 (2014-03-05) LocalDate date = Year.of(2014).atDay(64); We can use the plus and minus methods to add or subtract specific amounts of time. Note that these methods always return a new instance (Java 8 date/time classes are immutable). LocalDate tomorrow = LocalDate.now().plusDays(1); // before 5 houres and 30 minutes LocalDateTime dateTime = LocalDateTime.now().minusHours(5).minusMinutes(30); TemporalAdjusters are another nice way for date manipulation. TemporalAdjuster is a single method interface that is used to separate the process of adjustment from actual date/time objects. A set of common TemporalAdjusters can be accessed using static methods of the TemporalAdjusters class. LocalDate date = LocalDate.of(2014, Month.FEBRUARY, 25); // 2014-02-25 // first day of february 2014 (2014-02-01) LocalDate firstDayOfMonth = date.with(TemporalAdjusters.firstDayOfMonth()); // last day of february 2014 (2014-02-28) LocalDate lastDayOfMonth = date.with(TemporalAdjusters.lastDayOfMonth()); Static imports make this more fluent to read: import static java.time.temporal.TemporalAdjusters.*; ... // last day of 2014 (2014-12-31) LocalDate lastDayOfYear = date.with(lastDayOfYear()); // first day of next month (2014-03-01) LocalDate firstDayOfNextMonth = date.with(firstDayOfNextMonth()); // next sunday (2014-03-02) LocalDate nextSunday = date.with(next(DayOfWeek.SUNDAY)); Time Zones Working with time zones is another big topic that is simplified by the new API. The LocalDate/Time classes we have seen so far do not contain information about a time zone. If we want to work with a date/time in a certain time zone we can use ZonedDateTime or OffsetDateTime: ZoneId losAngeles = ZoneId.of("America/Los_Angeles"); ZoneId berlin = ZoneId.of("Europe/Berlin"); // 2014-02-20 12:00 LocalDateTime dateTime = LocalDateTime.of(2014, 02, 20, 12, 0); // 2014-02-20 12:00, Europe/Berlin (+01:00) ZonedDateTime berlinDateTime = ZonedDateTime.of(dateTime, berlin); // 2014-02-20 03:00, America/Los_Angeles (-08:00) ZonedDateTime losAngelesDateTime = berlinDateTime.withZoneSameInstant(losAngeles); int offsetInSeconds = losAngelesDateTime.getOffset().getTotalSeconds(); // -28800 // a collection of all available zones Set allZoneIds = ZoneId.getAvailableZoneIds(); // using offsets LocalDateTime date = LocalDateTime.of(2013, Month.JULY, 20, 3, 30); ZoneOffset offset = ZoneOffset.of("+05:00"); // 2013-07-20 03:30 +05:00 OffsetDateTime plusFive = OffsetDateTime.of(date, offset); // 2013-07-19 20:30 -02:00 OffsetDateTime minusTwo = plusFive.withOffsetSameInstant(ZoneOffset.ofHours(-2)); Timestamps Classes like LocalDate and ZonedDateTime provide a human view on time. However, often we need to work with time viewed from a machine perspective. For this we can use the Instant class which represents timestamps. An Instant counts the time beginning from the first second of January 1, 1970 (1970-01-01 00:00:00) also called the EPOCH. Instant values can be negative if they occured before the epoch. They followISO 8601 the standard for representing date and time. // current time Instant now = Instant.now(); // from unix timestamp, 2010-01-01 12:00:00 Instant fromUnixTimestamp = Instant.ofEpochSecond(1262347200); // same time in millis Instant fromEpochMilli = Instant.ofEpochMilli(1262347200000l); // parsing from ISO 8601 Instant fromIso8601 = Instant.parse("2010-01-01T12:00:00Z"); // toString() returns ISO 8601 format, e.g. 2014-02-15T01:02:03Z String toIso8601 = now.toString(); // as unix timestamp long toUnixTimestamp = now.getEpochSecond(); // in millis long toEpochMillis = now.toEpochMilli(); // plus/minus methods are available too Instant nowPlusTenSeconds = now.plusSeconds(10); Periods and Durations Period and Duration are two other important classes. Like the names suggest they represent a quantity or amount of time. A Period uses date based values (years, months, days) while a Duration uses seconds or nanoseconds to define an amount of time. Duration is most suitable when working with Instants and machine time. Periods and Durations can contain negative values if the end point occurs before the starting point. // periods LocalDate firstDate = LocalDate.of(2010, 5, 17); // 2010-05-17 LocalDate secondDate = LocalDate.of(2015, 3, 7); // 2015-03-07 Period period = Period.between(firstDate, secondDate); int days = period.getDays(); // 18 int months = period.getMonths(); // 9 int years = period.getYears(); // 4 boolean isNegative = period.isNegative(); // false Period twoMonthsAndFiveDays = Period.ofMonths(2).plusDays(5); LocalDate sixthOfJanuary = LocalDate.of(2014, 1, 6); // add two months and five days to 2014-01-06, result is 2014-03-11 LocalDate eleventhOfMarch = sixthOfJanuary.plus(twoMonthsAndFiveDays); // durations Instant firstInstant= Instant.ofEpochSecond( 1294881180 ); // 2011-01-13 01:13 Instant secondInstant = Instant.ofEpochSecond(1294708260); // 2011-01-11 01:11 Duration between = Duration.between(firstInstant, secondInstant); // negative because firstInstant is after secondInstant (-172920) long seconds = between.getSeconds(); // get absolute result in minutes (2882) long absoluteResult = between.abs().toMinutes(); // two hours in seconds (7200) long twoHoursInSeconds = Duration.ofHours(2).getSeconds(); Formatting and Parsing Formatting and parsing is another big topic when working with dates and times. In Java 8 this can be accomplished by using the format() and parse() methods: // 2014-04-01 10:45 LocalDateTime dateTime = LocalDateTime.of(2014, Month.APRIL, 1, 10, 45); // format as basic ISO date format (20140220) String asBasicIsoDate = dateTime.format(DateTimeFormatter.BASIC_ISO_DATE); // format as ISO week date (2014-W08-4) String asIsoWeekDate = dateTime.format(DateTimeFormatter.ISO_WEEK_DATE); // format ISO date time (2014-02-20T20:04:05.867) String asIsoDateTime = dateTime.format(DateTimeFormatter.ISO_DATE_TIME); // using a custom pattern (01/04/2014) String asCustomPattern = dateTime.format(DateTimeFormatter.ofPattern("dd/MM/yyyy")); // french date formatting (1. avril 2014) String frenchDate = dateTime.format(DateTimeFormatter.ofPattern("d. MMMM yyyy", new Locale("fr"))); // using short german date/time formatting (01.04.14 10:45) DateTimeFormatter formatter = DateTimeFormatter.ofLocalizedDateTime(FormatStyle.SHORT) .withLocale(new Locale("de")); String germanDateTime = dateTime.format(formatter); // parsing date strings LocalDate fromIsoDate = LocalDate.parse("2014-01-20"); LocalDate fromIsoWeekDate = LocalDate.parse("2014-W14-2", DateTimeFormatter.ISO_WEEK_DATE); LocalDate fromCustomPattern = LocalDate.parse("20.01.2014", DateTimeFormatter.ofPattern("dd.MM.yyyy")); Conversion Of course we do not always have objects of the type we need. Therefore, we need an option to convert different date/time related objects between each other. The following examples show some of the possible conversion options: // LocalDate/LocalTime <-> LocalDateTime LocalDate date = LocalDate.now(); LocalTime time = LocalTime.now(); LocalDateTime dateTimeFromDateAndTime = LocalDateTime.of(date, time); LocalDate dateFromDateTime = LocalDateTime.now().toLocalDate(); LocalTime timeFromDateTime = LocalDateTime.now().toLocalTime(); // Instant <-> LocalDateTime Instant instant = Instant.now(); LocalDateTime dateTimeFromInstant = LocalDateTime.ofInstant(instant, ZoneId.of("America/Los_Angeles")); Instant instantFromDateTime = LocalDateTime.now().toInstant(ZoneOffset.ofHours(-2)); // convert old date/calendar/timezone classes Instant instantFromDate = new Date().toInstant(); Instant instantFromCalendar = Calendar.getInstance().toInstant(); ZoneId zoneId = TimeZone.getDefault().toZoneId(); ZonedDateTime zonedDateTimeFromGregorianCalendar = new GregorianCalendar().toZonedDateTime(); // convert to old classes Date dateFromInstant = Date.from(Instant.now()); TimeZone timeZone = TimeZone.getTimeZone(ZoneId.of("America/Los_Angeles")); GregorianCalendar gregorianCalendar = GregorianCalendar.from(ZonedDateTime.now()); Conclusion With Java 8 we get a very rich API for working with date and time located in the java.time package. The API can completely replace old classes like java.util.Date or java.util.Calendar with newer, more flexible classes. Due to mostly immutable classes the new API helps in building thread safe systems. The source of the examples can be found on GitHub.

First of all, let me tell this out loud: Spring is no longer XML-heavy. As a matter of fact you can write Spring applications these days with minimal or no XML at all, using plenty of annotations, Java configuration and Spring Boot. Seriously stop ranting about Spring and XML, it's the thing the of the past. That being said you might still be using XML for couple of reasons: you are stuck with legacy code base, you chose XML for other reasons or you use Spring as a foundation for some framework/platform. The last case is actually quite common, for example Mule ESB and ActiveMQ use Spring underneath to wire up their dependencies. Moreover Spring XML is their way to configure the framework. However configuring message broker or enterprise service bus using plain Spring s would be cumbersome and verbose. Luckily Spring supports writing custom namespaces that can be embedded within standard Spring configuration files. These custom snippets of XML are preprocessed at runtime and can register many bean definitions at once in a concise and pleasantly looking (as far as XML allows) format. In a way custom namespaces are like macros that expand at runtime into multiple bean definitions. To give you a feeling of what are we aiming at, imagine a standard "enterprise" application that has several business entities. For each entity we define three, almost identical, beans: repository, service and controller. They are always wired in a similar way and only differ in small details. To begin with, our Spring XML looks like this (I am pasting screenshot with thumbnail to spare your eyes, it's huge and bloated): This is a "layered" architecture, thus we will call our custom namespace called onion - because onions have layers - and also because systems designed this way make me cry. By the end of this article you will learn how to collapse this pile of XML into: Look closely, it's still Spring XML file that is perfectly understandable by this framework - and you will learn how to achieve this. You can run arbitrary code for each top-level custom XML tag, e.g. single occurrence of registers repository, service and controller bean definitions all at once. The first thing to implement is writing a custom XML schema for our namespace. This is not that hard and will allow IntelliJ IDEA to show code completion in XML: Once the schema is completed we must register it in Spring using two files: /META-INF/spring.schemas: http\://nurkiewicz.blogspot.com/spring/onion/spring-onion.xsd=/com/blogspot/nurkiewicz/onion/ns/spring-onion.xsd /META-INF/spring.handlers: http\://nurkiewicz.blogspot.com/spring/onion/spring-onion.xsd=com.blogspot.nurkiewicz.onion.ns.OnionNamespaceHandler One maps schema URL into schema location locally, another points to so-called namespace handler. This class is fairly straightforward - it tells what to do with every top-level custom XML tag coming from this namespace encountered in Spring configuration file: import org.springframework.beans.factory.xml.NamespaceHandlerSupport; public class OnionNamespaceHandler extends NamespaceHandlerSupport { public void init() { registerBeanDefinitionParser("entity", new EntityBeanDefinitionParser()); registerBeanDefinitionParser("converter", new ConverterBeanDefinitionParser()); } } So, when piece of XML is found by Spring, it knows that our ConverterBeanDefinitionParser needs to be used. Remember that if our custom tag has children (like in the case of ), bean definition parser is called only for top-level tag. It is up to us how we parse and handle children. OK, so single tag is suppose to create the following two beans: The responsibility of bean definition parser is to programmatically register bean definitions otherwise defined in XML. I won't go into details of the API, but compare it with the XML snippet above, they match closely to each other: import org.w3c.dom.Element; public class ConverterBeanDefinitionParser extends AbstractBeanDefinitionParser { @Override protected AbstractBeanDefinition parseInternal(Element converterElement, ParserContext parserContext) { final String format = converterElement.getAttribute("format"); final String lenientStr = converterElement.getAttribute("lenient"); final boolean lenient = lenientStr != null? Boolean.valueOf(lenientStr) : true; final BeanDefinitionRegistry registry = parserContext.getRegistry(); final AbstractBeanDefinition converterBeanDef = converterBeanDef(format, lenient); registry.registerBeanDefinition(format + "Converter", converterBeanDef); final AbstractBeanDefinition readerBeanDef = readerBeanDef(format); registry.registerBeanDefinition(format + "Reader", readerBeanDef); return null; } private AbstractBeanDefinition readerBeanDef(String format) { return BeanDefinitionBuilder. rootBeanDefinition(ReaderFactoryBean.class). addPropertyValue("format", format). getBeanDefinition(); } private AbstractBeanDefinition converterBeanDef(String format, boolean lenient) { AbstractBeanDefinition converterBeanDef = BeanDefinitionBuilder. rootBeanDefinition(Converter.class.getName()). addConstructorArgValue(format + ".xml"). addConstructorArgValue(lenient). addPropertyReference("reader", format + "Reader"). getBeanDefinition(); converterBeanDef.setFactoryBeanName("convertersFactory"); converterBeanDef.setFactoryMethodName("build"); return converterBeanDef; } } Do you see how parseInternal() receives XML Element representing tag, extracts attributes and registers bean definitions? It's up to you how many beans you define in AbstractBeanDefinitionParser implementation. Just remember that we are barely constructing the configuration here, no instantiation took place yet. Once the XML file is fully parsed and all bean definition parsers triggered, Spring will start bootstrapping our application. One thing to keep in mind is returning null in the end. The API sort of expects you to return single bean definition. However no need to restrict ourselves, null is fine. The second custom tag that we support is that registers three beans at once. It's similar and thus not that interesting, see full source of EntityBeanDefinitionParser. One important implementation detail that can be found there is the usage of ManagedList. Documentation vaguely mentions it but it's quite valuable. If you want to define a list of beans to be injected knowing their IDs, a simple List is not sufficient, you must explicitly tell Spring you mean a list of bean references: List converterRefs = new ManagedList<>(); for (String converterName : converters) { converterRefs.add(new RuntimeBeanReference(converterName)); } return BeanDefinitionBuilder. rootBeanDefinition("com.blogspot.nurkiewicz.FooService"). addPropertyValue("converters", converterRefs). getBeanDefinition(); Using JAXB to simplify bean definition parsers OK, so by now you should be familiar with custom Spring namespaces and how they can help you. However they are quite low level by requiring you to parse custom tags using raw XML DOM API. However my team mate discovered that since we already have XSD schema file, why not use JAXB to handle XML parsing? First we ask maven to generate Java beans representing XML types and elements during build: org.jvnet.jaxb2.maven2 maven-jaxb22-plugin 0.8.3 xjc generate src/main/resources/com/blogspot/nurkiewicz/onion/ns com.blogspot.nurkiewicz.onion.ns.xml Under /target/generated-sources/xjc you will discover couple of Java files. I like generated JAXB models to have some commons prefix like Xml, which can be easily achieved with custom bindings.xjb file placed next to spring-onion.xsd: How does it change our custom bean definition parser? Previously we had this: final String clazz = entityElement.getAttribute("class"); //... final NodeList pageNodes = entityElement.getElementsByTagNameNS(NS, "page"); for (int i = 0; i < pageNodes.getLength(); ++i) { //...final String clazz = entityElement.getAttribute("class"); //... final NodeList pageNodes = entityElement.getElementsByTagNameNS(NS, "page"); for (int i = 0; i < pageNodes.getLength(); ++i) { //... Now we simply traverse Java beans: final XmlEntity entity = JaxbHelper.unmarshal(entityElement); final String clazz = entity.getClazz(); //... for (XmlPage page : entity.getPage()) { //... JaxbHelper is just a simple tool that hides checked exceptions and JAXB mechanics from outside: public class JaxbHelper { private static final Unmarshaller unmarshaller = create(); private static Unmarshaller create() { try { return JAXBContext.newInstance("com.blogspot.nurkiewicz.onion.ns.xml").createUnmarshaller(); } catch (JAXBException e) { throw Throwables.propagate(e); } } public static T unmarshal(Element elem) { try { return (T) unmarshaller.unmarshal(elem); } catch (JAXBException e) { throw Throwables.propagate(e); } } } Couple of words as a summary. First of all I don't encourage you to auto-generate repository/service/controller bean definitions for every entity. Actually it's a poor practice but the domain is familiar to all of us so I thought it will be a good example. Secondly, more important, custom XML namespaces are a powerful tool that should be used as a last resort when everything else fails, namely abstract beans, factory beans and Java configuration. Typically you'll want this kind of feature in frameworks or tools built in top of Spring. In that case check out full source code on GitHub.

We all write unit tests but the challenge we face at times is that the unit under test might be dependent on other components. And configuring other components for unit testing is definitely an overkill. Instead we can make use of Mocks in place of the other components and continue with the unit testing. To show how one can use mocks, I have a Data access layer(DAL), basically a class which provides an API for the application to access and modify the data in the data repository. I then unit test the DAL without actually the need to connect to the data repository. The data repository can be a local database or remote database or a file system or any place where we can store and retrieve the data. The use of a DAL class helps us in keeping the data mappers separate from the application code. Lets create a Java project using maven. mvn archetype:generate -DgroupId=info.sanaulla -DartifactId=MockitoDemo -DarchetypeArtifactId=maven-archetype-quickstart -DinteractiveMode=false The above creates a folder MockitoDemo and then creates the entire directory structure for source and test files. Consider the below model class for this example: package info.sanaulla.models; import java.util.List; /** * Model class for the book details. */ public class Book { private String isbn; private String title; private List authors; private String publication; private Integer yearOfPublication; private Integer numberOfPages; private String image; public Book(String isbn, String title, List authors, String publication, Integer yearOfPublication, Integer numberOfPages, String image){ this.isbn = isbn; this.title = title; this.authors = authors; this.publication = publication; this.yearOfPublication = yearOfPublication; this.numberOfPages = numberOfPages; this.image = image; } public String getIsbn() { return isbn; } public String getTitle() { return title; } public List getAuthors() { return authors; } public String getPublication() { return publication; } public Integer getYearOfPublication() { return yearOfPublication; } public Integer getNumberOfPages() { return numberOfPages; } public String getImage() { return image; } } The DAL class for operating on the Book model class is: package info.sanaulla.dal; import info.sanaulla.models.Book; import java.util.ArrayList; import java.util.Arrays; import java.util.Collections; import java.util.List; /** * API layer for persisting and retrieving the Book objects. */ public class BookDAL { private static BookDAL bookDAL = new BookDAL(); public List getAllBooks(){ return Collections.EMPTY_LIST; } public Book getBook(String isbn){ return null; } public String addBook(Book book){ return book.getIsbn(); } public String updateBook(Book book){ return book.getIsbn(); } public static BookDAL getInstance(){ return bookDAL; } } The DAL layer above currently has no functionality and we are going to unit test that piece of code (TDD). The DAL layer might communicate with a ORM Mapper or Database API which we are not concerned while designing the API. Test Driving the DAL Layer There are lot of frameworks for Unit testing and mocking in Java but for this example I would be picking JUnit for unit testing and Mockito for mocking. We would have to update the dependency in Maven’s pom.xml 4.0.0 info.sanaulla MockitoDemo jar 1.0-SNAPSHOT MockitoDemo http://maven.apache.org junit junit 4.10 test org.mockito mockito-all 1.9.5 test Now lets unit test the BookDAL. During the unit testing we will inject mock data into the BookDAL so that we can complete the testing of the API without depending on the data source. Initially we will have an empty test class: public class BookDALTest { public void setUp() throws Exception { } public void testGetAllBooks() throws Exception { } public void testGetBook() throws Exception { } public void testAddBook() throws Exception { } public void testUpdateBook() throws Exception { } } We will inject the mock BookDAL and mock data in the setUp() as shown below: public class BookDALTest { private static BookDAL mockedBookDAL; private static Book book1; private static Book book2; @BeforeClass public static void setUp(){ //Create mock object of BookDAL mockedBookDAL = mock(BookDAL.class); //Create few instances of Book class. book1 = new Book("8131721019","Compilers Principles", Arrays.asList("D. Jeffrey Ulman","Ravi Sethi", "Alfred V. Aho", "Monica S. Lam"), "Pearson Education Singapore Pte Ltd", 2008,1009,"BOOK_IMAGE"); book2 = new Book("9788183331630","Let Us C 13th Edition", Arrays.asList("Yashavant Kanetkar"),"BPB PUBLICATIONS", 2012,675,"BOOK_IMAGE"); //Stubbing the methods of mocked BookDAL with mocked data. when(mockedBookDAL.getAllBooks()).thenReturn(Arrays.asList(book1, book2)); when(mockedBookDAL.getBook("8131721019")).thenReturn(book1); when(mockedBookDAL.addBook(book1)).thenReturn(book1.getIsbn()); when(mockedBookDAL.updateBook(book1)).thenReturn(book1.getIsbn()); } public void testGetAllBooks() throws Exception {} public void testGetBook() throws Exception {} public void testAddBook() throws Exception {} public void testUpdateBook() throws Exception {} } In the above setUp() method I have: Created a mock object of BookDAL BookDAL mockedBookDAL = mock(BookDAL.class); Stubbed the API of BookDAL with mock data, such that when ever the API is invoked the mocked data is returned. //When getAllBooks() is invoked then return the given data and so on for the other methods. when(mockedBookDAL.getAllBooks()).thenReturn(Arrays.asList(book1, book2)); when(mockedBookDAL.getBook("8131721019")).thenReturn(book1); when(mockedBookDAL.addBook(book1)).thenReturn(book1.getIsbn()); when(mockedBookDAL.updateBook(book1)).thenReturn(book1.getIsbn()); Populating the rest of the tests we get: package info.sanaulla.dal; import info.sanaulla.models.Book; import org.junit.BeforeClass; import org.junit.Test; import static org.junit.Assert.*; import static org.mockito.Mockito.mock; import static org.mockito.Mockito.when; import java.util.Arrays; import java.util.List; public class BookDALTest { private static BookDAL mockedBookDAL; private static Book book1; private static Book book2; @BeforeClass public static void setUp(){ mockedBookDAL = mock(BookDAL.class); book1 = new Book("8131721019","Compilers Principles", Arrays.asList("D. Jeffrey Ulman","Ravi Sethi", "Alfred V. Aho", "Monica S. Lam"), "Pearson Education Singapore Pte Ltd", 2008,1009,"BOOK_IMAGE"); book2 = new Book("9788183331630","Let Us C 13th Edition", Arrays.asList("Yashavant Kanetkar"),"BPB PUBLICATIONS", 2012,675,"BOOK_IMAGE"); when(mockedBookDAL.getAllBooks()).thenReturn(Arrays.asList(book1, book2)); when(mockedBookDAL.getBook("8131721019")).thenReturn(book1); when(mockedBookDAL.addBook(book1)).thenReturn(book1.getIsbn()); when(mockedBookDAL.updateBook(book1)).thenReturn(book1.getIsbn()); } @Test public void testGetAllBooks() throws Exception { List allBooks = mockedBookDAL.getAllBooks(); assertEquals(2, allBooks.size()); Book myBook = allBooks.get(0); assertEquals("8131721019", myBook.getIsbn()); assertEquals("Compilers Principles", myBook.getTitle()); assertEquals(4, myBook.getAuthors().size()); assertEquals((Integer)2008, myBook.getYearOfPublication()); assertEquals((Integer) 1009, myBook.getNumberOfPages()); assertEquals("Pearson Education Singapore Pte Ltd", myBook.getPublication()); assertEquals("BOOK_IMAGE", myBook.getImage()); } @Test public void testGetBook(){ String isbn = "8131721019"; Book myBook = mockedBookDAL.getBook(isbn); assertNotNull(myBook); assertEquals(isbn, myBook.getIsbn()); assertEquals("Compilers Principles", myBook.getTitle()); assertEquals(4, myBook.getAuthors().size()); assertEquals("Pearson Education Singapore Pte Ltd", myBook.getPublication()); assertEquals((Integer)2008, myBook.getYearOfPublication()); assertEquals((Integer)1009, myBook.getNumberOfPages()); } @Test public void testAddBook(){ String isbn = mockedBookDAL.addBook(book1); assertNotNull(isbn); assertEquals(book1.getIsbn(), isbn); } @Test public void testUpdateBook(){ String isbn = mockedBookDAL.updateBook(book1); assertNotNull(isbn); assertEquals(book1.getIsbn(), isbn); } } One can run the test by using maven command: mvn test. The output is: ------------------------------------------------------- T E S T S ------------------------------------------------------- Running info.sanaulla.AppTest Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.029 sec Running info.sanaulla.dal.BookDALTest Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.209 sec Results : Tests run: 5, Failures: 0, Errors: 0, Skipped: 0 So we have been able to test the DAL class without actually configuring the data source by using mocks.

The WebLogic Server 12c has very nice support for Maven now. The doc for this is kinda hidden though, so here is a direct link http://docs.oracle.com/middleware/1212/core/MAVEN To summarize the doc, Oracle did not provide a public Maven repository manager hosting for their server artifacts. However they do now provide a tool for you to create and populate your own. You can setup either your local repository (if you are working mostly on your own in a single computer), or you may deploy them into your own internal Maven repository manager such as Archiva or Nexus. Here I would show how the local repository is done. First step is use a maven plugin provided by WLS to populate the repository. I am using a MacOSX for this demo and my WLS is installed in $HOME/apps/wls12120. If you are on Windows, you may install it under C:/apps/wls12120. $ cd $HOME/apps/wls12120/oracle_common/plugins/maven/com/oracle/maven/oracle-maven-sync/12.1.2/ $ mvn install:install-file -DpomFile=oracle-maven-sync.12.1.2.pom -Dfile=oracle-maven-sync.12.1.2.jar $ mvn com.oracle.maven:oracle-maven-sync:push -Doracle-maven-sync.oracleHome=$HOME/apps/wls12120 -Doracle-maven-sync.testingOnly=false The artifacts are placed under your local $HOME/.m2/repository/com/oracle. Now you may use Maven to build Java EE application with these WebLogic artifact as dependencies. Not only these are available, the push also populated some additional maven plugins that helps development more easy. For example, you can generate a template project using their archetype plugin. $ cd $HOME $ mvn archetype:generate \ -DarchetypeGroupId=com.oracle.weblogic.archetype \ -DarchetypeArtifactId=basic-webapp \ -DarchetypeVersion=12.1.2-0-0 \ -DgroupId=org.mycompany \ -DartifactId=my-basic-webapp-project \ -Dversion=1.0-SNAPSHOT Type 'Y' to confirm to finish. Notice that pom.xml it generated; it is using the "javax:javaee-web-api:6.0:provided" dependency. This is working because we setup the repository earlier. Now you may build it. $ cd my-basic-webapp-project $ mvn package After this build you should have the war file under the target directory.You may manually copy and deploy this into your WebLogic server domain. Or you may continue to configure the maven pom to do this all with maven. Here is how I do it. Edit the my-basic-webapp-project/pom.xml file and replace the weblogic-maven-plugin plugin like this: com.oracle.weblogic weblogic-maven-plugin 12.1.2-0-0 ${oracleMiddlewareHome} ${oracleServerUrl} ${oracleUsername} ${oraclePassword} ${project.build.directory}/${project.build.finalName}.${project.packaging} ${oracleServerName} true ${project.build.finalName} With this change, you may deploy the webapp into WebLogic server (well, assuming you already started your "mydomain" with "myserver" server running locally. See my previous blog for instructions) $ cd my-basic-webapp-project $ mvn weblogic:deploy -DoracleMiddlewareHome=$HOME/apps/wls12120 -DoracleServerName=myserver -DoracleUsername=admin -DoraclePassword=admin123 After the "BUILD SUCCESS" message, you may visit the http://localhost:7001/basicWebapp URL. Revisit the WLS doc again and you will find that they also provide other project templates (Maven calls these archetypes) for building EJB, MDB, or WebService projects. These should help you get your EE projects started quickly.

This story goes back at least a decade, when I was first approached by a PHB with a question “How big servers are we going to need to buy for our production deployment”. The new and shiny system we have been building was nine months from production rollout and apparently the company had promised to deliver the whole solution, including hardware. Oh boy, was I in trouble. With just a few years of experience down my belt, I could have pretty much just tossed a dice. Even though I am sure my complete lack of confidence was clearly visible, I still had to come up with the answer. Four hours of googling later I recall sitting there with the same question still hovering in front of my bedazzled face: “How to estimate the need for computing power?” In this post I start to open up the subject by giving you rough guidelines on how to estimate memory requirements for your brand new Java application. For the impatient ones – the answer will be to start with the memory equal to approximately 5 x [amount of memory consumed by Live Data] and start the fine-tuning from there. For the ones more curious about the logic behind, stay with me and I will walk you through the reasoning. First and foremost, I can only recommend to avoid answering a question phrased like this without detailed information being available. Your answer has to be based upon the performance requirements, so do not even start without clarifying those first. And I do not mean way-too-ambiguous “The system needs to support 700 concurrent users”, but a lot more specific ones about latency and throughput, taking into account the amount of data, usage patterns. One should also not forget about the budget also – we all can all dream about sub-millisecond latencies, but those without HFT banking backbone budgets – unfortunately it will only remain a dream. For now, lets assume you have those requirements in place. Next stop would be to create the load test scripts emulating user behaviour. If you are now able to launch those scripts concurrently you have built a foundation to the answer. As you might also have guessed, the next step involves our usual advice of measuring not guessing. But with a caveat. Live Data Size Namely, our quest for the optimal memory configuration requires capturing the Live Data Size. Having captured this, we have the baseline configuration in place for the fine-tuning. How does one define live data size? Charlie Hunt and Binu John in their “Java Performance” book have given it the following definition: Live data size is the heap size consumed by the set of long-lived objects required to run the application in its steady state. Equipped with the definition, we are ready to run your load tests against the application with the GC logging turned on (-XX:+PrintGCTimeStamps -Xloggc:/tmp/gc.log -XX:+PrintGCDetails) and visualize the logs (with the help of gcviewer for example) to determine the moment when the application has reached to the steady state. What you are after looks similar to the following: We can see the GC doing its job both with minor and Full GC runs in a familiar double-saw-toothed graphic. This particular application seems to have achieved a steady state already after the first full GC run on 21st second. In most cases however, it takes 10-20 Full GC runs to spot the change in trends. After four full GC runs we can estimate that the Live Data Size is equal to approximately 100MB. The aforementioned Java Performance book is now indicating that there is a strong correlation between the Live Data Size and the optimal memory configuration parameters in a typical Java EE application. The evidence from the field is also backing up their recommendations: Set the maximum heap size to 3-4 x [Live Data Size] So, for our application at hand, we should set the -Xmx to be in between 300m and 400m for the initial performance tests and take it from there. We have mixed feelings about other recommendations given in the book, recommending to set the maximum permanent generation size to 1.2-1.5 x [Live Data Size of the Permanent Generation] and the -XX:NewRatio being set to 1-1.5 x of the [Live Data Size]. We are currently gathering more data to determine whether the positive correlation exists, but until then I recommend to base your survival and eden configuration decisions on monitoring your allocation rate instead. Why should one bother you might now ask. Indeed, two reasons for not caring immediately surface: 8G memory chip is in sub $100 territory at the time of writing this article Virtualization, especially when using large providers such as Amazon AWS make adjusting the capacity easy Both of the reasons are partially valid and have definitely reduced the need for provisioning to be precise. But both of them are still putting you in the danger zone When tossing in huge amounts of memory “just in case” you are most likely going to significantly affect the latency – going into heaps above 8G it is darn easy to introduce Full GC pauses spanning over tens of seconds. When over-provisioning with the mindset of “lets tune it later”, the “later” part has a tendency of never arriving. I have faced numerous applications running on vastly over provisioned environments just because of this. For example the aforementioned application I discovered running on Amazon EC2 m1.xlarge instance was costing the company $4,200 per instance / year. Converting it to m1.small reduced the bill to just $520 for the instance. 8-fold cost reduction will be visible from your operations budget if your deployments are large, trust me on this. Summary Unfortunately I still see way too many decisions made exactly like I was forced to do a decade ago. This leads to the under- and over planning of capacity, both of which can be equally poor choices, especially if you cannot enjoy the benefits of virtualization. I got lucky with mine, but you might not get away with your guestimate, so I can only recommend to actually plan ahead using the simple framework described in this post. If you enjoyed the content, I can only recommend to follow our performance tuning advice in Twitter.