This article represents top 4 security vulnerabilities related coding practice to avoid while you are programming with Java language. Recently, I came across few Java projects where these instances were found. Please feel free to comment/suggest if I missed to mention one or more important points. Also, sorry for the typos. Following are the key points described later in this article: Executing a dynamically generated SQL statement Directly writing an Http Parameter to Servlet output Creating an SQL PreparedStatement from dynamic string Array is stored directly Executing a Dynamically Generated SQL Statement This is most common of all. One can find mention of this vulenrability at several places. As a matter of fact, many developers are also aware of this vulnerability, although this is a different thing they end up making mistakes once in a while. In several DAO classes, the instances such as following code were found which could lead to SQL injection attacks. StringBuilder query = new StringBuilder(); query.append( "select * from user u where u.name in (" + namesString + ")" ); try { Connection connection = getConnection(); Statement statement = connection.createStatement(); resultSet = statement.executeQuery(query.toString()); } Instead of above query, one could as well make use of prepared statement such as that demonstrated in the code below. It not only makes code less vulnerable to SQL injection attacks but also makes it more efficient. StringBuilder query = new StringBuilder(); query.append( "select * from user u where u.name in (?)" ); try { Connection connection = getConnection(); PreparedStatement statement = connection.prepareCall(query.toString()); statement.setString( 1, namesString ); resultSet = statement.execute(); } Directly writing an Http Parameter to Servlet Output In Servlet classes, I found instances where the Http request parameter was written as it is, to the output stream, without any validation checks. Following code demonstrate the same: public void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { String content = request.getParameter("some_param"); // // .... some code goes here // response.getWriter().print(content); } Note that above code does not persist anything. Code like above may lead to what is called reflected (or non-persistent) cross site scripting (XSS) vulnerability. Reflected XSS occur when an attacker injects browser executable code within a single HTTP response. As it goes by definition (being non-persistent), the injected attack does not get stored within the application; it manifests only users who open a maliciously crafted link or third-party web page. The attack string is included as part of the crafted URI or HTTP parameters, improperly processed by the application, and returned to the victim. You could read greater details on following OWASP page on reflect XSS Creating an SQL PreparedStatement from Dynamic Query String What it essentially means is the fact that although PreparedStatement was used, but the query was generated as a string buffer and not in the way recommended for prepared statement (parametrized). If unchecked, tainted data from a user would create a String where SQL injection could make it behave in unexpected and undesirable manner. One should rather make the query statement parametrized and, use the PreparedStatement appropriately. Take a look at following code to identify the vulnerable code. StringBuilder query = new StringBuilder(); query.append( "select * from user u where u.name in (" + namesString + ")" ); try { Connection connection = getConnection(); PreparedStatement statement = connection.prepareStatement(query.toString()); resultSet = statement.executeQuery(); } Array is Stored Directly Instances of this vulnerability, Array is stored directly, could help the attacker change the objects stored in array outside of program, and the program behave in inconsistent manner as the reference to the array passed to method is held by the caller/invoker. The solution is to make a copy within the object when it gets passed. In this manner, a subsequent modification of the collection won’t affect the array stored within the object. You could read the details on following stackoverflow page. Following code represents the vulnerability: // Note that values is a String array in the code below. // public void setValues(String[] somevalues) { this.values = somevalues; }

If you use Java for programming, read on to learn about tools like Eclipse IDE, the Java Development Kit, and other must-know tools.

eclipse projects have the nice features that they can link to files and folders: so instead of having the physical file, it is just a pointer to a file. this is very cool as that way i can point to shared files, or keep files in a common place referenced from projects, and so on. linked folder and file in eclipse as with most things in eclipse, there is not a single way how to do things. so i’m showing in this post several ways how to link to files and folders. creating a new link select the folder/project where to create a link to a file. use the context menu with n ew > file (or the menu file > new > other > general > file ): new file that opens a dialog to create a new file. i can select which project/folder i want to use for this: new file dialog next, click on ‘advanced’, enable ‘link to file in the file system’ and browse to the file you want to link to: link to file this will set the link to the file: specified link to file as an absolute path (c:\….) is probably not really what you want, there are eclipse path variables you can use: path variables :idea: note that these are eclipse path variables (not eclipse build variables), see “ eclipse build variables “. to use the path variable for the link, use the ‘extend…’ button and then select the file, then press ok: extending path variable this now uses a path variable for that link. note that it shows as well to which file it resolves: resolved path variable for linked file press finish, and the link will be created. note the small ‘arrow’ in the icon (see “ icon and label decorators in eclipse “) to show a linked file: linked file modifying a link if you right-click on that linked file and select ‘properties’ of it, you can see that it is really a linked file, with the link information, and you can change/edit that link any time: linked file properties linked folder as for link to files, its possible to create ‘link to folders’. it works the same way: select the folder, then use file > new > folder: creating new folder use default location : this creates a normal folder. virtual folder : this does not create a physical folder, but a virtual ‘container’ where i can place links or other virtual folders. this is useful to organize links and virtual folders, without the need for a physical folder. linked folder : like linked files, this links to a folder. :idea: the cool thing about linked (source) folders is: when i add new files to that folder where it links to, the projects with that linked folder to it will ‘see’ the extra files too, and that way new files are automatically added to the project. i do this many times, and it is like a ‘library’ folder for me: i can add a new source file to that ‘library’ folder, and every project linking to that folder will automatically have it added. :-) deleting links linked files and linked folders can be deleted from the project too. in that case, the destination of the link is *not* deleted, only the link: deleting a link to a folder using drag & drop as mentioned at the beginning: there are multiple ways to do the same thing in eclipse. instead using the top menu, or using the context menu, i can use drag & drop. to create links, i need to hold the ctrl key: drag and drop with ctrl pressed to create a link :idea: notice that during the drag&drop with ctrl key pressed the icon gets a ‘+’ to show copy/linking mode. when i drop the file: i get the usual choices how i want to create the link: link to files and folders dialog drag&drop of files and folders do not work inside eclipse. what works as well under windows is to drag&drop a file or folder from the windows explorer :-). summary links to files and folders are a cool thing in eclipse, and they can be created with menus or even simpler with drag&drop. this is not limited to inside eclipse: i can drag&drop from outside with the windows explorer and that way can link to files and folders everywhere :-) happy linking :-)

in “ semihosting with kinetis design studio ” i used printf() to exchange text and data between the target board and the host using the debug connection. kinetis design studio (kds) has that semihosting baked into its libraries. what about if using the gnu arm embedded (launchpad) tools and libraries (see “ switching arm gnu tool chain and libraries in kinetis design studio “)? actually it requires two more steps, but is very easy too. semihosting output there are three things to be in place to use semihosting with the gnu arm embedded (launchpad) libraries: option in the gnu linker settings enabling semihosting in the debugger settings initializing the gnu libraries linker option to enable semihosting for the gnu arm embedded ( launchpad ) libraries, i need to add --specs=rdimon.specs to the linker options: linker option to enable semihosting in case i’m using newlib-nano and want to use printf() and/or scanf() with floating point support, i need to pull in some symbols explicitly with the linker options ‘u': -u _scanf_float -u _printf_float debugger settings in the gnu arm eclipse plugins, i need to enable semihosting. segger j-link for segger j-link, i enable the console in the launch configuration: allocated semihosting console for segger additionally i enable semihosting options in the startup options of the debugger: enabled semihosting in the startup options for segger p&e multilink for p&e the following settings are used: semihosting settings for pne settings for openocd the following settings are used for openocd: openocd semihosting settings initializing the gnu libraries if you would now try to use semihosting with running the debugger, you probably will get error messages like this (e.g. from segger j-link): warning: semihosting command sys_flen failed. handle is 0. warning: semihosting command sys_write failed. handle is 0. warning: semihosting command sys_write failed. handle is 0. warning: semihosting command sys_write failed. handle is 0. the reason is that the semihosting needs to be enabled by the application. i need to call initialise_monitor_handles() before i’m using printf() : 1 2 3 4 5 6 7 8 extern void initialise_monitor_handles( void ); /* prototype */ int main( void ) { initialise_monitor_handles(); /* initialize handles */ for (;;) { printf ( "hello world!\r\n" ); } } with this, i can use printf() and scanf() through a debugger connection. semihosting printf output summary while i don’t like printf() for many reasons, sometimes it is useful to exchange data with the host. using semihosting no physical connection is required, as the communication goes through the debugger. it is somewhat intrusive, and adds code and data overhead, but the gnu arm embedded (launchpad) libraries (both newlib and newlib-nano) have semihosting built-in. it is a matter to enable it in the linker and debugger settings, and to initialize the handles in the application. happy semihosting :-)

Exposing a REST based endpoint for a Spring Boot application or for that matter a straight Spring MVC application is straightforward, the following is a controller exposing an endpoint to create an entity based on the content POST'ed to it: @RestController @RequestMapping("/rest/hotels") public class RestHotelController { .... @RequestMapping(method=RequestMethod.POST) public Hotel create(@RequestBody @Valid Hotel hotel) { return this.hotelRepository.save(hotel); } } Internally Spring MVC uses a component called a HttpMessageConverter to convert the Http request to an object representation and back. A set of default converters are automatically registered which supports a whole range of different resource representation formats - json, xml for instance. Now, if there is a need to customize the message converters in some way, Spring Boot makes it simple. As an example consider if the POST method in the sample above needs to be little more flexible and should ignore properties which are not present in the Hotel entity - typically this can be done by configuring the Jackson ObjectMapper, all that needs to be done with Spring Boot is to create a new HttpMessageConverter bean and that would end up overriding all the default message converters, this way: @Bean public MappingJackson2HttpMessageConverter mappingJackson2HttpMessageConverter() { MappingJackson2HttpMessageConverter jsonConverter = new MappingJackson2HttpMessageConverter(); ObjectMapper objectMapper = new ObjectMapper(); objectMapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false); jsonConverter.setObjectMapper(objectMapper); return jsonConverter; } This works well for a Spring-Boot application, however for straight Spring MVC applications which do not make use of Spring-Boot, configuring a custom converter is a little more complicated - the default converters are not registered by default and an end user has to be explicit about registering the defaults: @Configuration public class WebConfig extends WebMvcConfigurationSupport { @Bean public MappingJackson2HttpMessageConverter customJackson2HttpMessageConverter() { MappingJackson2HttpMessageConverter jsonConverter = new MappingJackson2HttpMessageConverter(); ObjectMapper objectMapper = new ObjectMapper(); objectMapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false); jsonConverter.setObjectMapper(objectMapper); return jsonConverter; } @Override public void configureMessageConverters(List> converters) { converters.add(customJackson2HttpMessageConverter()); super.addDefaultHttpMessageConverters(); } } Here WebMvcConfigurationSupport provides a way to more finely tune the MVC tier configuration of a Spring based application. In the configureMessageConverters method, the custom converter is being registered and then an explicit call is being made to ensure that the defaults are registered also. A little more work than for a Spring-Boot based application.

Apache Camel 2.14 is being released later this month. There is a slight holdup due some Apache infrastructure issue which is being worked on. This blog post is to talk about one of the new functions we have added to this release. Thanks to Lauri Kimmel who donated a camel-metrics component, we integrated with the excellent codehale metrics library. So I took this component one step further and integrated it with the Camel routes so we have additional metrics about the route performances using codehale metrics. This allows end users to seamless feed Camel routing information together with existing data they are gathering using codehale metrics. Also take note we have a lot of existing metrics from camel-core which of course is still around. What codehale brings to the table is that they have additional statistical data which we do not have in camel-core. To use the codehale metics all you need to do is: add camel-metrics component enable route metrics in XML or Java code To enable in XML you declare a as shown below: &;t;bean id="metricsRoutePolicyFactory" class="org.apache.camel.component.metrics. routepolicy.MetricsRoutePolicyFactory"/> And doing so in Java code is easy as well by calling this method on your CamelContext. context.addRoutePolicyFactory(new MetricsRoutePolicyFactory()); Now performance metrics is only useable if you have a way of displaying them, and for that you can use hawtio. Notice you can use any kind of monitoring tooling which can integrate with JMX, as the metrics is available over JMX. The actual data is 100% codehale json format, where a piece of the data is shown in the figure below. Sample of the route metrics JSON data The next release of hawtio supports Camel 2.14 and automatic detects if you have enabled route metrics and if so, then shows a sub, where the information can be seen in real time in a graphical charts. hawtio have detected that we have route metrics enabled, and shows a sub tab where we can see the data in real time The screenshot above is from the new camel-example-servlet-rest-tomcat which we ship out of the box. This example demonstrates another new functionality in Camel 2.14 which is the Rest DSL (I will do a blog about that later). This example enables the route metrics out of the box, so what I did was to deploy this example together with hawtio (the hawtio-default WAR) in Apache Tomcat 8. With hawtio you can also build custom dashboards, so here at the end I have put together a dashboard with various screens from hawtio to have a custom view of a Camel application. hawtio dashboard with Camel route and metrics as well control panel to control the route(s), and the logs in the bottom.

Originally written by Virgil Dodson Actuate recently released a new, free BIRT server called the BIRT iHub F-Type. It incorporates all the functionality of BIRT iHub and is limited only by the capacity of output it can deliver on a daily basis. It is ideal for departmental and smaller scale applications. When BIRT F-Type reaches its maximum output capacity, additional capacity can be purchased on a subscription based model. Some of the key features of BIRT iHub F-Type that will help improve your BIRT content applications are: Interactivity – Allow end-users to modify and personalize reports, and answer questions themselves. Scheduling – Automate report generation based on rules and calendar, and then notify users. Sharing – Secure document management and distribution that allows users to only access content/data they are entitled to. Excel Emitter – Export as native Excel (not CSV) with formulas/pivot tables/worksheets/charts. Integration – JavaScript API to embed dynamic reports and visualizations in your web app. Downloading BIRT iHub F-Type Before we get started with the installation process, we need to download BIRT iHub F-Type. There are three downloads available: Windows, Linux, and a VMware image. This blog will cover the Windows installation. If you’re installing either of the other types, you’ll find links to guides for them at the bottom of this blog post. Once you click on your chosen download, you’ll be asked to register. If you’ve already registered, click the “Click to Login” button. If not, fill out the short registration form to get started. Next, read and accept the license agreement. Once you’ve done that, click the checkbox, and a link for the download will appear. Click that to start your download. At this point, you should also receive an email with an activation code. Be sure to check your spam folder if you don’t see it in your inbox. Installing BIRT iHub F-Type After the download is complete, launch the executable file named ActuateBIRTiHubFType.exe. A welcome message will appear. Press Next to continue. You must read and accept the license agreement on the next screen. Choose a destination folder for the installation. The default is C:\Actuate\BIRTiHub. If you have existing BIRT designs that depend on a JDBC database driver, you can optionally specify the folder where these drivers are located. Press Next to continue. Once the installation has finished, press Finish to launch the BIRT iHub F-Type. A desktop shortcut is also created that points to the iHub F-Type URL at http://localhost:8700/iportal. The first time you launch the BIRT iHub F-Type, you will need to activate it. Enter the activation code that you should have received in an e-mail. After entering a valid activation code, you should receive a message that the code was accepted and the BIRT iHub F-Type should start initializing services. Once that has completed, you will be presented with the login screen. The default user name is “administrator” and the password is blank for your first log in. You’ll be able to change this after you have logged in. Press “Log In” to continue. The first time you launch the BIRT iHub F-Type, you will be in tutorial mode which will help you get started loading your BIRT content and required resources. You can bypass the tutorial mode at any time by pressing the “Exit Tutorial” button at the top right. Select a BIRT design (*.rptdesign) file and press the Upload button. If you don’t have a BIRT design, you can download a sample from the link on the same page. The BIRT design file is automatically inspected and if there are any dependent files needed, like images, data files, BIRT report libraries, CSS styles, or other linked BIRT designs, you will be asked to upload those files as well. Once your BIRT design and dependent files are uploaded, your BIRT report will be displayed in the BIRT iHub F-Type and is now ready to explore. Thanks for reading. Now, it’s time to unleash the full power of BIRT into your application. If you have any questions or comments, please feel free to use the comments section below or visit the BIRT iHub F-Type forum. -Virgil For more blogs in the “Introducing BIRT iHub F-Type” series, see the list below: Installing iHub F-Type: Linux | VMWare Image

The NetBeans Team has released NetBeans IDE 8.0.1, with significant enhancements to features relating to HTML5, JavaScript, and CSS3. An update to NetBeans IDE 8.0, this release includes the following notable changes: Modularity and enterprise JavaScript development support via integration of RequireJS Tools for working seamlessly with Grunt, Karma, and Bower Enhanced Java Editor, PHP Editor, and Git Tools Support for new versions of WebLogic, GlassFish, Tomcat, WildFly, and PrimeFaces Performance improvements and bug fixes Complete list of features: http://wiki.netbeans.org/NewAndNoteworthyNB801 To get the recent changes: Download and install NetBeans 8.0.1, which includes the recently released GlassFish 4.1 OR If you already have NetBeans IDE 8.0 installed, launch the IDE and an update notification will appear. Alternatively, choose Help | Check for Updates. Click the alert-box to install the updates. For details on upgrading, see the YouTube screencast "How to Upgrade to NetBeans IDE 8.0.1 from NetBeans IDE 8.0" on the NetBeans YouTube channel. NetBeans IDE 8.0.1 is available in English, Brazilian Portuguese, Japanese, Russian, and Simplified Chinese. We welcome feedback about your use of NetBeans software. Share your thoughts on the NetBeans mailing lists and forums; and keep track of NetBeans news by subscribing to the NetBeans Weekly Newsletter and following NetBeans on Twitter, Facebook, and YouTube.

Last week I was tasked with developing a quick prototype that used AngularJS for its client and Spring MVC for its server. A colleague developed the same application using Backbone.js and Spring MVC. At first, I considered using my boot-ionic project as a starting point. Then I realized I didn't need to develop a native mobile app, but rather a responsive web app. My colleague mentioned he was going to use RESThub as his starting point, so I figured I'd use JHipster as mine. We allocated a day to get our environments setup with the tools we needed, then timeboxed our first feature spike to four hours. My first experience with JHipster failed the 10-minute test. I spent a lot of time flailing about with various "npm" and "yo" commands, getting permissions issues along the way. After getting thinks to work with some sudo action, I figured I'd try its Docker development environment. This experience was no better. JHipster seems like a nice project, so I figured I'd try to find the causes of my issues. This article is designed to save you the pain I had. If you'd rather just see the steps to get up and running quickly, skip to the summary. The "npm" and "yo" issues I had seemed to be caused by a bad node/npm installation. To fix this, I removed node and installed nvm. Here's the commands I needed to remove node and npm: sudo rm -rf /usr/local/lib/node_modules sudo rm -rf /usr/local/include/node sudo rm /usr/local/bin/node sudo rm -rf /usr/local/bin/npm sudo rm /usr/local/share/man/man1/node.1 sudo rm -rf /usr/local/lib/dtrace/node.d sudo rm -rf ~/.npm Next, I ran "brew doctor" to make sure Homebrew was still happy. It told me some things were broken: $ brew doctor Warning: Broken symlinks were found. Remove them with `brew prune`: /usr/local/bin/yo /usr/local/bin/ionic /usr/local/bin/grunt /usr/local/bin/bower I ran brew update && brew prune, followed by brew install nvm. Next, I added the following to my ~/.profile: source $(brew --prefix nvm)/nvm.sh To install the latest version of node, I ran the commands below and set the latest version as the default: nvm ls-remote nvm install v0.11.13 nvm alias default v0.11.13 Once I had a fresh version of Node.js, I was able to run JHipster's local installation instructions. npm install -g yo npm install -g generator-jhipster Then I created my project: yo jhipster I was disappointed to find this created all the project files in my current directory, rather than in a subdirectory. I'd recommend you do the following instead: mkdir ~/projectname && cd ~/projectname && yo jhipster Before creating your project, JHipster asks you a number of questions. To see what they are, see its documentation on creating an application. Two things to be aware of: Hot reloading Java code doesn't work well (yet) with Java 8 Its OAuth2 implementation doesn't work with WebSockets In other words, I'd recommend using Java 7 + (cookie-based authentication with websockets) or (oauth2 authentication w/o websockets). After creating my project, I was able to run it using "mvn spring-boot:run" and view it at http://localhost:8080. To get hot-reloading for the client, I ran "grunt server" and opened my browser to http://localhost:9000. JHipster + Docker on OS X I had no luck getting the Docker instructions to work initially. I spent a couple hours on it, then gave up. A couple of days ago, I decided to give it another good ol' college-try. To make sure I figured out everything from scratch, I started by removing Docker. I re-installed Docker and pulled the JHipster image using the following: sudo docker pull jdubois/jhipster-docker The error I got from this was the following: 2014/09/05 19:43:38 Post http:///var/run/docker.sock/images/create?fromImage=jdubois%2Fjhipster-docker&tag=: dial unix /var/run/docker.sock: no such file or directory After doing some research, I learned I needed to run boot2docker init first. Next I ran boot2docker up to start the Docker daemon. Then I copied/pasted "export DOCKER_HOST=tcp://192.168.59.103:2375" into my console and tried to run docker pull again. It failed with the same error. The solution was simpler than you might think: don't use sudo. $ docker pull jdubois/jhipster-docker Pulling repository jdubois/jhipster-docker 01bdc74025db: Pulling dependent layers 511136ea3c5a: Download complete ... The next command that JHipster's documentation recommends is to run the Docker image, forward ports and share folders. When you run it, the terminal seems to hang and trying to ssh into it doesn't work. Others have recently reported a similar issue. I discovered the hanging is caused by a missing "-d" parameter and ssh doesn't work because you need to add a portmap to the VM to expose the port to your host. You can fix this by running the following: boot2docker down VBoxManage modifyvm "boot2docker-vm" --natpf1 "containerssh,tcp,,4022,,4022" VBoxManage modifyvm "boot2docker-vm" --natpf1 "containertomcat,tcp,,8080,,8080" VBoxManage modifyvm "boot2docker-vm" --natpf1 "containergruntserver,tcp,,9000,,9000" VBoxManage modifyvm "boot2docker-vm" --natpf1 "containergruntreload,tcp,,35729,,35729" boot2docker start After making these changes, I was able to start the image and ssh into it. docker run -d -v ~/jhipster:/jhipster -p 8080:8080 -p 9000:9000 -p 35729:35729 -p 4022:22 -t jdubois/jhipster-docker ssh -p 4022 jhipster@localhost I tried creating a new project within the VM (cd /jhipster && yo jhipster), but it failed with the following error: /usr/lib/node_modules/generator-jhipster/node_modules/yeoman-generator/node_modules/mkdirp/index.js:89 throw err0; ^ Error: EACCES, permission denied '/jhipster/src' The fix was giving the "jhipster" user ownership of the directory. sudo chown jhipster /jhipster After doing this, I was able to generate an app and run it using "mvn spring-boot:run" and access it from my Mac at http://localhost:8080. I was also able to run "grunt server" and see it at http://localhost:9000 However, I was puzzled to see that there was nothing in my ~/jhipster directory. After doing some searching, I found that the docker run -v /host/path:/container/path doesn't work on OS X. David Gageot's A Better Boot2Docker on OSX led me to svendowideit/samba, which solved this problem. The specifics are documented in boot2docker's folder sharing section. I shutdown my docker container by running "docker ps", grabbing the first two characters of the id and then running: docker stop [2chars] I started the JHipster container without the -v parameter, used "docker ps" to find its name (backstabbing_galileo in this case), then used that to add samba support. docker run -d -p 8080:8080 -p 9000:9000 -p 35729:35729 -p 4022:22 -t jdubois/jhipster-docker docker run --rm -v /usr/local/bin/docker:/docker -v /var/run/docker.sock:/docker.sock svendowideit/samba backstabbing_galileo Then I was able to connect using Finder > Go > Connect to Server, using the following for the server address: cifs://192.168.59.103/jhipster To make this volume appear in my regular development area, I created a symlink: ln -s /Volumes/jhipster ~/dev/jhipster After doing this, all the files were marked as read-only. To fix, I ran "chmod -R 777 ." in the directory on the server. I noticed that this also worked if I ran it from my Mac's terminal, but it took quite a while to traverse all the files. I noticed a similar delay when loading the project into IntelliJ. Summary Phew! That's a lot of information that can be condensed down into four JHipster + Docker on OS X tips. Make sure your npm installation doesn't require sudo rights. If it does, reinstall using nvm. Add portmaps to your VM to expose ports 4022, 8080, 9000 and 35729 to your host. Change ownership on the /jhipster in the Docker image: sudo chown jhipster /jhipster. Use svendowideit/samba to share your VM's directories with OS X.

Recently I had the opportunity to work on a project were I needed to create a custom SQL Server image for use with Azure VMs. The process was a little more challenging than I initially anticipated. I think this is mostly because I was not familiar with the process of preparing a SQL Server image. Perhaps this isn’t much of a challenge for an experienced SQL Server DBA or IT Pro. For me, it was a great learning experience. Why a Custom SQL Server Image? The Azure VM image gallery already contains a SQL Server image. It’s very easy to create a new SQL Server VM using this image. However, doing so has a few important trade-offs to consider: Unable to fully customize the base install of SQL Server. This is a template/image after all – you get a VM configured the way the image was configured. Unable to use your own SQL Server license. If your company has an Enterprise Agreement (EA) with Microsoft, it’s likely there is already some SQL Server licenses built into that agreement. Depending on the details, it may be significantly cheaper to use the licenses from the EA instead of paying the SQL Server VM image upcharge from Azure. The Basic Steps There are 6 basic steps to creating a custom SQL Server VM image for use in Azure. Provision a new base Windows Server VM Download the SQL Server installation media Run SQL Server setup to prepare an image Configure Windows to complete the installation of SQL Server Capture the image and add it to the Azure VM image gallery Create a new VM instance using the custom SQL Server image The basic idea here is to create a base VM, customize it with a SQL Server image, capture the VM to create an image, and then provision new VMs using that captured VM image. Let’s dive into each of these in a little more detail. Note: the terminology here can be a little confusing. When referring to the VM used to create the template/image, I’ll use the term “base VM”. When referring to the VM created from the base VM, I’ll use the term “VM instance”. 1. Provision a new base Windows Server VM There are multiple ways to create a Windows Server VM in Azure. Creating a VM via the Azure management portal and PowerShell are probably the two most popular options. Be sure to check out this tutorial to learn how to do so via the portal. For the purposes of this post, I’ll do so via PowerShell. $img = Get-AzureVMImage ` | where { ( $_.PublisherName -ilike "Microsoft*" -and $_.ImageFamily -ilike "Windows Server 2012 Datacenter" ) } ` | Sort-Object -Unique -Descending -Property ImageFamily ` | sort -Descending -Property PublishDate ` | select -First(1) $vmConfig = New-AzureVMConfig -Name "sql-1" -InstanceSize Small -ImageName $img.ImageName | Add-AzureProvisioningConfig -Windows -AdminUsername "[admin-username-here]" -Password "[admin-password-here]" New-AzureVM -ServiceName "SQLServerVMTemplate" -VMs $vmConfig -Location "East US" -WaitForBoot 2. Download the SQL Server installation media With the base Windows Server 2012 VM created, we can now get ready to prepare (sysprep) the SQL Server installation. To do that, we need to get the SQL Server installation media onto the machine. The easiest way I found to do this was to leverage Azure blob storage. Upload the SQL Server ISO file to Azure blob storage Remote Desktop (RDP) into the base VM From the VM, download the SQL Server ISO file to the local disk Mount the SQL Server ISO file to the VM Copy the ISO contents (not the ISO file itself) to the VM’s C:\ drive. For example, use C:\sql The SQL Server installation media files need to be copied to the local C: drive so it can be used later to complete the SQL Server installation (when provisioning the actual SQL Server VM instance). 3. Run SQL Server setup to prepare an image In order to prepare the (sysprep’d) SQL Server VM image (which we can use as a template for future VMs), we need to run the SQL Server installation and instruct it topreparean image – not run the full installation. An easy way to do this is with a SQL Server configuration file, an example of which I’ve included below. ConfigurationFile.ini ;SQL Server 2012 Configuration File [OPTIONS] ; Specifies a Setup workflow, like INSTALL, UNINSTALL, or UPGRADE. This is a required parameter. ACTION="PrepareImage" ; Detailed help for command line argument ENU has not been defined yet. ENU="True" ; Parameter that controls the user interface behavior. Valid values are Normal for the full UI, AutoAdvance for a simplified UI, and EnableUIOnServerCore for bypassing Server Core setup GUI block. ;UIMODE="Normal" ; Specifies setup not display any user interface. ;QUIET="False" ; Specifies setup to display progress only, without any user interaction. QUIETSIMPLE="True" ; Specifies whether SQL Server Setup should discover and include product updates. The valid values are True and False or 1 and 0. By default SQL Server Setup will include updates that are found. UpdateEnabled="True" ; Specifies features to install, uninstall, or upgrade. The list of top-level features include SQL, AS, RS, IS, MDS, and Tools. The SQL feature will install the Database Engine, Replication, Full-Text, and Data Quality Services (DQS) server. The Tools feature will install Management Tools, Books online components, SQL Server Data Tools, and other shared components. FEATURES=SQLENGINE ; Specifies the location where SQL Server Setup will obtain product updates. The valid values are "MU" to search Microsoft Update, a valid folder path, a relative path such as .\MyUpdates or a UNC share. By default SQL Server Setup will search Microsoft Update or a Windows Update service through the Window Server Update Services. UpdateSource="MU" ; Displays the command line parameters usage HELP="False" ; Specifies that the detailed Setup log should be piped to the console. INDICATEPROGRESS="False" ; Specifies that Setup should install into WOW64. This command line argument is not supported on an IA64 or a 32-bit system. X86="False" ; Specifies the root installation directory for shared components. This directory remains unchanged after shared components are already installed. INSTALLSHAREDDIR="C:\Program Files\Microsoft SQL Server" ; Specifies the root installation directory for the WOW64 shared components. This directory remains unchanged after WOW64 shared components are already installed. INSTALLSHAREDWOWDIR="C:\Program Files (x86)\Microsoft SQL Server" ; Specifies the Instance ID for the SQL Server features you have specified. SQL Server directory structure, registry structure, and service names will incorporate the instance ID of the SQL Server instance. INSTANCEID="MSSQLSERVER" ; Specifies the installation directory. INSTANCEDIR="C:\Program Files\Microsoft SQL Server" There are two steps in this process: Copy the ConfigurationFile.ini file (from your local PC) to the same location as the SQL Server installation media (i.e.c:\sql) on the base VM. Run SQL Server setup to prepare an image. From a command prompt (on the base VM), navigate to theC:\sqlfolder and then execute the following command: Setup.exe /ConfigurationFile=ConfigurationFile.ini /IAcceptSQLServerLicenseTerms=true 4. Configure Windows to complete the installation of SQL Server At this point the base VM should have an “installation” of SQL Server that is not fully completed. The SQL Server bits are in place, but they’re not configured for a full server install . . . at least not yet. The final configuration of SQL Server will take place when the VM instance (of which this template/image is the base) is provisioned and boots up for the first time. This is accomplished by using a CMD file with the following content: @ECHO OFF && SETLOCAL && SETLOCAL ENABLEDELAYEDEXPANSION && SETLOCAL ENABLEEXTENSIONS REM All commands will be executed during first Virtual Machine boot "C:\Program Files\Microsoft SQL Server\110\Setup Bootstrap\SQLServer2012\setup.exe" /QS /ACTION=CompleteImage /INSTANCEID=MSSQLSERVER /INSTANCENAME=MSSQLSERVER /IACCEPTSQLSERVERLICENSETERMS=1 /SQLSYSADMINACCOUNTS=%COMPUTERNAME%\Administrators /BROWSERSVCSTARTUPTYPE=AUTOMATIC /INDICATEPROGRESS /TCPENABLED=1 /PID="[YOUR-SQL-SERVER-PRODUCT-ID-HERE]" On your local PC, save the file as SetupComplete2.cmd RDP / log into the base VM Copy the SetupComplete2.cmd from your local PC file to the c:\Windows\OEM folder on the base VM Change the value for the SQLSYSADMINACCOUNTS value to be that of the administrative account created on the VM (or better yet – the local Administrators group account) If needed, supply the SQL Server product ID (PID) value. When Windows starts on the new VM instance for the first time, the SetupComplete2.cmd file should automatically run. It is invoked by the SetupComplete.cmd file already on the machine. 5. Capture the image and add it to the Azure VM image gallery At this point a base SQL Server VM has been created and the groundwork laid to complete the install. Now it is time to create the VM image from the base VM, and do to that you sysprep and capture the base VM. Please follow the guide on How to Capture a Windows Virtual Machine to Use as a Template. 6. Create a new VM using the custom SQL Server image With a new custom VM image template available in the VM image gallery, you can provision a new VM instance using that custom template. Upon first boot, the newly provisioned VM should complete the full SQL Server installation as laid out in your SetupComplete2.cmd file. Please follow the guide on How to Create a Custom Virtual Machine for more information on creating the VM from the template. Closing Thoughts One of the quirks I noticed when preparing the base SQL Server image is that it was not possible to prepare the image with SQL Server Management Studio (SSMS). I would have to do the install after the newly provisioned VM instance is created. Not hard, but time consuming (an annoying if doing this on multiple VM instances). I later learned that SQL Server 2012 Cumulative Update 1 does allow for preparing a SQL Server image with SSMS installed. I’ve included a link below that describes the process for creating a SQL Server image with CU1. In the end, this process really is not all that hard. Time consuming? Yes! The worst part (at least for me) was really just understanding how the SQL Server installation and sysprep process works. Once I wrapped my head around that, the process was a lot smoother. Helpful Resources While I was learning how to create a custom SQL Server VM image, the following resources were very helpful: How to: Create a Windows Azure Virtual Machine Operating System Image for Microsoft Dynamics NAV. This MSDN article provided the jumping off point on learning how to install SQL Server by using a sysprep image. Install SQL Server 2012 from the Command Prompt Install SQL Server 2012 Using a Configuration File Install SQL Server 2012 Using SysPrep How to create a slipstream SQL Server 2012 and Cumulative Update 1 image –http://sqlperformance.com/2012/12/system-configuration/sql-2012-slipstream I would like to thank Scott Klein for his assistance in verifying these steps. His help was extremely valuable to ensure I was doing this the right way.

I’ve been working on migrating some batch jobs for Podcastpedia.org to Spring Batch. Before, these jobs were developed in my own kind of way, and I thought it was high time to use a more “standardized” approach. Because I had never used Spring with java configuration before, I thought this were a good opportunity to learn about it, by configuring the Spring Batch jobs in java. And since I am all into trying new things with Spring, why not also throw Spring Boot into the boat… Before you begin with this tutorial I recommend you read first Spring’s Getting started – Creating a Batch Service, because the structure and the code presented here builds on that original. 1. What I’ll build So, as mentioned, in this post I will present Spring Batch in the context of configuring it and developing with it some batch jobs for Podcastpedia.org. Here’s a short description of the two jobs that are currently part of the Podcastpedia-batch project: addNewPodcastJob reads podcast metadata (feed url, identifier, categories etc.) from a flat file transforms (parses and prepares episodes to be inserted with Http Apache Client) the data and in the last step, insert it to the Podcastpedia database and inform the submitter via emailabout it notifyEmailSubscribersJob – people can subscribe to their favorite podcasts on Podcastpedia.orgvia email. For those who did it is checked on a regular basis (DAILY, WEEKLY, MONTHLY) if new episodes are available, and if they are the subscribers are informed via email about those; read from database, expand read data via JPA, re-group it and notify subscriber via email Source code: The source code for this tutorial is available on GitHub – Podcastpedia-batch. Note: Before you start I also highly recommend you read the Domain Language of Batch, so that terms like “Jobs”, “Steps” or “ItemReaders” don’t sound strange to you. 2. What you’ll need A favorite text editor or IDE JDK 1.7 or later Maven 3.0+ 3. Set up the project The project is built with Maven. It uses Spring Boot, which makes it easy to create stand-alone Spring based Applications that you can “just run”. You can learn more about the Spring Boot by visiting theproject’s website. 3.1. Maven build file Because it uses Spring Boot it will have the spring-boot-starter-parent as its parent, and a couple of other spring-boot-starters that will get for us some libraries required in the project: pom.xml of the podcastpedia-batch project 4.0.0 org.podcastpedia.batch podcastpedia-batch 0.1.0 1.1.6.RELEASE 1.7 org.springframework.boot spring-boot-starter-parent 1.1.6.RELEASE org.springframework.boot spring-boot-starter-batch org.springframework.boot spring-boot-starter-data-jpa org.apache.httpcomponents httpclient 4.3.5 org.apache.httpcomponents httpcore 4.3.2 org.apache.velocity velocity 1.7 org.apache.velocity velocity-tools 2.0 org.apache.struts struts-core rome rome 1.0 rome rome-fetcher 1.0 org.jdom jdom 1.1 xerces xercesImpl 2.9.1 mysql mysql-connector-java 5.1.31 org.springframework.boot spring-boot-starter-freemarker org.springframework.boot spring-boot-starter-remote-shell javax.mail mail javax.mail mail 1.4.7 javax.inject javax.inject 1 org.twitter4j twitter4j-core [4.0,) org.springframework.boot spring-boot-starter-test maven-compiler-plugin org.springframework.boot spring-boot-maven-plugin Note: One big advantage of using the spring-boot-starter-parent as the project’s parent is that you only have to upgrade the version of the parent and it will get the “latest” libraries for you. When I started the project spring boot was in version 1.1.3.RELEASE and by the time of finishing to write this post is already at 1.1.6.RELEASE. 3.2. Project directory structure I structured the project in the following way: └── src └── main └── java └── org └── podcastpedia └── batch └── common └── jobs └── addpodcast └── notifysubscribers Note: the org.podcastpedia.batch.jobs package contains sub-packages having specific classes to particular jobs. the org.podcastpedia.batch.jobs.common package contains classes used by all the jobs, like for example the JPA entities that both the current jobs require. 4. Create a batch Job configuration I will start by presenting the Java configuration class for the first batch job: package org.podcastpedia.batch.jobs.addpodcast; import org.podcastpedia.batch.common.configuration.DatabaseAccessConfiguration; import org.podcastpedia.batch.common.listeners.LogProcessListener; import org.podcastpedia.batch.common.listeners.ProtocolListener; import org.podcastpedia.batch.jobs.addpodcast.model.SuggestedPodcast; import org.springframework.batch.core.Job; import org.springframework.batch.core.Step; import org.springframework.batch.core.configuration.annotation.EnableBatchProcessing; import org.springframework.batch.core.configuration.annotation.JobBuilderFactory; import org.springframework.batch.core.configuration.annotation.StepBuilderFactory; import org.springframework.batch.item.ItemProcessor; import org.springframework.batch.item.ItemReader; import org.springframework.batch.item.ItemWriter; import org.springframework.batch.item.file.FlatFileItemReader; import org.springframework.batch.item.file.LineMapper; import org.springframework.batch.item.file.mapping.BeanWrapperFieldSetMapper; import org.springframework.batch.item.file.mapping.DefaultLineMapper; import org.springframework.batch.item.file.transform.DelimitedLineTokenizer; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import org.springframework.context.annotation.Import; import org.springframework.core.io.ClassPathResource; import com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException; @Configuration @EnableBatchProcessing @Import({DatabaseAccessConfiguration.class, ServicesConfiguration.class}) public class AddPodcastJobConfiguration { @Autowired private JobBuilderFactory jobs; @Autowired private StepBuilderFactory stepBuilderFactory; // tag::jobstep[] @Bean public Job addNewPodcastJob(){ return jobs.get("addNewPodcastJob") .listener(protocolListener()) .start(step()) .build(); } @Bean public Step step(){ return stepBuilderFactory.get("step") .chunk(1) //important to be one in this case to commit after every line read .reader(reader()) .processor(processor()) .writer(writer()) .listener(logProcessListener()) .faultTolerant() .skipLimit(10) //default is set to 0 .skip(MySQLIntegrityConstraintViolationException.class) .build(); } // end::jobstep[] // tag::readerwriterprocessor[] @Bean public ItemReader reader(){ FlatFileItemReader reader = new FlatFileItemReader(); reader.setLinesToSkip(1);//first line is title definition reader.setResource(new ClassPathResource("suggested-podcasts.txt")); reader.setLineMapper(lineMapper()); return reader; } @Bean public LineMapper lineMapper() { DefaultLineMapper lineMapper = new DefaultLineMapper(); DelimitedLineTokenizer lineTokenizer = new DelimitedLineTokenizer(); lineTokenizer.setDelimiter(";"); lineTokenizer.setStrict(false); lineTokenizer.setNames(new String[]{"FEED_URL", "IDENTIFIER_ON_PODCASTPEDIA", "CATEGORIES", "LANGUAGE", "MEDIA_TYPE", "UPDATE_FREQUENCY", "KEYWORDS", "FB_PAGE", "TWITTER_PAGE", "GPLUS_PAGE", "NAME_SUBMITTER", "EMAIL_SUBMITTER"}); BeanWrapperFieldSetMapper fieldSetMapper = new BeanWrapperFieldSetMapper(); fieldSetMapper.setTargetType(SuggestedPodcast.class); lineMapper.setLineTokenizer(lineTokenizer); lineMapper.setFieldSetMapper(suggestedPodcastFieldSetMapper()); return lineMapper; } @Bean public SuggestedPodcastFieldSetMapper suggestedPodcastFieldSetMapper() { return new SuggestedPodcastFieldSetMapper(); } /** configure the processor related stuff */ @Bean public ItemProcessor processor() { return new SuggestedPodcastItemProcessor(); } @Bean public ItemWriter writer() { return new Writer(); } // end::readerwriterprocessor[] @Bean public ProtocolListener protocolListener(){ return new ProtocolListener(); } @Bean public LogProcessListener logProcessListener(){ return new LogProcessListener(); } } The @EnableBatchProcessing annotation adds many critical beans that support jobs and saves us configuration work. For example you will also be able to @Autowired some useful stuff into your context: a JobRepository (bean name “jobRepository”) a JobLauncher (bean name “jobLauncher”) a JobRegistry (bean name “jobRegistry”) a PlatformTransactionManager (bean name “transactionManager”) a JobBuilderFactory (bean name “jobBuilders”) as a convenience to prevent you from having to inject the job repository into every job, as in the examples above a StepBuilderFactory (bean name “stepBuilders”) as a convenience to prevent you from having to inject the job repository and transaction manager into every step The first part focuses on the actual job configuration: @Bean public Job addNewPodcastJob(){ return jobs.get("addNewPodcastJob") .listener(protocolListener()) .start(step()) .build(); } @Bean public Step step(){ return stepBuilderFactory.get("step") .chunk(1) //important to be one in this case to commit after every line read .reader(reader()) .processor(processor()) .writer(writer()) .listener(logProcessListener()) .faultTolerant() .skipLimit(10) //default is set to 0 .skip(MySQLIntegrityConstraintViolationException.class) .build(); } The first method defines a job and the second one defines a single step. As you’ve read in The Domain Language of Batch, jobs are built from steps, where each step can involve a reader, a processor, and a writer. In the step definition, you define how much data to write at a time (in our case 1 record at a time). Next you specify the reader, processor and writer. 5. Spring Batch processing units Most of the batch processing can be described as reading data, doing some transformation on it and then writing the result out. This mirrors somehow the Extract, Transform, Load (ETL) process, in case you know more about that. Spring Batch provides three key interfaces to help perform bulk reading and writing: ItemReader, ItemProcessor and ItemWriter. 5.1. Readers ItemReader is an abstraction providing the mean to retrieve data from many different types of input: flat files, xml files, database, jms etc., one item at a time. See the Appendix A. List of ItemReaders and ItemWriters for a complete list of available item readers. In the Podcastpedia batch jobs I use the following specialized ItemReaders: 5.1.1. FlatFileItemReader which, as the name implies, reads lines of data from a flat file that typically describe records with fields of data defined by fixed positions in the file or delimited by some special character (e.g. Comma). This type of ItemReader is being used in the first batch job, addNewPodcastJob. The input file used is named suggested-podcasts.in, resides in the classpath (src/main/resources) and looks something like the following: FEED_URL; IDENTIFIER_ON_PODCASTPEDIA; CATEGORIES; LANGUAGE; MEDIA_TYPE; UPDATE_FREQUENCY; KEYWORDS; FB_PAGE; TWITTER_PAGE; GPLUS_PAGE; NAME_SUBMITTER; EMAIL_SUBMITTER http://www.5minutebiographies.com/feed/; 5minutebiographies; people_society, history; en; Audio; WEEKLY; biography, biographies, short biography, short biographies, 5 minute biographies, five minute biographies, 5 minute biography, five minute biography; https://www.facebook.com/5minutebiographies;https://twitter.com/5MinuteBios; ; Adrian Matei; [email protected] http://notanotherpodcast.libsyn.com/rss; NotAnotherPodcast; entertainment; en; Audio; WEEKLY; Comedy, Sports, Cinema, Movies, Pop Culture, Food, Games; https://www.facebook.com/notanotherpodcastusa;https://twitter.com/NAPodcastUSA;https://plus.google.com/u/0/103089891373760354121/posts; Adrian Matei; [email protected] As you can see the first line defines the names of the “columns”, and the following lines contain the actual data (delimited by “;”), that needs translating to domain objects relevant in the context. Let’s see now how to configure the FlatFileItemReader: @Bean public ItemReader reader(){ FlatFileItemReader reader = new FlatFileItemReader(); reader.setLinesToSkip(1);//first line is title definition reader.setResource(new ClassPathResource("suggested-podcasts.in")); reader.setLineMapper(lineMapper()); return reader; } You can specify, among other things, the input resource, the number of lines to skip, and a line mapper. 5.1.1.1. LineMapper The LineMapper is an interface for mapping lines (strings) to domain objects, typically used to map lines read from a file to domain objects on a per line basis. For the Podcastpedia job I used the DefaultLineMapper, which is two-phase implementation consisting of tokenization of the line into a FieldSet followed by mapping to item: @Bean public LineMapper lineMapper() { DefaultLineMapper lineMapper = new DefaultLineMapper(); DelimitedLineTokenizer lineTokenizer = new DelimitedLineTokenizer(); lineTokenizer.setDelimiter(";"); lineTokenizer.setStrict(false); lineTokenizer.setNames(new String[]{"FEED_URL", "IDENTIFIER_ON_PODCASTPEDIA", "CATEGORIES", "LANGUAGE", "MEDIA_TYPE", "UPDATE_FREQUENCY", "KEYWORDS", "FB_PAGE", "TWITTER_PAGE", "GPLUS_PAGE", "NAME_SUBMITTER", "EMAIL_SUBMITTER"}); BeanWrapperFieldSetMapper fieldSetMapper = new BeanWrapperFieldSetMapper(); fieldSetMapper.setTargetType(SuggestedPodcast.class); lineMapper.setLineTokenizer(lineTokenizer); lineMapper.setFieldSetMapper(suggestedPodcastFieldSetMapper()); return lineMapper; } the DelimitedLineTokenizer splits the input String via the “;” delimiter. if you set the strict flag to false then lines with less tokens will be tolerated and padded with empty columns, and lines with more tokens will simply be truncated. the columns names from the first line are set lineTokenizer.setNames(...); and the fieldMapper is set (line 14) Note: The FieldSet is an “interface used by flat file input sources to encapsulate concerns of converting an array of Strings to Java native types. A bit like the role played by ResultSet in JDBC, clients will know the name or position of strongly typed fields that they want to extract.“ 5.1.1.2. FieldSetMapper The FieldSetMapper is an interface that is used to map data obtained from a FieldSet into an object. Here’s my implementation which maps the fieldSet to the SuggestedPodcast domain object that will be further passed to the processor: public class SuggestedPodcastFieldSetMapper implements FieldSetMapper { @Override public SuggestedPodcast mapFieldSet(FieldSet fieldSet) throws BindException { SuggestedPodcast suggestedPodcast = new SuggestedPodcast(); suggestedPodcast.setCategories(fieldSet.readString("CATEGORIES")); suggestedPodcast.setEmail(fieldSet.readString("EMAIL_SUBMITTER")); suggestedPodcast.setName(fieldSet.readString("NAME_SUBMITTER")); suggestedPodcast.setTags(fieldSet.readString("KEYWORDS")); //some of the attributes we can map directly into the Podcast entity that we'll insert later into the database Podcast podcast = new Podcast(); podcast.setUrl(fieldSet.readString("FEED_URL")); podcast.setIdentifier(fieldSet.readString("IDENTIFIER_ON_PODCASTPEDIA")); podcast.setLanguageCode(LanguageCode.valueOf(fieldSet.readString("LANGUAGE"))); podcast.setMediaType(MediaType.valueOf(fieldSet.readString("MEDIA_TYPE"))); podcast.setUpdateFrequency(UpdateFrequency.valueOf(fieldSet.readString("UPDATE_FREQUENCY"))); podcast.setFbPage(fieldSet.readString("FB_PAGE")); podcast.setTwitterPage(fieldSet.readString("TWITTER_PAGE")); podcast.setGplusPage(fieldSet.readString("GPLUS_PAGE")); suggestedPodcast.setPodcast(podcast); return suggestedPodcast; } } 5.2. JdbcCursorItemReader In the second job, notifyEmailSubscribersJob, in the reader, I only read email subscribers from a single database table, but further in the processor a more detailed read(via JPA) is executed to retrieve all the new episodes of the podcasts the user subscribed to. This is a common pattern employed in the batch world. Follow this link for more Common Batch Patterns. For the initial read, I chose the JdbcCursorItemReader, which is a simple reader implementation that opens a JDBC cursor and continually retrieves the next row in the ResultSet: @Bean public ItemReader notifySubscribersReader(){ JdbcCursorItemReader reader = new JdbcCursorItemReader(); String sql = "select * from users where is_email_subscriber is not null"; reader.setSql(sql); reader.setDataSource(dataSource); reader.setRowMapper(rowMapper()); return reader; } Note I had to set the sql, the datasource to read from and a RowMapper. 5.2.1. RowMapper The RowMapper is an interface used by JdbcTemplate for mapping rows of a Result’set on a per-row basis. My implementation of this interface, , performs the actual work of mapping each row to a result object, but I don’t need to worry about exception handling: public class UserRowMapper implements RowMapper { @Override public User mapRow(ResultSet rs, int rowNum) throws SQLException { User user = new User(); user.setEmail(rs.getString("email")); return user; } } 5.2. Writers ItemWriter is an abstraction that represents the output of a Step, one batch or chunk of items at a time. Generally, an item writer has no knowledge of the input it will receive next, only the item that was passed in its current invocation. The writers for the two jobs presented are quite simple. They just use external services to send email notifications and post tweets on Podcastpedia’s account. Here is the implementation of the ItemWriterfor the first job – addNewPodcast: package org.podcastpedia.batch.jobs.addpodcast; import java.util.Date; import java.util.List; import javax.inject.Inject; import javax.persistence.EntityManager; import org.podcastpedia.batch.common.entities.Podcast; import org.podcastpedia.batch.jobs.addpodcast.model.SuggestedPodcast; import org.podcastpedia.batch.jobs.addpodcast.service.EmailNotificationService; import org.podcastpedia.batch.jobs.addpodcast.service.SocialMediaService; import org.springframework.batch.item.ItemWriter; import org.springframework.beans.factory.annotation.Autowired; public class Writer implements ItemWriter{ @Autowired private EntityManager entityManager; @Inject private EmailNotificationService emailNotificationService; @Inject private SocialMediaService socialMediaService; @Override public void write(List items) throws Exception { if(items.get(0) != null){ SuggestedPodcast suggestedPodcast = items.get(0); //first insert the data in the database Podcast podcast = suggestedPodcast.getPodcast(); podcast.setInsertionDate(new Date()); entityManager.persist(podcast); entityManager.flush(); //notify submitter about the insertion and post a twitt about it String url = buildUrlOnPodcastpedia(podcast); emailNotificationService.sendPodcastAdditionConfirmation( suggestedPodcast.getName(), suggestedPodcast.getEmail(), url); if(podcast.getTwitterPage() != null){ socialMediaService.postOnTwitterAboutNewPodcast(podcast, url); } } } private String buildUrlOnPodcastpedia(Podcast podcast) { StringBuffer urlOnPodcastpedia = new StringBuffer( "http://www.podcastpedia.org"); if (podcast.getIdentifier() != null) { urlOnPodcastpedia.append("/" + podcast.getIdentifier()); } else { urlOnPodcastpedia.append("/podcasts/"); urlOnPodcastpedia.append(String.valueOf(podcast.getPodcastId())); urlOnPodcastpedia.append("/" + podcast.getTitleInUrl()); } String url = urlOnPodcastpedia.toString(); return url; } } As you can see there’s nothing special here, except that the write method has to be overriden and this is where the injected external services EmailNotificationService and SocialMediaService are used to inform via email the podcast submitter about the addition to the podcast directory, and if a Twitter page was submitted a tweet will be posted on the Podcastpedia’s wall. You can find detailed explanation on how to send email via Velocity and how to post on Twitter from Java in the following posts: How to compose html emails in Java with Spring and Velocity How to post to Twittter from Java with Twitter4J in 10 minutes 5.3. Processors ItemProcessor is an abstraction that represents the business processing of an item. While theItemReader reads one item, and the ItemWriter writes them, the ItemProcessor provides access to transform or apply other business processing. When using your own Processors you have to implement the ItemProcessor interface, with its only method O process(I item) throws Exception, returning a potentially modified or a new item for continued processing. If the returned result is null, it is assumed that processing of the item should not continue. While the processor of the first job requires a little bit of more logic, because I have to set the etag andlast-modified header attributes, the feed attributes, episodes, categories and keywords of the podcast: public class SuggestedPodcastItemProcessor implements ItemProcessor { private static final int TIMEOUT = 10; @Autowired ReadDao readDao; @Autowired PodcastAndEpisodeAttributesService podcastAndEpisodeAttributesService; @Autowired private PoolingHttpClientConnectionManager poolingHttpClientConnectionManager; @Autowired private SyndFeedService syndFeedService; /** * Method used to build the categories, tags and episodes of the podcast */ @Override public SuggestedPodcast process(SuggestedPodcast item) throws Exception { if(isPodcastAlreadyInTheDirectory(item.getPodcast().getUrl())) { return null; } String[] categories = item.getCategories().trim().split("\\s*,\\s*"); item.getPodcast().setAvailability(org.apache.http.HttpStatus.SC_OK); //set etag and last modified attributes for the podcast setHeaderFieldAttributes(item.getPodcast()); //set the other attributes of the podcast from the feed podcastAndEpisodeAttributesService.setPodcastFeedAttributes(item.getPodcast()); //set the categories List categoriesByNames = readDao.findCategoriesByNames(categories); item.getPodcast().setCategories(categoriesByNames); //set the tags setTagsForPodcast(item); //build the episodes setEpisodesForPodcast(item.getPodcast()); return item; } ...... } the processor from the second job uses the ‘Driving Query’ approach, where I expand the data retrieved from the Reader with another “JPA-read” and I group the items on podcasts with episodes so that it looks nice in the emails that I am sending out to subscribers: @Scope("step") public class NotifySubscribersItemProcessor implements ItemProcessor { @Autowired EntityManager em; @Value("#{jobParameters[updateFrequency]}") String updateFrequency; @Override public User process(User item) throws Exception { String sqlInnerJoinEpisodes = "select e from User u JOIN u.podcasts p JOIN p.episodes e WHERE u.email=?1 AND p.updateFrequency=?2 AND" + " e.isNew IS NOT NULL AND e.availability=200 ORDER BY e.podcast.podcastId ASC, e.publicationDate ASC"; TypedQuery queryInnerJoinepisodes = em.createQuery(sqlInnerJoinEpisodes, Episode.class); queryInnerJoinepisodes.setParameter(1, item.getEmail()); queryInnerJoinepisodes.setParameter(2, UpdateFrequency.valueOf(updateFrequency)); List newEpisodes = queryInnerJoinepisodes.getResultList(); return regroupPodcastsWithEpisodes(item, newEpisodes); } ....... } Note: If you’d like to find out more how to use the Apache Http Client, to get the etag and last-modifiedheaders, you can have a look at my post – How to use the new Apache Http Client to make a HEAD request 6. Execute the batch application Batch processing can be embedded in web applications and WAR files, but I chose in the beginning the simpler approach that creates a standalone application, that can be started by the Java main() method: package org.podcastpedia.batch; //imports ...; @ComponentScan @EnableAutoConfiguration public class Application { private static final String NEW_EPISODES_NOTIFICATION_JOB = "newEpisodesNotificationJob"; private static final String ADD_NEW_PODCAST_JOB = "addNewPodcastJob"; public static void main(String[] args) throws BeansException, JobExecutionAlreadyRunningException, JobRestartException, JobInstanceAlreadyCompleteException, JobParametersInvalidException, InterruptedException { Log log = LogFactory.getLog(Application.class); SpringApplication app = new SpringApplication(Application.class); app.setWebEnvironment(false); ConfigurableApplicationContext ctx= app.run(args); JobLauncher jobLauncher = ctx.getBean(JobLauncher.class); if(ADD_NEW_PODCAST_JOB.equals(args[0])){ //addNewPodcastJob Job addNewPodcastJob = ctx.getBean(ADD_NEW_PODCAST_JOB, Job.class); JobParameters jobParameters = new JobParametersBuilder() .addDate("date", new Date()) .toJobParameters(); JobExecution jobExecution = jobLauncher.run(addNewPodcastJob, jobParameters); BatchStatus batchStatus = jobExecution.getStatus(); while(batchStatus.isRunning()){ log.info("*********** Still running.... **************"); Thread.sleep(1000); } log.info(String.format("*********** Exit status: %s", jobExecution.getExitStatus().getExitCode())); JobInstance jobInstance = jobExecution.getJobInstance(); log.info(String.format("********* Name of the job %s", jobInstance.getJobName())); log.info(String.format("*********** job instance Id: %d", jobInstance.getId())); System.exit(0); } else if(NEW_EPISODES_NOTIFICATION_JOB.equals(args[0])){ JobParameters jobParameters = new JobParametersBuilder() .addDate("date", new Date()) .addString("updateFrequency", args[1]) .toJobParameters(); jobLauncher.run(ctx.getBean(NEW_EPISODES_NOTIFICATION_JOB, Job.class), jobParameters); } else { throw new IllegalArgumentException("Please provide a valid Job name as first application parameter"); } System.exit(0); } } The best explanation for SpringApplication-, @ComponentScan- and @EnableAutoConfiguration-magic you get from the source – Getting Started – Creating a Batch Service: “The main() method defers to the SpringApplication helper class, providing Application.class as an argument to its run() method. This tells Spring to read the annotation metadata from Application and to manage it as a component in the Spring application context. The @ComponentScan annotation tells Spring to search recursively through theorg.podcastpedia.batchpackage and its children for classes marked directly or indirectly with Spring’s @Component annotation. This directive ensures that Spring finds and registers BatchConfiguration, because it is marked with @Configuration, which in turn is a kind of @Component annotation. The @EnableAutoConfiguration annotation switches on reasonable default behaviors based on the content of your classpath. For example, it looks for any class that implements the CommandLineRunner interface and invokes its run() method.” Execution construction steps: the JobLauncher, which is a simple interface for controlling jobs, is retrieved from the ApplicationContext. Remember this is automatically made available via the@EnableBatchProcessing annotation. now based on the first parameter of the application (args[0]), I will retrieve the correspondingJob from the ApplicationContext then the JobParameters are prepared, where I use the current date - .addDate("date", new Date()), so that the job executions are always unique. once everything is in place, the job can be executed: JobExecution jobExecution = jobLauncher.run(addNewPodcastJob, jobParameters); you can use the returned jobExecution to gain access to BatchStatus, exit code, or job name and id. Note: I highly recommend you read and understand the Meta-Data Schema for Spring Batch. It will also help you better understand the Spring Batch Domain objects. 6.1. Running the application on dev and prod environments To be able to run the Spring Batch / Spring Boot application on different environments I make use of the Spring Profiles capability. By default the application runs with development data (database). But if I want the job to use the production database I have to do the following: provide the following environment argument -Dspring.profiles.active=prod have the production database properties configured in the application-prod.properties file in the classpath, right besides the default application.properties file Summary In this tutorial we’ve learned how to configure a Spring Batch project with Spring Boot and Java configuration, how to use some of the most common readers in batch processing, how to configure some simple jobs, and how to start Spring Batch jobs from a main method. Note: As I mentioned, I am fairly new to Spring Batch, and especially to Spring Boot and Spring Configuration with Java, so if you see any potential for improvement (code, job design etc.) please make a pull request or leave a comment below. Thanks a lot.

Some of the hardest problems a Java Developer will ever have to face are classpath errors: ClassNotFoundException, NoClassDefFoundError, Jar Hell, Xerces Hell and company. In this post we will go through the root causes of these problems, and see how a minimal tool (JHades) can help solving them quickly. We will see why Maven cannot (always) prevent classpath duplicates, and also: The only way to deal with Jar Hell Class loaders The Class loader chain Class loader priority: Parent First vs Parent Last Debugging server startup problems Making sense of Jar Hell with jHades Simple strategy for avoiding classpath problems The classpath gets fixed in Java 9? The only way to deal with Jar Hell Classpath problems can be time-consuming to debug, and tend to happen at the worst possible times and places: before releases, and often in environments where there is little to no access by the development team. They can also happen at the IDE level, and become a source of reduced productivity. We developers tend to find these problems early and often, and this is the usual response: Let's try to save us some hair and get to the bottom of this. These type of problems are hard to approach via trial and error. The only real way to solve them is to really understand what is going on, but where to start? It turns out that Jar Hell problems are simpler than what they look, and only a few concepts are needed to solve them. In the end, the common root causes for Jar Hell problems are: a Jar is missing there is one Jar too many a class is not visible where it should be But if it's that simple, then why are classpath problems so hard to debug? Jar Hell stack traces are incomplete One reason is that the stack traces for classpath problems have a lot of information missing that is needed to troubleshoot the problem. Take for example this stack trace: java.lang.IncompatibleClassChangeError: Class org.jhades.SomeServiceImpl does not implement the requested interfaceorg.jhades.SomeService org.jhades.TestServlet.doGet(TestServlet.java:19) It says that a class does not implement a certain interface. But if we look at the class source: publicclassSomeServiceImpl implementsSomeService { @Override publicvoiddoSomething() { System.out.println( "Call successful!"); } Well, the class clearly implements the missing interface! So what is going on then? The problem is that the stack trace is missing a lot of information that is critical to understanding the problem. The stack trace should have probably contained an error message such as this (we will learn what this means): The Class SomeServiceImpl of class loader /path/to/tomcat/lib does not implement the interface SomeService loaded from class loader Tomcat - WebApp - /path/to/tomcat/webapps/test This would be at least an indication of where to start: Someone new learning Java would at least know that there is this notion of class loader that is essential to understand what is going on It would make clear that one class involved was not being loaded from a WAR, but somehow from some directory on the server (SomeServiceImpl). What is a Class Loader? To start, a Class Loader is just a Java class, more exactly an instance of a class at runtime. It is NOT an inaccessible internal component of the JVM like for example the garbage collector. Take for example the WebAppClassLoader of Tomcat, here is it's javadoc. As you can see it's just a plain Java class, we can even write our own class loader if needed. Any subclass of ClassLoader will qualify as a class loader. The main responsibilities of a class loader is to known where class files are located, and then load classes on JVM demand. Everything is linked to a class loader Each object in the JVM is linked to it's Class via getClass(), and each class is linked to a class loader via getClassLoader(). This means that: Every object in the JVM is linked to a class loader! Let's see how this fact can be used to troubleshoot a classpath error scenario. How-To find where a class file really is Let's take an object and see where it's class file is located in the file system: System.out.println(service.getClass() .getClassLoader() .getResource("org/jhades/SomeServiceImpl.class")); This is the full path to the class file: jar:file:/Users/user1/.m2/repository/org/jhades/jar-2/1.0-SNAPSHOT/jar-2-1.0-SNAPSHOT.jar!/org/jhades/SomeServiceImpl.class As we can see the class loader is just a runtime component that knowns where in the file system to look for class files and how to load them. But what happens if the class loader cannot find a given class? The Class loader Chain By default in the JVM, if a class loader does not find a class, it will then ask it's parent class loader for that same class and so forth. This continues all the way up until the JVM bootstrap class loader (more on this later). This chain of class loaders is the class loader delegation chain. Class loader priority: Parent First vs Parent Last Some class loaders delegate requests immediately to the parent class loader, without searching first in their own known set of directories for the class file. A class loader operating on this mode is said to be in Parent First mode. If a class loader first looks for a class locally and only after queries the parent if the class is not found, then that class loader is said to be working in Parent Last mode. Do all applications have a class loader chain ? Even the most simple Hello World main method has 3 class loaders: The Application class loader, responsible for loading the application classes (parent first) The Extensions class loader, that loads jars from $JAVA_HOME/jre/lib/ext (parent first) The Bootstrap class loader, that loads any class shipped with the JDK such as java.lang.String (no parent class loader) What does the class loader chain of a WAR application look like? In the case of application servers like Tomcat or Websphere, the class loader chain is configured differently than a simple Hello World main method program. Take for example the case of the Tomcat class loader chain: Here we wee that each WAR runs in a WebAppClassLoader, that works in parent last mode (it can be set to parent first as well). The Common class loader loads libraries installed at the level of the server. What does the Servlet spec say about class loading? Only a small part of the class loader chain behavior is defined by the Servlet container specification: The WAR application runs on it's own application class loader, that might be shared with other applications or not The files in WEB-INF/classes take precedence over everything else After that, it's anyones guess! The rest is completely open for interpretation by container providers. Why isn't there a common approach for class loading across vendors? Usually open source containers like Tomcat or Jetty are configured by default to look for classes in the WAR first, and only then search in server class loaders. This allows for applications to use their own versions of libraries that override the ones available on the server. What about the big iron servers? Commercial products like Websphere will try to 'sell' you their own server provided libraries, that by default take precedence over the ones installed on the WAR. This is done assuming that if you bought the server you want also to use the JEE libraries and versions it provides, which is often NOT the case. This makes deploying to certain commercial products a huge hassle, as they behave differently then the Tomcat or Jetty that developers use to run applications in their workstation. We will see further on a solution for this. Common Problem: duplicate class versions At this moment you probably have a huge question: What if there are two jars inside a WAR that contain the exact same class? The answer is that the behavior is undetermined and only at runtime one of the two classes will be chosen. Which one gets chosen depends on the internal implementation of the class loader, there is no way to know upfront. But luckily most projects these days use Maven, and Maven solves this problem by ensuring only one version of a given jar is added to the WAR. So a Maven project is immune to this particular type of Jar Hell, right? Why Maven does not prevent classpath duplicates Unfortunately Maven cannot help in all Jar Hell situations. In fact, many Maven projects that don't use certain quality control plugins can have hundreds of duplicate class files on the classpath (I saw trunks with over 500 duplicates). There are several reasons for that: Library publishers occasionally change the artifact name of a jar: This happens due to re-branding or other reasons. Take for example the example of the JAXB jar. There is no way Maven can identify those artifacts as being the same jar! Some jars are published with and without dependencies: Some library providers provide a 'with dependencies' version of a jar, which includes other jars inside. If we have transitive dependencies with the two versions, we will end up with duplicates. Some classes are copied between jars: Some library creators, when faced with the need for a certain class will just grab it from another project and copy it to a new jar without changing the package name. Are all class files duplicates dangerous? If the duplicate class files exist inside the same class loader, and the two duplicate class files are exactly identical then it does not matter which one gets chosen first - this situation is not dangerous. If the two class files are inside the same class loader and they are not identical, then there is no way which one will be chosen at runtime - this is problematic and can manifest itself when deploying to different environments. If the class files are in two different class loaders, then they are never considered identical (see the class identity crisis section further on). How can WAR classpath duplicates be avoided? This problem can be avoided for example by using the Maven Enforcer Plugin, with the extra rule of Ban Duplicate Classes turned on. You can quickly check if your WAR is clean using the JHades WAR duplicate classes report as well. This tool has an option to filter 'harmless' duplicates (same class file size). But even a clean WAR might have deployment problems: Classes missing, classes taken from the server instead of the WAR and thus with the wrong version, class cast exceptions, etc. Debugging the classpath with JHades Classpath problems often show up when the application server is starting up, which is a particularly bad moment specially when deploying to an environment where there is limited access. JHades is a tool to help deal it with Jar Hell (disclaimer: I wrote it). It's a single Jar with no dependencies other than the JDK7 itself. This is an example of how to use it: newJHades() .printClassLoaders() .printClasspath() .overlappingJarsReport() .multipleClassVersionsReport() .findClassByName("org.jhades.SomeServiceImpl") This prints to the screen the class loader chain, jars, duplicate classes, etc. Debugging server startup problems JHades works works well in scenarios where the server does not start properly. A servlet listener is provided that allows to print classpath debugging information even before any other component of the application starts running. ClassCastException and the Class Identity Crisis When troubleshooting Jar Hell, beware of ClassCastExceptions. A class is identified in the JVM not only by it's fully qualified class name, but also by it's class loader. This is counterintuitive but in hindsight makes sense: We can create two different classes with the same package and name, ship them in two jars and put them in two different class loaders. One let's say extends ArrayList and the other is a Map. The classes are therefore completely different (despite the same name) and cannot be cast to each other! The runtime will throw a CCE to prevent this potential error case, because there is no guarantee that the classes are castable. Adding the class loader to the class identifier was the outcome of the Class Identity Crisis that occurred in earlier Java days. A Strategy for Avoiding Classpath Problems This is easier said then done, but the best way to avoid classpath related deployment problems is to run the production server in Parent Last mode. This way the class versions of the WAR take precedence over the ones on the server, and the same classes are used in production and in a developer workstation where it's likely that Tomcat, Jetty or other open source Parent Last server is being used. In certain servers like Websphere, this is not sufficient and you also have to provide special properties on the manifest file to explicitly turn off certain libraries like for example JAX-WS. Fixing the classpath in Java 9 In Java 9 the classpath gets completely revamped with the new Jigsaw modularity system. In Java 9 a jar can be declared as a module and it will run in it's own isolated class loader, that reads class files from other similar module class loaders in an OSGI sort of way. This will allow multiple versions of the same Jar to coexist in the same application if needed. Conclusions In the end, Jar Hell problems are not that low level or unapproachable as they might seem at first. It's all about zip files (jars) being present/ not being present in certain directories, how to find those directories, and how to debug the classpath in environments with limited access. By knowing a limited set of concepts such as Class Loaders, the Class Loader Chain and Parent First / Parent Last modes, these problems can be tackled effectively. External links This presentation Do you really get class loaders from Jevgeni Kabanov of ZeroTurnaround (JRebel company) is a great resource about Jar Hell and the different type of classpath related exceptions.

This article lists some of the best practices that would be useful for developers while they are coding with AngularJS. These are out of my own experiences while working on AngularJS and do not warranty the entire list. I am sure there can be more to this list and thus, request my readers to suggest/comment such that they could be added to the list below. Found some of the following pages which presents a set of good practices you would want to refer. Thanks to the readers for the valuable contribution. AngularJS Style Guide App Structure Best practices Initialization One should try and place the

In an earlier post I showed how easy it is to integrate Hystrix into a Spring Boot application. Now I’m going to show you a neat trick which combines the health indicator endpoint in Spring Boot and the metrics provided by Hystrix. Hystrix has a built-in system to query the metrics that drive the framework. For example you can query the metrics of each command such as the mean execution time or whether the circuit breaker for that command has tripped. And it’s that last one that is very interesting to the health indicator of your application. Most production environments have a dashboard that show the health of an application’s instances. If the circuitbreaker has tripped, your application is essentially in an unhealthy state. The circuit breaker mechanism will ensure that failures won’t cascade, but in a clustered environment you’d want that server removed from the pool or at least have a general indication that something is wrong. Spring Boot’s health endpoints works by querying various indicators. Like most things in Spring Boot, indicators are only active if there are components that can be checked. For example, if you have a datasource, an indicator will become active checking the state of that datasource. The same thing happens with NoSQL or AMQP connections. A simple implementation with Hystrix, which I’ll show in a minute, could be that when there is a tripped circuitbreaker in the system, the health of the application might be ‘out of service’. This is actually very easy to do. You just need to add a bean in your configurations returning an implementation of AbstractHealthIndicator: class HystrixMetricsHealthIndicator extends AbstractHealthIndicator { @Override protected void doHealthCheck(Health.Builder builder) throws Exception { def breakers = [] HystrixCommandMetrics.instances.each { def breaker = HystrixCircuitBreaker.Factory.getInstance(it.commandKey) def breakerOpen = breaker.open?:false if(breakerOpen) { breakers << it.commandGroup.name() + "::" + it.commandKey.name() } } breakers ? builder.outOfService().withDetail("openCircuitBreakers", breakers) : builder.up() } } Whenever a circuitbreaker gets tripped, the health endpoint will return the state of the application as OUT_OF_SERVICE and will also return the name of the open circuit breakers (the command key and the group it’s in). Now, this implementation can go a whole lot further. For example, you can add a new state to the health indication, for example UNSTABLE. This will however require you to change to order of the health aggregator, as Spring Boot will aggregate all the indicators and show a single application state. The new state needs to be fit in the existing order of states (DOWN > OUT_OF_SERVICE > UP > UNKNOWN). In the case of UNSTABLE, it would probably be between OUT_OF_SERVICE and UP. I can also think of a use-case in which the tripping of certain circuit breakers may be more critical than others, in which case the state of the application might really become OUT_OF_SERVICE. In that case you might decide to remove the instance from the pool of available instances (in a clustered environment) or restart the server. Or you can automate the process :). The last use case I’ll discuss is when your application is slow or is getting hammered by requests, which can be detected by Hystrix as well. In this case, you can introduce yet another state STRUGGLING, which would logically be between UNSTABLE and UP. In this case you can automate a process that starts up another instance and add it automatically to the pool. You can also see this the other way around, adding a state UNUSED which is on the same level as UP. This might indicate you have too many instances running and can possible shutdown that node (if it’s not the only one), or that you need to take a look at the load-balancing. As you can see, with such mechanisms it becomes possible to create a self-regulating instance pool, creating and removing instances as it goes. The health indicators in Spring Boot are an invaluable tool for DevOps teams and show how versatile Spring Boot actually is. UPDATE: Normally, if you want to alter the order in which statuses are aggregated, you can use a property in your application.properties like health.status.order = DOWN,OUT_OF_SERVICE,UNSTABLE,STRUGGLING,UP,UNKNOWN as documented. However, if you’re using the YAML-style properties, you’re out of luck, as there’s an annoying bugthat’s restricting you from using this feature. So if you’re using YAML properties, you’ll have to configure the HealthAggregator yourself. Luckily, this isn’t that hard, just add this bean to your application context: @Bean HealthAggregator healthAggregator() { def healthAggregator = new OrderedHealthAggregator(); healthAggregator.setStatusOrder(["DOWN", "OUT_OF_SERVICE", "UNSTABLE", "UP", "UNKNOWN"]); return healthAggregator; } Why they didn’t use the @EnableConfigurationProperties in the HealthIndicatorAutoConfiguration is a mystery to me, as this would have solved the issue. Perhaps I’ll do it myself and make a pull request.

Learn Fibonacci Series patterns and best practices with easy Java 8 source code examples in this outstanding tutorial by Pierre-Yves Saumont

Overview : Recently, I was working on a project which uses a REST services layer to communicate with the client application (GWT application). So I have spent a lot of to time to figure out how to secure the REST services with Spring Security. This article describes the solution I found, and I have implemented. I hope that this solution will be helpful to someone and will save a much valuable time. The solution : In a normal web application, whenever a secured resource is accessed Spring Security check the security context for the current user and will decide either to forward him to login page (if the user is not authenticated), or to forward him to the resource not authorised page (if he doesn’t have the required permissions). In our scenario this is different, because we don’t have pages to forward to, we need to adapt and override Spring Security to communicate using HTTP protocols status only, below I liste the things to do to make Spring Security works best : The authentication is going to be managed by the normal form login, the only difference is that the response will be on JSON along with an HTTP status which can either code 200 (if the autentication passed) or code 401 (if the authentication failed) ; Override the AuthenticationFailureHandler to return the code 401 UNAUTHORIZED ; Override the AuthenticationSuccessHandler to return the code 20 OK, the body of the HTTP response contain the JSON data of the current authenticated user ; Override the AuthenticationEntryPoint to always return the code 401 UNAUTHORIZED. This will override the default behavior of Spring Security which is forwarding the user to the login page if he don’t meet the security requirements, because on REST we don’t have any login page ; Override the LogoutSuccessHandler to return the code 20 OK ; Like a normal web application secured by Spring Security, before accessing a protected service, it is mandatory to first authenticate by submitting the password and username to the Login URL. Note: The following solution requires Spring Security in version minimum 3.2. Overriding the AuthenticationEntryPoint : Class extends org.springframework.security.web.AuthenticationEntryPoint, and implements only one method, which sends response error (with 401 status code) in cause of unauthorized attempt. @Component public class HttpAuthenticationEntryPoint implements AuthenticationEntryPoint { @Override public void commence(HttpServletRequest request, HttpServletResponse response, AuthenticationException authException) throws IOException { response.sendError(HttpServletResponse.SC_UNAUTHORIZED, authException.getMessage()); } } Overriding the AuthenticationSuccessHandler : The AuthenticationSuccessHandler is responsible of what to do after a successful authentication, by default it will redirect to an URL, but in our case we want it to send an HTTP response with data. @Component public class AuthSuccessHandler extends SavedRequestAwareAuthenticationSuccessHandler { private static final Logger LOGGER = LoggerFactory.getLogger(AuthSuccessHandler.class); private final ObjectMapper mapper; @Autowired AuthSuccessHandler(MappingJackson2HttpMessageConverter messageConverter) { this.mapper = messageConverter.getObjectMapper(); } @Override public void onAuthenticationSuccess(HttpServletRequest request, HttpServletResponse response, Authentication authentication) throws IOException, ServletException { response.setStatus(HttpServletResponse.SC_OK); NuvolaUserDetails userDetails = (NuvolaUserDetails) authentication.getPrincipal(); User user = userDetails.getUser(); userDetails.setUser(user); LOGGER.info(userDetails.getUsername() + " got is connected "); PrintWriter writer = response.getWriter(); mapper.writeValue(writer, user); writer.flush(); } } Overriding the AuthenticationFailureHandler : The AuthenticationFaillureHandler is responsible of what to after a failed authentication, by default it will redirect to the login page URL, but in our case we just want it to send an HTTP response with the 401 UNAUTHORIZED code. @Component public class AuthFailureHandler extends SimpleUrlAuthenticationFailureHandler { @Override public void onAuthenticationFailure(HttpServletRequest request, HttpServletResponse response, AuthenticationException exception) throws IOException, ServletException { response.setStatus(HttpServletResponse.SC_UNAUTHORIZED); PrintWriter writer = response.getWriter(); writer.write(exception.getMessage()); writer.flush(); } } Overriding the LogoutSuccessHandler : The LogoutSuccessHandler decide what to do if the user logged out successfully, by default it will redirect to the login page URL, because we don’t have that I did override it to return an HTTP response with the 20 OK code. @Component public class HttpLogoutSuccessHandler implements LogoutSuccessHandler { @Override public void onLogoutSuccess(HttpServletRequest request, HttpServletResponse response, Authentication authentication) throws IOException { response.setStatus(HttpServletResponse.SC_OK); response.getWriter().flush(); } } Spring security configuration : This is the final step, to put all what we did together, I prefer using the new way to configure Spring Security which is with Java no XML, but you can easily adapt this configuration to XML. @Configuration @EnableWebSecurity public class WebSecurityConfig extends WebSecurityConfigurerAdapter { private static final String LOGIN_PATH = ApiPaths.ROOT + ApiPaths.User.ROOT + ApiPaths.User.LOGIN; @Autowired private NuvolaUserDetailsService userDetailsService; @Autowired private HttpAuthenticationEntryPoint authenticationEntryPoint; @Autowired private AuthSuccessHandler authSuccessHandler; @Autowired private AuthFailureHandler authFailureHandler; @Autowired private HttpLogoutSuccessHandler logoutSuccessHandler; @Bean @Override public AuthenticationManager authenticationManagerBean() throws Exception { return super.authenticationManagerBean(); } @Bean @Override public UserDetailsService userDetailsServiceBean() throws Exception { return super.userDetailsServiceBean(); } @Bean public AuthenticationProvider authenticationProvider() { DaoAuthenticationProvider authenticationProvider = new DaoAuthenticationProvider(); authenticationProvider.setUserDetailsService(userDetailsService); authenticationProvider.setPasswordEncoder(new ShaPasswordEncoder()); return authenticationProvider; } @Override protected void configure(AuthenticationManagerBuilder auth) throws Exception { auth.authenticationProvider(authenticationProvider()); } @Override protected AuthenticationManager authenticationManager() throws Exception { return super.authenticationManager(); } @Override protected void configure(HttpSecurity http) throws Exception { http.csrf().disable() .authenticationProvider(authenticationProvider()) .exceptionHandling() .authenticationEntryPoint(authenticationEntryPoint) .and() .formLogin() .permitAll() .loginProcessingUrl(LOGIN_PATH) .usernameParameter(USERNAME) .passwordParameter(PASSWORD) .successHandler(authSuccessHandler) .failureHandler(authFailureHandler) .and() .logout() .permitAll() .logoutRequestMatcher(new AntPathRequestMatcher(LOGIN_PATH, "DELETE")) .logoutSuccessHandler(logoutSuccessHandler) .and() .sessionManagement() .maximumSessions(1); http.authorizeRequests().anyRequest().authenticated(); } } This was a sneak peak at the overall configuration, I attached in this article a Github repository containing a sample project https://github.com/imrabti/gwtp-spring-security. I hope this will help some of you developers struggling to figure out a solution, please feel free to ask any questions, or post any enhancements that can make this solution better.

Making your application resilient to failure can seem like a daunting task. Those who read “Release It!” know how many aspects there can be to making your application ready for the apocalypse. Luckily we live in a world where a lot of software needs such resilience and where there are companies who are willing to share their solutions. Enter what Netflix has created: Hystrix. Hystrix is a Java library aimed towards making integration points less susceptible to failures and mitigating the impact a failure might have on your application. It provides the means to incorporate bulkheads, circuit breakers and metrics into your framework. Those not familiar with these concepts should read the book I mentioned earlier. For example, a circuit breaker makes sure that if a certain integration point is having trouble, your application will not be affected. If for example a integration point takes 20 seconds to reply instead of the normal 50ms, you can configure a circuit breaker that trips if 10 calls within 10 seconds take longer than 5 seconds. When tripped, you can configure a quick fallback or fail fast. Hystrix has an elegant solution for this. Every command to an external integration point should get wrapped in a HystrixCommand. HystrixCommand provide support for circuit breakers, timeouts, fallbacks and other disaster recovery methods. So instead of directly calling the integration point, you’ll call a command that in turn calls the integration point. Hystrix also allows you to choose whether you want to do this synchronously or asynchronously (returning a Future). One of the really nice things about Hystrix is that it also has support for metrics and even has a nice dashboard to show those metrics. I can almost imagine that every development team has this on the dashboard next to the Hudson/Jenkins monitor in the near future, just because it’s so trivial to incorporate. Now, creating a new subclass for each and every distinct call to an integration endpoint may seems like a lot of work. It is, but the reasoning behind this is that incorporating Hystrix in your application should be explicit. However, if you really don’t like this, Hystrix also supports Spring AOP and has a aspect that does most of the work for you, using a contributed module (javanica). The only thing you need to do is annotate the methods you want covered by Hystrix. Whenever I see decent Spring integration, I now immediately look at Spring Boot support. Hystrix doesn’t have autoconfiguration for Spring Boot yet, but it’s really easy to implement. I used the annotation/aspect approach because I’m lazy and I like the transparency of going down this path. First you need to add a couple of dependencies. Here’s what you need in Gradle: compile("com.netflix.hystrix:hystrix-javanica:1.3.16") compile("com.netflix.hystrix:hystrix-metrics-event-stream:1.3.16") Then you need to create a configuration for Hystrix. I opted to create the configuration just like any other autoconfiguration module in Spring Boot (an @Configuration annotated class and a class describing the configuration properties). I also used conditional beans so that the . /** * {@link EnableAutoConfiguration Auto-configuration} for Hystrix. * * @author Lieven Doclo */ @Configuration @EnableConfigurationProperties(HystrixProperties) @ConditionalOnExpression("\${hystrix.enabled:true}") class HystrixConfiguration { @Autowired HystrixProperties hystrixProperties; @Bean @ConditionalOnClass(HystrixCommandAspect) HystrixCommandAspect hystrixCommandAspect() { new HystrixCommandAspect(); } @Bean @ConditionalOnClass(HystrixMetricsStreamServlet) @ConditionalOnExpression("\${hystrix.streamEnabled:false}") public ServletRegistrationBean hystrixStreamServlet(){ new ServletRegistrationBean(new HystrixMetricsStreamServlet(), hystrixProperties.streamUrl); } } /** * Configuration properties for Hystrix. * * @author Lieven Doclo */ @ConfigurationProperties(prefix = "hystrix", ignoreUnknownFields = true) class HystrixProperties { boolean enabled = true boolean streamEnabled = false String streamUrl = "/hystrix.stream" } In short, if you add this to your Spring Boot application, Hystrix will be automatically integrated in your application. As you might have seen, I’ve also added some configuration properties. I added support for the event stream that powers the dashboard and which is only activated if you add hystrix.streamEnabled = true to your application.properties. The URL through which the stream is served is also configurable (but has a sensible default). If you want, you can disable Hystrix as a whole by adding hystrix.enabled = false to your application.properties. This code is actually ready to be put into Spring Boot’s autoconfigure module :). Two simple classes and two simple dependencies and your code is ready for the apocalypse. Doesn’t seem like a bad deal to me. Hystrix has a lot more to offer than I touched in this article (command aggregation, reactive calls through events, …). If your application has a lot of integration points, certainly have a look at this library. Your application may be stable, but that doesn’t mean that all the REST services you’re calling are.

i am eclipse user. so when i wanted to analyze my code with pmd, i needed to use “pmd for eclipse” plugin. this plugin used to be very buggy, which was enhanced in later versions (currently 4.0.3). but the performance is really bad sometimes. especially when you are dealing with relatively big codebase and have option “check code after saving” on. ecplise-pmd plugin so when i realized that there is new alternative pmd plugin called eclipse-pmd out there i evaluated it immediately with great happiness. installation uses modern eclipse marketplace method. you just need to go “help” -> “eclipse marketplace…” and search for “eclipse-pmd” . than hit “install” and follow instructions. after installation i was a little bit confused because i didn’t find any configuration options it general settings ( “window” -> “preferences” ). i discovered that you need to turn on pmd for each project separately. which make sense, because you can have different rule set per project. so to turn it on, right click on project -> “preferences” -> “pmd” (there would be two pmd sections if you didn’t uninstall old pmd plugin) -> “enable pmd for this project” -> “add…” . now you should pick a location of pmd ruleset file. unlike old pmd plugin, eclipse-pmd don’t import ruleset. it is using ruleset file directly. this is very handy, because typically you want to have it in source control. when you pull changes to ruleset file from source control system, they are applied without re-import (re-import was needed for old pmd plugin). problem can be when you (or your team) don’t have existing ruleset. i would suggest to start with full ruleset and exclude rules you don’t want to use. your ruleset would evolve anyway, so starting with most restrictive (default) deck make perfect sense for me. unfortunately eclipse-pmd plugin doesn’t provide option to generate ruleset file. so i created full ruleset for pmd 5.1.1 (5.1.1 is pmd version not plugin version) . i have to admit that it was created with help of old pmd plugin. you can see that i literally included all the rule categories. i would suggest to specify your set this way and exclude/configure rules explicitly as needed. here is link to pmd site that explains how to customize your ruleset . this approach can be handy when pmd version will be updated. new rules can appear in category and they will be automatically included into your ruleset when you are listing categories, not rules individually. but you have to keep eye on new rules/categories when updating pmd version anyway, because categories often change with new pmd version. so now we should have rulset configured and working. here are some screen shots of rules in action: when you hover over left side panel warning: when you hover over problematic snippet: when you do quick fix on problematic snippet: generating suppress warning annotation for pmd rules is very nice feature. it also provide quick fixes for some rules. take a look at its change log site for full list. these pmd warning sometimes clash with eclipse native warnings, so there is possibility to make them more visible. go to “window” -> “preferences” -> “general” -> “editors” -> “text editors” -> “annotations” and find “pmd violations” . here you can configure your own style of highlighting pmd issues. this is mine: to explore full feature list of this plugin take a look at its change log site. there are some features in old plugin i miss though. for example i would appreciate some quick link or full description of the rule. short description provided is sometimes not enough. i encourage you to take a look at full pmd rule description if you are not sure what’s source of the problem. you will learn a lot about java language itself or about libraries you are using. quick links would help a lot in such case. also some rules doesn’t use code highlighting (only side panel markers). it is sometimes hard to distinguish between compiler and pmd issues. this is problem for me because our team doesn’t use javadoc warnings but i do. so i get a lot of javadoc warnings from code written by teammates. and sometimes i can miss pmd issue because it is lost in javadoc warnings. (fortunately svn commit is rejected if i forget to fix some rule). conclusion this plugin enhanced my eclipse workflow. no more disruptions because of endless “checking code…” processing by old plugin.

Security is one of the most important features while running a JBoss server in a production environment. Implementing SSL and securing communications is a must do, to avoid malicious use. This blogs details the steps you could take to secure JBoss EAP 6 running in Domain mode. These are probably documented by RedHat but the documentation seems a bit scattered. The idea behind this blog is to put together everything in one place. In Order to enhance security in JBoss EAP 6, SSL/encryption can be implemented for the following Admin console access – enable https access for admin console Domain Controller – Host controller communication – Communication between the main domain controller and all the other host controllers should be secured. Jboss CLI – enable ssl for the command line interface The below example uses a single keystore being both the key and truststore and also uses CA signed certificates. You could use self-signed certificates and/or separated keystores and truststores if required. Create the keystores (certificates for each of the servers) keytool -genkeypair -alias testServer.prd -keyalg RSA -keysize 2048 -validity 730 -keystore testServer.prd.jks Generate a certificate signing request (CSR) for the Java keystore keytool -certreq -alias testServer.prd -keystore testServer.prd.jks -file testServer.prd.csr Get the CSR signed by the Certificate Authorities Import a root or intermediate CA certificate to the existing Java keystore keytool -import -trustcacerts -alias root -file rootCA.crt -keystore testServer.prd.jks Import the signed primary certificate to the existing Java keystore. Keytool -importcert -keystore testServer.prd.jks -trustcacerts -alias testServer.prd -file testServer.prd.crt Repeat steps 1-6 for each of the servers. In order to establish trust between the master and slave hosts, Import the signed certificates of all the (slave) servers that the Domain Controller must trust onto the Domain Controllers Keystore keytool -importcert -keystore testServer.prd.jks -trustcacerts -alias slaveServer.prd -file slaveServers.prd.crt repeat step for all slave hosts. Import the signed certificate of the Domain controller onto the slave hosts keytool -importcert -keystore slaveServer.prd.jks -trustcacerts -alias testServer.prd -file testServer.prd.crt repeat steps for all slave hosts This has be to done because (as per RedHat’s Documentation) There is a problem with this methodology when trying to configure one way SSL between the servers, because there the HC's and the DC (depending on what action is being performed) switch roles (client, server). Because of this one way SSL configuration will not work and it is recommended that if you need SSL between these two endpoints that you configure two way SSL Once this is done, we now have signed certificates loaded onto the java keystore. In Jboss EAP 6 , the http-interface which provides access to the admin console, by default uses the ManagementRealm to provide file based authentication. (mgmt.-users.properties).The next step is to modify the configurations in the host.xml, to make the ManagementRealm use the certificates we created above. The host.xml should be modified to look like: view source print? 01. 02. 03. 04. 05. 06. 07. 08. 09. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31. 32. 33. 34. 35. 36. 37. 38. 39. 40. 41. 42. 43. On the Slave hosts, In addition to the above configuration, the following needs to be changed view source print? 1. 2. 3. " 4. 5. Once you make the above changes and restart the servers, you should be able to access the admin console via https. https://testServer.prd:9443/console Finally, in order to secure cli authentication Modify /opt/jboss/jboss-eap-6.1/bin/jboss-cli.xml for each server and add view source print? 01. 02. 03. testServer.prd 04. 05. /opt/jboss/jboss-eap-6.1/domain/configuration/testServer.prd.jks 06. 07. xxxx 08. 09. /opt/jboss/jboss-eap-6.1/domain/configuration/testServer.prd.jks 10. 11. xxxx 12. 13. true 14. 15.

[This article was written by John Wetherill.] This is a continuation of the Microservices and PaaS - Part I blog post I wrote last week, which was an attempt to distil the wealth of information presented at the microservices meetup hosted by Cisco, with Adrian Cockcroft and others presenting. Part I provided a brief background on microservices, with a summary of some lessons learned by microservices pioneers. In this installment I will cover a number of practices related to microservices that were discussed during the meetup. A followup article will dive into the advantages that Platform as a Service brings to microservice development. Microservices Practices I'm calling these "Microservice Practices," not "Microservices Best Practices" because microservices-based architectures are still evolving, with new practices, techniques, tools, and patterns emerging constantly. At the meetup a number of practices were highlighted that Netflix and other microservices pioneers have spearheaded in their efforts to adopt a microservices mentality across their organizations. Break Things Deliberately According to Netflix: "We have found that the best defense against major unexpected failures is to fail often." Netflix has brought us "Chaos Monkey" which is a powerful tool the sole purpose of which is to break things, often and randomly. They use this tool continuously on their production systems to bring down essential services, to ensure that doing so doesn't disrupt the user experience or their overall service. It's much better to deliberately break the system in the middle of the morning when all teams are assembled and sufficient caffeine has been consumed, than to be informed of a breakage by a page at 3am. No Manual "Anything" In a world where microservices come and go, grow and shrink, and migrate around racks and data centers in seconds - there's absolutely no room for manual intervention. All aspects of deployment, monitoring, testing, and recovery must be fully automated. For example, monitoring a service should occur instantly and automatically by virtue of it being deployed, not requiring a separate manual step. Similarly failure discovery and rerouting to old code, as described in Part I of this blog, must be fully automated, no human intervention required. Respect Human Attention Span Speaking of humans, a typical human's attention span, say when filling out a shopping cart, is around 10 seconds. If a failure occurs when deploying an updated shopping cart microservice, it's important that the time between the failure, reporting, and rerouting to existing, working code is kept under around this 10 second range. Obviously this shouldn't happen too often, but the occasional 10 second gap in response will probably not lose the customer. A five minute, or 5 hour lag, resulting from manual intervention and rollback, will. Denormalize like Crazy Refactor database schemas, and de-normalize everything, to allow complete separation and partitioning of data. That is, do not use underlying tables that serve multiple microservices. There should be no sharing of underlying tables that span multiple microservices, and no sharing of data. Instead, if several services need access to the same data, it should be shared via a service API (such as a published REST or a message service interface). Polyglot Persistence Each microservice can have its own persistence layer. Gone are the days of a single monolithic database instance that's shared across all parts of an application. Databases are getting cheaper and easier. As an example, Neo4J allows you to embed an industry-strength self-contained graph database in your microservice at the cost of a few megabytes in a jarfile, with startup time on the order of milliseconds. That's essentially free. Even better, any PaaS worth its salt will provide multiple database services that can be spawned and accessed at the drop of a hat. With technology like this at our disposal, it makes sense to use the persistence layer that fits, both to the problem being solved, and to the expertise - and passions - of the team that's solving the problem. Avoid Trunk Conflicts The old mindset had all code for a large project contained in a single source repository. This can be slightly easier to setup and manage, but it ties the microservices together and makes it much more difficult to evolve them independently. Instead each microservice should have its own scm repository so it can truly be updated and enhanced independent of other services. One Service, One Manifest Each microservice must have its own manifest and dependencies, instead of maintaining a global dependency list for all services. This allows, for example, one microservice to depend on Spring v3.2, while another can require Spring 4.1. The dependencies for one microservice can change over time with no effect on the dependencies of other microservices. Contain Everything All microservices should run in a container, such as Tomcat, Docker, or in whatever container system is provided by the PaaS (you are running a PaaS aren't you?). Do not run microservices on bare metal, or directly on a VM. Containerization brings countless advantages, particularly a consistent, isolated runtime environment that can easily migrate around the datacenter or around the globe. With Docker and other modern containerization approaches, there is very little overhead in running in a container, and considerable upside. No State Do not build stateful services. Instead, maintain state in a dedicated persistence service, or elsewhere. This is a well-known practice brought to us by the cloud. When an application instance maintains state, it can't easily be moved, scaling is more complex, and it's more likely to cause problems when it fails. This practice applies even more to microservices which in general should be light-weight, instantly replaceable on failure, and should be able to hop around data-centers. Don't Name your Chickens People who raise chickens soon learn that naming chickens is a bad idea: after naming a chicken you get attached to it, at least the kids do, and it can be uncomfortable to have to explain at the dinner table that the chicken pot pie is really "Molly." Instead, number your chickens, so you can say "that was chicken #38" or even better, "that was chicken 586ec9bd." Makes for a much more enjoyable meal. The same can be said of computer systems. Do not name systems after planets, or animals, or philosophers, or prisons, as was common practice in the UNIX world for decades. Instead, assign them guid's, and don't attach any sort of significance to them, like assigning them specific roles or purposes. Systems should be commodities, like McDonalds Franchises. Each McDonalds is eerily similar, with the advantage that if one shuts down you can just walk an extra few blocks and be served the exact same burger at the same price in the same amount of time. Create and Curate Access Libraries Microservices are accessed by externally published APIs or protocols. This allows the microservice implementation to completely change with no effect on its consumers, as long as the API remains constant. But just publishing an API is not enough. The microservice provider should also be responsible for building and stewarding client libraries used to access the service. If this is not done, the construction of these libraries will be left to third parties, and will likely result in fragmentation where various implementations might have slight differences, or implementors may incorrectly interpret the spec and introduce inconsistencies which then stick. Optimize the Interaction One downside of a microservices architecture is the "fanout" problem where a single request to the overall application results in 10 or 20 requests bubbling throughout the various microservices the application relies on. This dramatic increase in network traffic calls for more optimal communication between microservices. Instead of transmitting the standard text/html REST content type, consider using something like Google Protocol Buffers, Simple Binary Encoding, or Apache Thrift, to decrease the size of the payload and optimize the inter-microservice communications. Release the Monkeys Netflix has released what they call the "Simian Army," a suite of tools including Chaos Monkey, mentioned above, whose purpose is to help an organization build resilient, scalable, fault-tolerant software. The suite includes such tools as Janitor Monkey, to reclaim unused resources, Security Monkey which looks for security vulnerabilities, Latency Monkey, which induces artificial delays in the REST layer to scare out latency issues, and many more. As Phil described last week in his blog Devops: Tools vs. Culture, most organizations don't have the resources or luxury of being able to build their own toolsets when evolving to a microservices and devops culture. Instead they must leverage existing tools, and fortunately lots of tools are constantly appearing. It's worth spending the effort searching and researching these tools, and incorporating them into your overall development process when they make sense. To be continued... Again I originally intended to cover last week's microservices meetup in a single blog post, which then expanded to two. I have yet to address the power of PaaS in microservices architectures, and I'm out of space already. So I will continue this Microservices and PaaS theme next week, finally getting into PaaS, and discuss how Platform as a Service can significantly streamline the microservices development process.