Release Engineering & Parallel Deployment

“tomcat” is always considered as a “poor-man’s server”, but the tomcat 7 has something really interesting to offer i.e. parallel deployment .this was contributed by springsource (now vmware ).parallel deployment is an approach that allows several online deployment operations to run simultaneously or in simple words it is the ability to deploy more than one version of your web application in parallel, making all versions available under the exact same url. the parallel deployment also helps in the patch releases and advocating the best version of the application based on the performance and traffic

the most exciting thing with this feature is “ zero down-time release (deployment and rollback) process ”. whenever we go for a deployment process, we shutdown the server, remove all active connections, drain out the older version and move or fire the updated version of the application. considering from a real-time project scenario, the developer does not have to get approval from his manager and the manager does not have to get necessary approvals from the customer – as there is no downtime impact involved. so, the developer himself can push the latest code and test/monitor the application with the live users.

if, we can co-relate this with the approach, facebook, flickr, google and many other companies have been following where the developer directly pushes the code to production and shepards it until any unforeseen issue is reported by the user. as a quick statistics, facebook developers push the code to production “daily twice” with millions lines of code five days a week. similarly flickr pushes 10 deploys per day ( full article - slideshare)

one of the finest videos explaining the release engineering at facebook – click here

here is a simple usecase for testing with multiple variants of the application :

scenarios :

• push three different versions of a hello world program v001, v002 and v003, one after the other and then monitoring the static application url that keeps changing as and when the updated version is deployed/rolled-out
• undeploy a couple of versions (the case of application rollback) to see, how the application responds.

pictorial representation of the scenario:

steps :

• create a simple index.html as represented below (the versions shows the changes that needs to be incorporated with multiple version releases)

• create a build.xml as shown below, the undeploy target is used during rollback. during the deploy step execution a new version of war file is created “pdrelease##001” and is copied to the tomcat/webapps folder. for multiple release versions the count is increased. the general approach for creating a new versioned war file is to have the format <applicationname>##< version number>.war :

<?xml version="1.0" encoding="iso-8859-1"?>
<project name="testing the parallel deployment process" basedir=".">
<property name="warfile" value="pdrelease##001"/>
<target name="unpack">
<unwar src="${warfile}.war" dest="${warfile}" />
</target>
<target name="create">
<war destfile="${warfile}.war" webxml="web-inf/web.xml" update="true">
<fileset dir=".">
<include name="index.html"/>
</fileset>
</war>
</target>
<target name="copy">
<copy todir="e:\test\releasemgmt_tomcat\webapps" overwrite="true">
<fileset dir=".">
<include name="*.war"/>
</fileset>
</copy>
</target>
<target name="undeploy">
<delete dir="e:\test\releasemgmt_tomcat\webapps\sanfran##004"/>
<delete>
<fileset dir="e:\test\releasemgmt_tomcat\webapps">
<include name=" pdrelease##002.war"/>    
</fileset>
</delete>
</target>  
<target name="deploy">
<antcall target="create"/>
<antcall target="copy"/>
</target>
</project>

• after a successful execution of the build.xml,  the portal is accessed :
• buildfile: c:\users\user\workspace\paralleldeployment\build.xml
• create:
• [war] building war: c:\users\user\workspace\paralleldeployment\pdrelease##001.war
• copy:
• [copy] copying 5 files to e:\test\releasemgmt_tomcat\webapps
• build successful
• total time: 1 second

• next, the changes in the index.html are incorporated and the new versions of war files are deployed to access the portal. (for showcase – opened it in 3 different browsers). otherwise it will always take the highest version of the application. with each individual browser, the different versions of the applications can be accessed.

• quick snapshot of the webapps folder :

• the version v003 is un-deployed or rolled back , so that the highest version of the application available is pdrelease##002

• the logs can be verified to understand the history, as to which application version was deployed or un-deployed

nov 03, 2012 8:02:14 pm org.apache.catalina.startup.hostconfig deploywar
info: deploying web application archive e:\test\releasemgmt_tomcat\webapps\pdrelease##001.war
nov 03, 2012 8:02:14 pm org.apache.catalina.startup.hostconfig deploywar
info: deploying web application archive e:\test\releasemgmt_tomcat\webapps\pdrelease##002.war
nov 03, 2012 8:02:14 pm org.apache.catalina.startup.hostconfig deploywar
info: deploying web application archive e:\test\releasemgmt_tomcat\webapps\pdrelease##003.war
nov 03, 2012 8:02:15 pm org.apache.catalina.startup.hostconfig deploywar
nov 03, 2012 8:10:00 pm org.apache.catalina.startup.hostconfig checkresources
info: undeploying context [/pdrelease##003]

things to be taken care of :

• internal cache should be written separately as multiple versions of the application will be accessing and going on/off. a perfect example to visualize this situation would be to updating the database based on a caching mechanism.
• logging (where does the logs get written for different versions of the application)
• session handling is currently taken care of by tomcat. so any changes to the session handling configurations will affect the parallel deployment approach.
• only one listener can listen on any given port.
• from an enterprise application perspective, there might be a situation wherein multiple versions of the app, will be accessing a common file store of database row. so how do we handle the locking mechanism for a smooth and hassle free access.
• rolling back of changed schemas, data entry row updates in the database, webservice configuration changes

best practice:

• as a best practice, the version numbers for the war files can be numbered in sync with the cvs or svn version of the application in the repository. so, the war file name can be mentioned as <applicationname>##<svn or cvs version number>.war
• strategize and draft a schedule for the deployment approach when and under what condition the code needs to be pushed or rollback

parallel deployment is not suited for cases like :

• dealing with web-applications requiring in-memory caching would be challenging as the cache will have the stale information and will not serve the purpose.
• web applications which require, offline deployment and restart or server–in that case parallel deployment approach is not a best fit.

advantages of parallel deployment or phased release engineering:

• speed and scale of deployment
• easy deployment and rollback – facilitating continuity of business without any show-stopper with issues related to the new deployment. the last best version can always be rolled back.
• a/b or split testing
• better end user testing – pushing the code to production or any higher environment is always a nail-biting situation. with parallel deployment and patch release approach the code can be silently pushed to production and get it tested with the end-user.
• provides a better ground for experimentation and deploy without any fear.
• a cushion for the developers and the customers with incremental releases.
• a developer can act as a release engineer and vice-versa.
• the developer shepard’s his change to the world

other tools to explore:

there are many other tools which the companies like facebook and google use  pushing the code on an incremental and decremental fashion - ‘need to explore to figure out how exactly it is done using the tools. couple of other tools 'came across are

• bosh (developed by the cloudfoundry - vmware team and now opensourced)
• capistrano (another oss).
• asgard (oss) – also provides an application deployment and version rollback feature on amazon ec2 instance

references:

• apache tomcat 7
• tomcat – parallel deployment
• what is parallel deployment ?
• advantages of parallel deployment
• wiki - release engineering
• interesting article : four principles of low-risk software releases
• interesting videos and tech talks : facebook engineering unleashed
• pushing millions lines of code 5 days a week - fb

pn : please do keep posting your valuable comments and suggestions on the same and feel free to correct me if my understanding is wrong.

happy learning :-)