in my last post on extjs 4 mvc, i have demonstrated the use of extjs 4 mvc to create a simple create-read-update-delete application using extjs only. today we will go to see how to use that extjs part for ui and use spring mvc to manage the books records on server side using spring. let first start by creating spring’s webmvc configurer class. i am using spring’s new xml-free approach and if you are new to it, i recommend you to please go through my previous post. package org.techzoo.springmvc.bootstrap; import org.springframework.context.annotation.bean; import org.springframework.context.annotation.componentscan; import org.springframework.context.annotation.configuration; import org.springframework.web.servlet.config.annotation.enablewebmvc; import org.springframework.web.servlet.config.annotation.resourcehandlerregistry; import org.springframework.web.servlet.config.annotation.viewcontrollerregistry; import org.springframework.web.servlet.config.annotation.webmvcconfigureradapter; import org.springframework.web.servlet.view.internalresourceviewresolver; @configuration @enablewebmvc @componentscan(basepackages = {"org.techzoo.springmvc"}) public class mywebmvcconfigurer extends webmvcconfigureradapter { @override public void addresourcehandlers(resourcehandlerregistry registry) { registry.addresourcehandler("/resources/**") .addresourcelocations("/resources/"); } @override public void addviewcontrollers(viewcontrollerregistry registry) { registry.addviewcontroller("/") .setviewname("index"); } @bean public internalresourceviewresolver viewresolver() { internalresourceviewresolver resolver = new internalresourceviewresolver(); resolver.setprefix("/web-inf/views/"); resolver.setsuffix(".jsp"); return resolver; } } create a book bean now, this bean is work as a value object and represent a single record in grid. package org.techzoo.springmvc.vo; /** * author: tousif khan */ public class book { private int id; private string title; private string author; private int price; private int qty; public book() {} public book(int id, string title, string author, int price, int qty) { super(); this.id = id; this.title = title; this.author = author; this.price = price; this.setqty(qty); } //getter-setters goes here... @override public string tostring() { return string.format("book [title = %s, author = %s]", title, author); } } now create a bookdao interface and implement it. package org.techzoo.springmvc.dao; import java.util.list; import org.springframework.stereotype.component; import org.techzoo.springmvc.vo.book; @component public interface bookdao { public boolean addbook(book book); public void updatebook(book book); public list listbooks(); public book getbookbyid(integer bookid); public boolean removebook(book book); } to make it more simple, i am using book list but you can extend it to save it relational table. see my springmvc-hibernate grud tutorial for more details. package org.techzoo.springmvc.dao; import java.util.arraylist; import java.util.list; import org.springframework.stereotype.repository; import org.techzoo.springmvc.vo.book; @repository public class bookdaoimpl implements bookdao { private static list books = new arraylist(); static { books.add(new book(1001, "jdbc, servlet and jsp", "santosh kumar", 300, 12000)); books.add(new book(1002, "head first java", "kathy sierra", 550, 2500)); books.add(new book(1003, "java scjp certification", "khalid mughal", 650, 5500)); books.add(new book(1004, "spring and hinernate", "santosh kumar", 350, 2500)); books.add(new book(1005, "mastering c++", "k. r. venugopal", 400, 1200)); } @override public boolean addbook(book book) { return books.add(book); } @override public boolean removebook(book book) { return books.remove(book); } @override public list listbooks() { return books; } @override public void updatebook(book book) { int index = books.indexof(book); if(index != -1) { books.add(index, book); } } @override public book getbookbyid(integer bookid) { book b = null; for(book b1 : books) { if(b1.getid() == bookid) { return b1; } } return b; } } now it’s time to create a book controller. package org.techzoo.springmvc.controller; import java.util.hashmap; import java.util.list; import java.util.map; import org.springframework.beans.factory.annotation.autowired; import org.springframework.stereotype.controller; import org.springframework.validation.objecterror; import org.springframework.web.bind.methodargumentnotvalidexception; import org.springframework.web.bind.annotation.exceptionhandler; import org.springframework.web.bind.annotation.requestbody; import org.springframework.web.bind.annotation.requestmapping; import org.springframework.web.bind.annotation.requestmethod; import org.springframework.web.bind.annotation.responsebody; import org.techzoo.springmvc.dao.bookdao; import org.techzoo.springmvc.vo.book; @controller @requestmapping("api") public class bookcontroller { bookdao bookbao; @autowired public bookcontroller(bookdao bookbao) { this.bookbao = bookbao; } @requestmapping(value = "/index") public string index() { return "index"; } @requestmapping (value = "book/save", method = requestmethod.post) @responsebody public boolean savebook(@requestbody book book) { return bookbao.addbook(book); } @responsebody @requestmapping (value = "book/loadbooks") public map> loadallbooks() { map> books = new hashmap>(); books.put("books", bookbao.listbooks()); return books; } @requestmapping (value = "book/delete", method = requestmethod.post) @responsebody public boolean deletebooks(@requestbody book book) { return bookbao.removebook(book); } @requestmapping (value = "book/updatebook", method = requestmethod.post) @responsebody public boolean updatebooks(@requestbody book book) { bookbao.updatebook(book); return true; } } extjs code looks similar to previous example with a little change. i have changes the local store to server ajax proxy and after each delete/add/update operation, we will load the all record from server to make grid update. ext.onready(function () { ext.define('techzoo.model.book', { extend: 'ext.data.model', fields: [ {name: 'id', type: 'int'}, {name: 'title', type: 'string'}, {name: 'author', type: 'string'}, {name: 'price', type: 'int'}, {name: 'qty', type: 'int'} ] }); ext.define('techzoo.store.books', { extend : 'ext.data.store', storeid: 'bookstore', model : 'techzoo.model.book', fields : ['id', 'title', 'author','price', 'qty'], proxy: { type: 'ajax', url: '/springmvc-extjs-crud/api/book/loadbooks', reader: { type: 'json', root: 'books' } }, autoload: true }); ext.define('techzoo.view.bookslist', { extend: 'ext.grid.panel', alias: 'widget.bookslist', title: 'books list - (extjs springmvc example - @ tousif khan)', store: 'books', initcomponent: function () { this.tbar = [{ text : 'add book', action : 'add', iconcls : 'book-add' }]; this.columns = [ { header: 'title', dataindex: 'title', flex: 1 }, { header: 'author', dataindex: 'author' }, { header: 'price', dataindex: 'price' , width: 60 }, { header: 'quantity', dataindex: 'qty', width: 80 }, { header: 'action', width: 50, renderer: function (v, m, r) { var id = ext.id(); ext.defer(function () { ext.widget('image', { renderto: id, name: 'delete', src : 'resources/images/book_delete.png', listeners : { afterrender: function (me) { me.getel().on('click', function() { var grid = ext.componentquery.query('bookslist')[0]; if (grid) { var sm = grid.getselectionmodel(); var rs = sm.getselection(); if (!rs.length) { ext.msg.alert('info', 'no book selected'); return; } ext.msg.confirm('remove book', 'are you sure you want to delete?', function (button) { if (button == 'yes') { //grid.store.remove(rs[0]); var book = rs[0].getdata(); ext.ajax.request({ url: '/springmvc-extjs-crud/api/book/delete', method : 'post', jsondata: book, success: function(response){ var grid = ext.componentquery.query('bookslist')[0]; grid.getstore().load(); } }); } }); } }); } } }); }, 50); return ext.string.format('', id); } } ]; this.callparent(arguments); } }); ext.define('techzoo.view.booksform', { extend : 'ext.window.window', alias : 'widget.booksform', title : 'add book', width : 350, layout : 'fit', resizable: false, closeaction: 'hide', modal : true, config : { recordindex : 0, action : '' }, items : [{ xtype : 'form', layout: 'anchor', bodystyle: { background: 'none', padding: '10px', border: '0' }, defaults: { xtype : 'textfield', anchor: '100%' }, items : [{ name : 'title', fieldlabel: 'book title' },{ name: 'author', fieldlabel: 'author name' },{ name: 'price', fieldlabel: 'price' },{ name: 'qty', fieldlabel: 'quantity' }] }], buttons: [{ text: 'ok', action: 'add' },{ text : 'reset', handler : function () { this.up('window').down('form').getform().reset(); } },{ text : 'cancel', handler: function () { this.up('window').close(); } }] }); ext.define('techzoo.controller.books', { extend : 'ext.app.controller', stores : ['books'], views : ['bookslist', 'booksform'], refs : [{ ref : 'formwindow', xtype : 'booksform', selector: 'booksform', autocreate: true }], init: function () { this.control({ 'bookslist > toolbar > button[action=add]': { click: this.showaddform }, 'bookslist': { itemdblclick: this.onrowdblclick }, 'booksform button[action=add]': { click: this.doaddbook } }); }, onrowdblclick: function(me, record, item, index) { var win = this.getformwindow(); win.settitle('edit book'); win.setaction('edit'); win.setrecordindex(index); win.down('form').getform().setvalues(record.getdata()); win.show(); }, showaddform: function () { var win = this.getformwindow(); win.settitle('add book'); win.setaction('add'); win.down('form').getform().reset(); win.show(); }, doaddbook: function () { var win = this.getformwindow(); var store = this.getbooksstore(); var values = win.down('form').getvalues(); var action = win.getaction(); // var book = ext.create('techzoo.model.book', values); var url = ''; if(action == 'edit') { url = '/springmvc-extjs-crud/api/book/updatebook'; } else { url = '/springmvc-extjs-crud/api/book/save'; } ext.ajax.request({ url: url, method : 'post', jsondata: values, success: function(response){ store.load(); } }); win.close(); } }); ext.application({ name : 'techzoo', controllers: ['books'], launch: function () { ext.widget('bookslist', { width : 500, height: 300, renderto: 'output' }); } } ); }); output: open the html file in any browser, the output will look similar to below. you can click to add new book record in grid.

I've begun work on trying to answer the questions I gathered concerning Cordova's FileSystem support. As I work through the questions I'm trying to build "real" samples to go along with the text. My first sample is a simple one, but I think it is pretty relevant for the types of things folks may do with Cordova and the file system - checking to see if a file exists locally and if not - fetching it. I'll begin by sharing the code and then explaining the parts. Here is the entire JavaScript file for the application. (Earlier today, Andrew Grieve shared a way my code could be simplified by a good 1/3rd. The code below reflects his update and has been changed since my original writing of the blog post.) document.addEventListener("deviceready", init, false); //The directory to store data var store; //Used for status updates var $status; //URL of our asset var assetURL = "https://raw.githubusercontent.com/cfjedimaster/Cordova-Examples/master/readme.md"; //File name of our important data file we didn't ship with the app var fileName = "mydatafile.txt"; function init() { $status = document.querySelector("#status"); $status.innerHTML = "Checking for data file."; store = cordova.file.dataDirectory; //Check for the file. window.resolveLocalFileSystemURL(store + fileName, appStart, downloadAsset); } function downloadAsset() { var fileTransfer = new FileTransfer(); console.log("About to start transfer"); fileTransfer.download(assetURL, store + fileName, function(entry) { console.log("Success!"); appStart(); }, function(err) { console.log("Error"); console.dir(err); }); } //I'm only called when the file exists or has been downloaded. function appStart() { $status.innerHTML = "App ready!"; } Ok, let's break it down. The first step is to check to see if our file exists already. The question is - where should we store the file? If you look at the docs for the FileSystem, you will see that the latest version of the plugin adds some useful aliases for common folders. Unfortunately, the docs are not exactly clear about how some of these aliases work. I asked for help (both on the PhoneGap Google group and the Cordova development list) and got some good responses from Kerri Shotts and Julio Sanchez. The directory that I thought made sense, cordova.file.applicationStorageDirectory, is incorrectly documented as being writeable in iOS. A pull request has already been filed to fix this mistake. For my application, the most appropriate directory is the next one, cordova.file.dataDirectory. Once I have my directory alias, I can make use of resolveLocalFileSystem on the directory plus desired file name to see if it exists. The third argument, downloadAsset, will only be run on an error, in this case a file not existing. If the file does not exist, we then have to download it. For this we use a second plugin, FileTransfer. This is where one more point of confusion comes in. We need to convert that earlier DirectoryEntry object, the one we used to get an API for files and directories, back to a URL so we can give a path to the Download API. So to recap - we've got a few moving parts here. We've got a directory alias, built into the plugin for easily finding common folders for our application. Again, the docs here are currently a bit wrong but they should be corrected soon. From that we can quickly see if our desired file exists, and if not, use the FileTransfer plugin to download it. Simple... but even a simple application caused me a bit of trouble, so hopefully this helps others. You can get the full source code here: https://github.com/cfjedimaster/Cordova-Examples/tree/master/checkanddownload

Learn about Identity, Sequence, and Table in Hibernate.

When I was learning programming, back in the days of Turbo Pascal, I managed to list files in directory using FindFirst,FindNext and FindClose functions. First I came up with a procedure printing contents of a given directory. You can imagine how proud I was to discover I can actually call that procedure from itself to traverse file system recursively. Well, I didn't know the term recursion back then, but it worked. Similar code in Java would look something like this: public void printFilesRecursively(final File folder) { for (final File entry : listFilesIn(folder)) { if (entry.isDirectory()) { printFilesRecursively(entry); } else { System.out.println(entry.getAbsolutePath()); } } } private File[] listFilesIn(File folder) { final File[] files = folder.listFiles(); return files != null ? files : new File[]{}; } Didn't know File.listFiles() can return null, did ya? That's how it signals I/O errors, like if IOException never existed. But that's not the point. System.out.println() is rarely what we need, thus this method is neither reusable nor composable. It is probably the best counterexample of Open/Closed principle. I can imagine several use cases for recursive traversal of file system: Getting a complete list of all files for display purposes Looking for all files matching given pattern/property (also check out File.list(FilenameFilter)) Searching for one particular file Processing every single file, e.g. sending it over network Every use case above has a unique set of challenges. For example we don't want to build a list of all files because it will take a significant amount of time and memory before we can start processing it. We would like to process files as they are discovered and lazily - by pipe-lining computation (but without clumsy visitor pattern). Also we want to short-circuit searching to avoid unnecessary I/O. Luckily in Java 8 some of these issues can be addressed with streams: final File home = new File(FileUtils.getUserDirectoryPath()); final Stream files = Files.list(home.toPath()); files.forEach(System.out::println); Remember that Files.list(Path) (new in Java 8) does not look into subdirectories - we'll fix that later. The most important lesson here is: Files.list() returns a Stream - a value that we can pass around, compose, map, filter, etc. It's extremely flexible, e.g. it's fairly simple to count how many files I have in a directory per extension: import org.apache.commons.io.FilenameUtils; //... final File home = new File(FileUtils.getUserDirectoryPath()); final Stream files = Files.list(home.toPath()); final Map> byExtension = files .filter(path -> !path.toFile().isDirectory()) .collect(groupingBy(path -> getExt(path))); byExtension. forEach((extension, matchingFiles) -> System.out.println( extension + "\t" + matchingFiles.size())); //... private String getExt(Path path) { return FilenameUtils.getExtension(path.toString()).toLowerCase(); } OK, just another API, you might say. But it becomes really interesting once we need to go deeper, recursively traversing subdirectories. One amazing feature of streams is that you can combine them with each other in various ways. Old Scala saying "flatMap that shit" is applicable here as well, check out this recursive Java 8 code: //WARNING: doesn't compile, yet: private static Stream filesInDir(Path dir) { return Files.list(dir) .flatMap(path -> path.toFile().isDirectory() ? filesInDir(path) : singletonList(path).stream()); } Stream lazily produced by filesInDir() contains all files within directory including subdirectories. You can use it as any other stream by calling map(), filter(), anyMatch(), findFirst(), etc. But how does it really work?flatMap() is similar to map() but while map() is a straightforward 1:1 transformation, flatMap() allows replacing single entry in input Stream with multiple entries. If we had used map(), we would have end up with Stream>(or maybe Stream>). But flatMap() flattens this structure, in a way exploding inner entries. Let's see a simple example. Imagine Files.list() returned two files and one directory. For files flatMap() receives a one-element stream with that file. We can't simply return that file, we have to wrap it, but essentially this is no-operation. It gets way more interesting for a directory. In that case we call filesInDir() recursively. As a result we get a stream of contents of that directory, which we inject into our outer stream. Code above is short, sweet and... doesn't compile. These pesky checked exceptions again. Here is a fixed code, wrapping checked exceptions for sanity: public static Stream filesInDir(Path dir) { return listFiles(dir) .flatMap(path -> path.toFile().isDirectory() ? filesInDir(path) : singletonList(path).stream()); } private static Stream listFiles(Path dir) { try { return Files.list(dir); } catch (IOException e) { throw Throwables.propagate(e); } } Unfortunately this quite elegant code is not lazy enough. flatMap() evaluates eagerly, thus it always traverses all subdirectories, even if we barely ask for first file. You can try with my tiny LazySeq library that tries to provide even lazier abstraction, similar to streams in Scala or lazy-seq in Clojure. But even standard JDK 8 solution might be really helpful and simplify your code significantly.

Spring Security offers an authentication replacement feature, often referred to as Run-As, that can replace the current user's authentication (and thus permissions) during a single secured object invocation. Using this feature makes sense when a backend system invoked during request processing requires different privileges than the current application. For example, an application might want to expose a financial transaction log to the currently logged in user, but the backend system that provides it only permits this action to the members of a special "auditor" role. The application can not simply assign this role to the user as that would potentially permit them to execute other restricted actions. Instead, the user can be given this right exclusively for viewing their transaction log. Only two classes are used to implement this feature. Instances of RunAsManager are tasked with producing the actual replacement authentication tokens. A sensible default implementation is already provided by Spring Security. As with other types of authentication, it is also necessary to register an instance of an appropriate AuthenticationProvider. Tokens produced by runAsManager are signed with the provided key (my_run_as_key in the example above) and are later checked against the same key by runAsAuthenticationProvider, in order to mitigate the risk of fake tokens being provided. These keys can have any value, but need to be the same in both objects. Otherwise, runAsAuthenticationProvider will reject the produced tokens as invalid. If an instance is registered, RunAsManager will be invoked by AbstractSecurityInterceptor for every intercepted object invocation for which the user has already been given access. If RunAsManager returns a token, this token will be used be used instead of the original one for the duration of the invocation, thus granting the user different privileges. There are two key points here. In order for the authentication replacement feature to do anything, the call has to actually be secured (and thus intercepted), and the user has to already have been granted access. To register a RunAsManager instance with the method security interceptor, something similar to the following is needed: Now, all methods secured by the @Secured annotation will be able to trigger RunAsManager. One important point here is that global-method-security will only work in the Spring context in which it is defined. In Spring MVC applications, there usually are two Spring contexts: the parent context, attached to ContextLoaderListener, and the child context, attached toDispatcherServlet. To secure Controller methods in this way, global-method-security must be added to DispatcherServlet's context. To secure methods in beans not in this context, global-method-security should also be added to ContextLoaderListener's context. Otherwise, security annotations will be ignored. The default implementation of RunAsManager (RunAsManagerImpl) will inspect the secured object's configuration and if it finds any attributes prefixed with RUN_AS_, it will create a token identical to the original, with the addition of one new GrantedAuthorty per RUN_AS_ attribute found. The new GrantedAuthority will be a role (prefixed by ROLE_ by default) named like the found attribute without the RUN_AS_ prefix. So, if a user with a role ROLE_REGISTERED_USER invokes a method annotated with @Secured({"ROLE_REGISTERED_USER","RUN_AS_AUDITOR"}), e.g. @Controller public class TransactionLogController { @Secured({"ROLE_REGISTERED_USER","RUN_AS_AUDITOR"}) //Authorities needed for method access and authorities added by RunAsManager prefixed with RUN_AS_ @RequestMapping(value = "/transactions", method = RequestMethod.GET) //Spring MVC configuration. Not related to security @ResponseBody //Spring MVC configuration. Not related to security public List getTransactionLog(...) { ... //Invoke something in the backend requiring ROLE_AUDITOR { ... //User does not have ROLE_AUDITOR here } the resulting token created by RunAsManagerImpl with be granted ROLE_REGISTERED_USER and ROLE_AUDITOR. Thus, the user will also be allowed actions, normally reserved for ROLE_AUDITOR members, during the current invocation, permitting them, in this case, to access the transaction log.To enable runAsAuthenticationProvider, register it as usual: ... other authentication-providers used by the application ... This is all that is necessary to have the default implementation activated. Still, this setting will not work for methods secured by @PreAuthorize and @PostAuthorize annotations as their configuration attributes are differently evaluated (they are SpEL expressions and not a simple list or required authorities like with @Secured) and will not be recognized by RunAsManagerImpl. For this scenario to work, a custom RunAsManager implementation is required, as, at least at the time of writing, no applicable implementation is provided by Spring. A custom RunAsManager implementation for use with @PreAuthorize/@PostAuthorize A convenient implementation relying on a custom annotation is provided below: public class AnnotationDrivenRunAsManager extends RunAsManagerImpl { @Override public Authentication buildRunAs(Authentication authentication, Object object, Collection attributes) { if(!(object instanceof ReflectiveMethodInvocation) || ((ReflectiveMethodInvocation)object).getMethod().getAnnotation(RunAsRole.class) == null) { return super.buildRunAs(authentication, object, attributes); } String roleName = ((ReflectiveMethodInvocation)object).getMethod().getAnnotation(RunAsRole.class).value(); if (roleName == null || roleName.isEmpty()) { return null; } GrantedAuthority runAsAuthority = new SimpleGrantedAuthority(roleName); List newAuthorities = new ArrayList(); // Add existing authorities newAuthorities.addAll(authentication.getAuthorities()); // Add the new run-as authority newAuthorities.add(runAsAuthority); return new RunAsUserToken(getKey(), authentication.getPrincipal(), authentication.getCredentials(), newAuthorities, authentication.getClass()); } } This implementation will look for a custom @RunAsRole annotation on a protected method (e.g. @RunAsRole("ROLE_AUDITOR")) and, if found, will add the given authority (ROLE_AUDITOR in this case) to the list of granted authorities. RunAsRole itself is just a simple custom annotation: @Retention(RetentionPolicy.RUNTIME) @Target(ElementType.METHOD) public @interface RunAsRole { String value(); } This new implementation would be instantiated in the same way as before: And registered in a similar fashion: The expression-handler is always required for pre-post-annotations to work. It is a part of the standard Spring Security configuration, and not related to the topic described here. Both pre-post-annotations and secured-annotations can be enabled at the same time, but should never be used in the same class. The protected controller method from above could now look like this: @Controller public class TransactionLogController { @PreAuthorize("hasRole('ROLE_REGISTERED_USER')") //Authority needed to access the method @RunAsRole("ROLE_AUDITOR") //Authority added by RunAsManager @RequestMapping(value = "/transactions", method = RequestMethod.GET) //Spring MVC configuration. Not related to security @ResponseBody //Spring MVC configuration. Not related to security public List getTransactionLog(...) { ... //Invoke something in the backend requiring ROLE_AUDITOR { ... //User does not have ROLE_AUDITOR here }

There are many advantages to using CSS pre-processors like SASS, some of the features allow you to end up writing less CSS code by using inheritance and functions in SASS to reuse the same code on your different CSS classes and IDs. To learn more about getting started with SASS you can refer to a previous articles. Getting started with SASS One of my favourite features of SASS is the ability to use loops to dynamically create your CSS classes. A good example of this is when you want to make a set of classes to use for changing the text colours and background colours of elements you would normally have to write CSS like this. .red-background { background: #FF0000; } .red-color { color: #FF0000; } .blue-background { background: #001EFF; } .blue-color { color: #001EFF; } .green-background { background: #00FF00; } .green-color { color: #00FF00; } .yellow-background { background: #F6FF00; } .yellow-color { color: #F6FF00; } If you want to add additional colours to this later you will have to remember to write both background and colour classes. With SASS we can create a list of our colours and then loop through these to create the CSS classes. To create a list in SASS all you have to do is create a comma separated list of key value pairs like the following. $colours: "red" #FF0000, "blue" #001EFF, "green" #00FF00, "yellow" #F6FF00; Using the @each keyword in SASS we can loop through each of the colours and then use the nth() function to get the name of the class and the value of the class to dynamically create the classes in our CSS. The following each loop will generate exactly the same colour classes as above with only a few lines of code. @each $i in $colours{ .#{nth($i, 1)}-background { background: nth($i, 2); } .#{nth($i, 1)}-color { color:nth($i, 2); } }

SpringBoot...there is a lot of buzz about SpringBoot nowadays. So what is SpringBoot? SpringBoot is a new spring portfolio project which takes opinionated view of building production-ready Spring applications by drastically reducing the amount of configuration required. Spring Boot is taking the convention over configuration style to the next level by registering the default configurations automatically based on the classpath libraries available at runtime. Well.. you might have already read this kind of introduction to SpringBoot on many blogs. So let me elaborate on what SpringBoot is and how it helps developing Spring applications more quickly. Spring framework was created by Rod Johnson when many of the Java developers are struggling with EJB 1.x/2.x for building enterprise applications. Spring framework makes developing the business components easy by using Dependency Injection and Aspect Oriented Programming concepts. Spring became very popular and many more Spring modules like SpringSecurity, Spring Batch, Spring Data etc become part of Spring portfolio. As more and more features added to Spring, configuring all the spring modules and their dependencies become a tedious task. Adding to that Spring provides atleast 3 ways of doing anything :-). Some people see it as flexibility and some others see it as confusing. Slowly, configuring all the Spring modules to work together became a big challenge. Spring team came up with many approaches to reduce the amount of configuration needed by introducing Spring XML DSLs, Annotations and JavaConfig. In the very beginning I remember configuring a big pile of jar version declarations in section and lot of declarations. Then I learned creating maven archetypes with basic structure and minimum required configurations. This reduced lot of repetitive work, but not eliminated completely. Whether you write the configuration by hand or generate by some automated ways, if there is code that you can see then you have to maintain it. So whether you use XML or Annotations or JavaConfig, you still need to configure(copy-paste) the same infrastructure setup one more time. On the other hand, J2EE (which is dead long time ago) emerged as JavaEE and since JavaEE6 it became easy (compared to J2EE and JavaEE5) to develop enterprise applications using JavaEE platform. Also JavaEE7 released with all the cool CDI, WebSockets, Batch, JSON support etc things became even more simple and powerful as well. With JavaEE you don't need so much XML configuration and your war file size will be in KBs (really??? for non-helloworld/non-stageshow apps also :-)). Naturally this "convention over configuration" and "you no need to glue APIs together appServer already did it" arguments became the main selling points for JavaEE over Spring. Then Spring team addresses this problem with SpringBoot :-). Now its time to JavaEE to show whats the SpringBoot's counterpart in JavaEE land :-) JBoss Forge?? I love this Spring vs JavaEE thing which leads to the birth of powerful tools which ultimately simplify the developers life :-). Many times we need similar kind of infrastructure setup using same libraries. For example, take a web application where you map DispatcherServlet url-pattern to "/", implement RESTFul webservices using Jackson JSON library with Spring Data JPA backend. Similarly there could be batch or spring integration applications which needs similar infrastructure configuration. SpringBoot to the rescue. SpringBoot look at the jar files available to the runtime classpath and register the beans for you with sensible defaults which can be overridden with explicit settings. Also SpringBoot configure those beans only when the jars files available and you haven't define any such type of bean. Altogether SpringBoot provides common infrastructure without requiring any explicit configuration but lets the developer overrides if needed. To make things more simpler, SpringBoot team provides many starter projects which are pre-configured with commonly used dependencies. For example Spring Data JPA starter project comes with JPA 2.x with Hibernate implementation along with Spring Data JPA infrastructure setup. Spring Web starter comes with Spring WebMVC, Embedded Tomcat, Jackson JSON, Logback setup. Aaah..enough theory..lets jump onto coding. I am using latest STS-3.5.1 IDE which provides many more starter project options like Facebbok, Twitter, Solr etc than its earlier version. Create a SpringBoot starter project by going to File -> New -> Spring Starter Project -> select Web and Actuator and provide the other required details and Finish. This will create a Spring Starter Web project with the following pom.xml and Application.java 4.0.0 com.sivalabs hello-springboot 1.0-SNAPSHOT jar hello-springboot Spring Boot Hello World org.springframework.boot spring-boot-starter-parent 1.1.3.RELEASE org.springframework.boot spring-boot-starter-actuator org.springframework.boot spring-boot-starter-web org.springframework.boot spring-boot-starter-test test UTF-8 com.sivalabs.springboot.Application 1.7 org.springframework.boot spring-boot-maven-plugin package com.sivalabs.springboot; import org.springframework.boot.SpringApplication; import org.springframework.boot.autoconfigure.EnableAutoConfiguration; import org.springframework.context.annotation.ComponentScan; import org.springframework.context.annotation.Configuration; @Configuration @ComponentScan @EnableAutoConfiguration public class Application { public static void main(String[] args) { SpringApplication.run(Application.class, args); } } Go ahead and run this class as a standalone Java class. It will start the embedded Tomcat server on 8080 port. But we haven't added any endpoints to access, lets go ahead and add a simple REST endpoint. @Configuration @ComponentScan @EnableAutoConfiguration @Controller public class Application { public static void main(String[] args) { SpringApplication.run(Application.class, args); } @RequestMapping(value="/") @ResponseBody public String bootup() { return "SpringBoot is up and running"; } } Now point your browser to http://localhost:8080/ and you should see the response "SpringBoot is up and running". Remember while creating project we have added Actuator starter module also. With Actuator you can obtain many interesting facts about your application. Try accessing the following URLs and you can see lot of runtime environment configurations that are provided by SpringBoot. http://localhost:8080/beans http://localhost:8080/metrics http://localhost:8080/trace http://localhost:8080/env http://localhost:8080/mappings http://localhost:8080/autoconfig http://localhost:8080/dump SpringBoot actuator deserves a dedicated blog post to cover its vast number of features, I will cover it in my upcoming posts. I hope this article provides some basic introduction to SpringBoot and how it simplifies the Spring application development. More on SpringBoot in upcoming articles. - See more at: http://www.sivalabs.in/2014/07/springboot-introducing-springboot.html#sthash.7syCIt8V.dpuf

In an earlier blog entry I had touched on a fictitious rube goldberg flow for capitalizing a string through a complicated series of steps, the premise of the article was to introduce Spring Integration Java DSL as an alternative to defining integration flows through xml configuration files. I learned a few new things after writing that blog entry, thanks to Artem Bilan and wanted to document those learnings here: So, first my original sample, here I have the following flow(the one's in bold): Take in a message of this type - "hello from spring integ" Split it up into individual words(hello, from, spring, integ) Send each word to a ActiveMQ queue Pick up the word fragments from the queue and capitalize each word Place the response back into a response queue Pick up the message, re-sequence based on the original sequence of the words Aggregate back into a sentence("HELLO FROM SPRING INTEG") and Return the sentence back to the calling application. EchoFlowOutbound.java: @Bean public DirectChannel sequenceChannel() { return new DirectChannel(); } @Bean public DirectChannel requestChannel() { return new DirectChannel(); } @Bean public IntegrationFlow toOutboundQueueFlow() { return IntegrationFlows.from(requestChannel()) .split(s -> s.applySequence(true).get().getT2().setDelimiters("\\s")) .handle(jmsOutboundGateway()) .get(); } @Bean public IntegrationFlow flowOnReturnOfMessage() { return IntegrationFlows.from(sequenceChannel()) .resequence() .aggregate(aggregate -> aggregate.outputProcessor(g -> Joiner.on(" ").join(g.getMessages() .stream() .map(m -> (String) m.getPayload()).collect(toList()))) , null) .get(); } @Bean public JmsOutboundGateway jmsOutboundGateway() { JmsOutboundGateway jmsOutboundGateway = new JmsOutboundGateway(); jmsOutboundGateway.setConnectionFactory(this.connectionFactory); jmsOutboundGateway.setRequestDestinationName("amq.outbound"); jmsOutboundGateway.setReplyChannel(sequenceChannel()); return jmsOutboundGateway; } It turns out, based on Artem Bilan's feedback, that a few things can be optimized here. First notice how I have explicitly defined two direct channels, "requestChannel" for starting the flow that takes in the string message and the "sequenceChannel" to handle the message once it returns back from the jms message queue, these can actually be totally removed and the flow made a little more concise this way: @Bean public IntegrationFlow toOutboundQueueFlow() { return IntegrationFlows.from("requestChannel") .split(s -> s.applySequence(true).get().getT2().setDelimiters("\\s")) .handle(jmsOutboundGateway()) .resequence() .aggregate(aggregate -> aggregate.outputProcessor(g -> Joiner.on(" ").join(g.getMessages() .stream() .map(m -> (String) m.getPayload()).collect(toList()))) , null) .get(); } @Bean public JmsOutboundGateway jmsOutboundGateway() { JmsOutboundGateway jmsOutboundGateway = new JmsOutboundGateway(); jmsOutboundGateway.setConnectionFactory(this.connectionFactory); jmsOutboundGateway.setRequestDestinationName("amq.outbound"); return jmsOutboundGateway; } "requestChannel" is now being implicitly created just by declaring a name for it. The sequence channel is more interesting, quoting Artem Bilan - do not specify outputChannel for AbstractReplyProducingMessageHandler and rely on DSL , what it means is that here jmsOutboundGateway is a AbstractReplyProducingMessageHandler and its reply channel is implicitly derived by the DSL. Further, two methods which were earlier handling the flows for sending out the message to the queue and then continuing once the message is back, is collapsed into one. And IMHO it does read a little better because of this change. The second good change and the topic of this article is the introduction of the Jms namespace factories, when I had written the previous blog article, DSL had support for defining the AMQ inbound/outbound adapter/gateway, now there is support for Jms based inbound/adapter adapter/gateways also, this simplifies the flow even further, the flow now looks like this: @Bean public IntegrationFlow toOutboundQueueFlow() { return IntegrationFlows.from("requestChannel") .split(s -> s.applySequence(true).get().getT2().setDelimiters("\\s")) .handle(Jms.outboundGateway(connectionFactory) .requestDestination("amq.outbound")) .resequence() .aggregate(aggregate -> aggregate.outputProcessor(g -> Joiner.on(" ").join(g.getMessages() .stream() .map(m -> (String) m.getPayload()).collect(toList()))) , null) .get(); } The inbound Jms part of the flow also simplifies to the following: @Bean public IntegrationFlow inboundFlow() { return IntegrationFlows.from(Jms.inboundGateway(connectionFactory) .destination("amq.outbound")) .transform((String s) -> s.toUpperCase()) .get(); } Thus, to conclude, Spring Integration Java DSL is an exciting new way to concisely configure Spring Integration flows. It is already very impressive in how it simplifies the readability of flows, the introduction of the Jms namespace factories takes it even further for JMS based flows.

Closely approaching the one year mark of when I first joined MongoLab (and the MongoDB community), I had the pleasure of attending the inaugural MongoDB World conference put together by the incredible MongoDB team. Second only to the excitement around major MongoDB feature announcements was the collective disbelief that this was MongoDB’s first multi-day conference ever. A big congratulations to all those that worked hard to put on such a massive (did you see the Intrepid!?) event. All this planning would have been for naught if MongoDB leaders and engineers failed to deliver announcements and features that would meet and exceed expectations. From major public cloud announcements to the reveal of document-level locking in version 2.8, developers and conference goers had plenty to be excited about. There was a lot to digest from the conference… we’ll cover the major highlights in case you missed them. Big announcements in public cloud Our time at the MongoLab booth yielded many high-quality conversations, predominantly those about offloading previously internal processes and workloads to the public cloud. It was remarkable to see and hear so many enterprise teams with the exact same message: the public cloud is the future, and the future is now. It’s no surprise then that MongoDB, Inc. released not one, but two press releases around MongoDB solutions for the public cloud. Fully-managed MongoDB on the Microsoft Azure Store Nearly one year ago, MongoDB, Inc. chose to partner with the MongoLab team to build a production-ready MongoDB solution for developers on Microsoft Azure. On the first day of World, MongoDB, Inc. announced the product of our collaboration – a fully-managed highly available MongoDB-as-a-Service Add-On offering on the Microsoft Azure Store. This new service runs MongoDB Enterprise and offers replication, monitoring and support from MongoDB, Inc. It’s also backed by MongoDB Management Service (MMS), allowing for point-in-time recovery of MongoDB deployments. Now, teams without the expertise or resources to manage their MongoDB deployment(s) can outsource all the database operations (monitoring and alerting, backups, performance tuning, etc.) to both MongoLab and MongoDB’s expert support teams. You can check out the MongoDB add-on in the Azure Store: https://azure.microsoft.com/en-us/gallery/store/mongodb/mongodb-inc/ MongoDB solutions on Google Cloud Platform MongoDB, Inc. also announced the arrival of new resources to help Google Cloud Platform customers deploy MongoDB on Google Compute Engine. These resources include a “Click to Deploy” feature and a MongoDB on Google Compute Engine Solutions paper covering MongoDB best practices. If you are looking for a fully-managed solution, with automated provisioning, backups, integrated monitoring and alerting, along with expert support, MongoLab recently announced the arrival of production-ready replica sets on Google. Product Roadmap – MongoDB version 2.8 On the second day of MongoDB World, Eliot Horowitz, MongoDB, Inc. CTO & Co-founder, took center stage and announced two huge changes to the MongoDB core project: document-level locking and pluggable storage engines. These features not only reflect improvements to the core project, but also signal to the community that the MongoDB team is listening to its users and is capable of delivering the software needed to power the workloads of tomorrow. Document-level locking The slides above from Eliot’s keynote point to a current obstacle (database-level locking) in MongoDB that limits overall scalability. With database-level locking, any write operation to the database holds the write lock and prevents subsequent writes from executing on the database until the original operation holding the write lock completes. Eliot’s announcement of document-level locking moves the write lock contention from the database level to the document (MongoDB equivalent to SQL “records”) level. This change will allow users to achieve much higher write throughput (we saw a 10x performance improvement in the live demo) across their MongoDB deployments, improving write scalability. If you’d like to try out document-level locking, the MongoDB team has already pushed the feature to the master branch on GitHub. This should only be used for experimentation, not to be run in production. Pluggable storage engine As MongoDB matures, feature releases like document level locking will continue to allow developers to build robust systems on top of MongoDB. But as the number of use cases grows, different tooling tailored to specific use cases may prove to be extremely beneficial. For example, if Company X decides that they want to use MongoDB to warehouse some of their data, they would likely want to optimize their database for slow-moving data and storage efficiency (compression). With the introduction of pluggable storage engines, many new possibilities are open to the community. Teams can now write their own storage engine for a particular use case, configure replica set nodes with different storage engines for specific situations, or collaborate with the open-source community to architect innovative solutions. This feature not only allows for more granular control of the database, but also encourages the MongoDB community to work together. Takeaways: A maturing and thriving ecosystem Roughly a year ago, MongoLab CTO Todd Dampier recapped MongoSF 2013 and spoke to the health of the MongoDB ecosystem. How far we’ve come! After attending the inaugural MongoDB World and chatting with MongoDB Masters, interns, hackathon winners, power users and those new to the community, the enthusiasm is still surging and as positive as ever. This enthusiasm is well placed. Developers and hackers use MongoDB because so much rich data on the web is shared as JSON (think Facebook, Twitter, Google, etc.). As a result, MongoDB is the de-facto database for hackathons and bootstrapped projects. Just learn the API for the site you want to mine, throw the JSON in MongoDB and query your data with the rich query language- it’s that easy. The MongoDB ecosystem is maturing as well. Take a look at the Customer Success Stories and you’ll get a feel for the extent in which enterprises leverage the solution and use it in production. To further drive enterprise adoption, MongoDB, Inc.’s public cloud solutions and product roadmap features aim to help teams run MongoDB in production and give teams the confidence that MongoDB will continue to improve scalability and meet their growing project requirements. Congratulations again to the MongoDB team on their big announcements and for creating such a fantastic forum at which to learn and meet fellow MongoDB users. Our team at MongoLab had a great time making new friends and talking shop; we look forward to meeting more MongoDB users soon (at a MongoDB Days near you)! -Chris@MongoLab

some diagrams for the spring petclinic application following on from my previous post ( software architecture as code ) where i demonstrated how to create a software architecture model as code, i decided to throw together a quick implementation of a spring component finder that could be used to (mostly) automatically create a model of a spring mvc web application. spring has a bunch of annotations (e.g. @controller, @component, @service and @repository) and these are often/can be used to signify the major building blocks of a web application. to illustrate this, i took the spring petclinic application and produced some diagrams for it. first is a context diagram. next up are the containers, which in this case are just a web server (e.g. apache tomcat) and a database (hsqldb by default). and finally we have a diagram showing the components that make up the web application. these, and their dependencies, were found by scanning the compiled version of the application (i cloned the project from github and ran the maven build). here is the code that i used to generate the model behind the diagrams. package com.structurizr.example; import com.fasterxml.jackson.databind.objectmapper; import com.fasterxml.jackson.databind.serializationfeature; import com.structurizr.componentfinder.componentfinder; import com.structurizr.componentfinder.springcomponentfinderstrategy; import com.structurizr.model.*; import com.structurizr.view.componentview; import com.structurizr.view.containerview; import com.structurizr.view.contextview; /** * this is a c4 representation of the spring petclinic sample app (https://github.com/spring-projects/spring-petclinic/). */ public class springpetclinic { public static void main(string[] args) throws exception { model model = new model(); // create the basic model (the stuff we can't get from the code) softwaresystem springpetclinic = model.addsoftwaresystem(location.internal, "spring petclinic", ""); person user = model.addperson(location.external, "user", ""); user.uses(springpetclinic, "uses"); container webapplication = springpetclinic.addcontainer("web application", "", "apache tomcat 7.x"); container relationaldatabase = springpetclinic.addcontainer("relational database", "", "hsqldb"); user.uses(webapplication, "uses"); webapplication.uses(relationaldatabase, "reads from and writes to"); // and now automatically find all spring @controller, @component, @service and @repository components componentfinder componentfinder = new componentfinder(webapplication, "org.springframework.samples.petclinic", new springcomponentfinderstrategy()); componentfinder.findcomponents(); // finally create some views contextview contextview = model.createcontextview(springpetclinic); contextview.addallsoftwaresystems(); contextview.addallpeople(); containerview containerview = model.createcontainerview(springpetclinic); containerview.addallpeople(); containerview.addallsoftwaresystems(); containerview.addallcontainers(); componentview componentview = model.createcomponentview(springpetclinic, webapplication); componentview.addallcomponents(); objectmapper objectmapper = new objectmapper(); objectmapper.enable(serializationfeature.indent_output); string modelasjson = objectmapper.writevalueasstring(model); system.out.println(modelasjson); } } the resulting json representing the model was then copy-pasted across into my simple (and very much in progress) diagramming tool . admittedly the diagrams are lacking on some details (i.e. component responsibilities and arrow annotations, although those can be fixed), but this approach proves you can expend very little effort to get something that is relatively useful. as i've said before, it's all about getting the abstractions right.

Earlier this month I wrote about vmstat iowait cpu numbers, and some of the comments I got were advertising the use of util% as reported by the iostat tool instead. I find this number even more useless for MySQL performance tuning and capacity planning. Now let me start by saying this is a really tricky and deceptive number. Many DBAs who report instances of their systems having a very busy IO subsystem said the util% in vmstat was above 99% and therefore they believe this number is a good indicator of an overloaded IO subsystem. Indeed – when your IO subsystem is busy, up to its full capacity, the utilization should be very close to 100%. However, it is perfectly possible for the IO subsystem and MySQL with it to have plenty more capacity than when utilization is showing 100% – as I will show in an example. Before that though lets see what the iostat manual page has to say on this topic – from this main page we can read: %util Percentage of CPU time during which I/O requests were issued to the device (bandwidth utilization for the device). Device saturation occurs when this value is close to 100% for devices serving requests serially. But for devices serving requests in parallel, such as RAID arrays and modern SSDs, this number does not reflect their performance limits. Which says right here that the number is useless for pretty much any production database server that is likely to be running RAID, Flash/SSD, SAN or cloud storage (such as EBS) capable of handling multiple requests in parallel. Let’s look at the following illustration. I will run sysbench on a system with a rather slow storage data size larger than buffer pool and uniform distribution to put pressure on the IO subsystem. I will use a read-only benchmark here as it keeps things more simple… sysbench –num-threads=1 –max-requests=0 –max-time=6000000 –report-interval=10 –test=oltp –oltp-read-only=on –db-driver=mysql –oltp-table-size=100000000 –oltp-dist-type=uniform –init-rng=on –mysql-user=root –mysql-password= run I’m seeing some 9 transactions per second, while disk utilization from iostat is at nearly 96%: [ 80s] threads: 1, tps: 9.30, reads/s: 130.20, writes/s: 0.00 response time: 171.82ms (95%) [ 90s] threads: 1, tps: 9.20, reads/s: 128.80, writes/s: 0.00 response time: 157.72ms (95%) [ 100s] threads: 1, tps: 9.00, reads/s: 126.00, writes/s: 0.00 response time: 215.38ms (95%) [ 110s] threads: 1, tps: 9.30, reads/s: 130.20, writes/s: 0.00 response time: 141.39ms (95%) Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util dm-0 0.00 0.00 127.90 0.70 4070.40 28.00 31.87 1.01 7.83 7.52 96.68 This makes a lot of sense – with read single thread read workload the drive should be only used getting data needed by the query, which will not be 100% as there is some extra time needed to process the query on the MySQL side as well as passing the result set back to sysbench. So 96% utilization; 9 transactions per second, this is a close to full-system capacity with less than 5% of device time to spare, right? Let’s run a benchmark with more concurrency – 4 threads at the time; we’ll see… [ 110s] threads: 4, tps: 21.10, reads/s: 295.40, writes/s: 0.00 response time: 312.09ms (95%) [ 120s] threads: 4, tps: 22.00, reads/s: 308.00, writes/s: 0.00 response time: 297.05ms (95%) [ 130s] threads: 4, tps: 22.40, reads/s: 313.60, writes/s: 0.00 response time: 335.34ms (95%) Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util dm-0 0.00 0.00 295.40 0.90 9372.80 35.20 31.75 4.06 13.69 3.38 100.01 So we’re seeing 100% utilization now, but what is interesting – we’re able to reclaim much more than less than 5% which was left if we look at utilization – throughput of the system increased about 2.5x Finally let’s do the test with 64 threads – this is more concurrency than exists at storage level which is conventional hard drives in RAID on this system… [ 70s] threads: 64, tps: 42.90, reads/s: 600.60, writes/s: 0.00 response time: 2516.43ms (95%) [ 80s] threads: 64, tps: 42.40, reads/s: 593.60, writes/s: 0.00 response time: 2613.15ms (95%) [ 90s] threads: 64, tps: 44.80, reads/s: 627.20, writes/s: 0.00 response time: 2404.49ms (95%) Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util dm-0 0.00 0.00 601.20 0.80 19065.60 33.60 31.73 65.98 108.72 1.66 100.00 In this case we’re getting 4.5x of throughput compared to single thread and 100% utilization. We’re also getting almost double throughput of the run with 4 thread where 100% utilization was reported. This makes sense – there are 4 drives which can work in parallel and with many outstanding requests they are able to optimize their seeks better hence giving a bit more than 4x. So what have we so ? The system which was 96% capacity but which could have driven still to provide 4.5x throughput – so it had plenty of extra IO capacity. More powerful storage might have significantly more ability to run requests in parallel so it is quite possible to see 10x or more room after utilization% starts to be reported close to 100% So if utilization% is not very helpful what can we use to understand our database IO capacity better ? First lets look at the performance reported from those sysbench runs. If we look at 95% response time you can see 1 thread and 4 threads had relatively close 95% time growing just from 150ms to 250-300ms. This is the number I really like to look at- if system is able to respond to the queries with response time not significantly higher than it has with concurrency of 1 it is not overloaded. I like using 3x as multiplier – ie when 95% spikes to be more than 3x of the single concurrency the system might be getting to the overload. With 64 threads the 95% response time is 15-20x of the one we see with single thread so it is surely overloaded. Do we have anything reported by iostat which we can use in a similar way? It turns out we do! Check out the “await” column which tells us how much the requester had to wait for the IO request to be serviced. With single concurrency it is 7.8ms which is what this drives can do for random IO and is as good as it gets. With 4 threads it is 13.7ms – less than double of best possible, so also good enough… with concurrency of 64 it is however 108ms which is over 10x of what this device could produce with no waiting and which is surely sign of overload. A couple words of caution. First, do not look at svctm which is not designed with parallel processing in mind. You can see in our case it actually gets better with high concurrency while really database had to wait a lot longer for requests submitted. Second, iostat mixes together reads and writes in single statistics which specifically for databases and especially on RAID with BBU might be like mixing apples and oranges together – writes should go to writeback cache and be acknowledged essentially instantly while reads only complete when actual data can be delivered. The tool pt-diskstats from Percona Tookit breaks them apart and so can be much more for storage tuning for database workload. Some of the recent operating systems also ship with sysstat/iostat which breaks out await to r_await and w_await which is much more useful. Final note – I used a read-only workload on purpose – when it comes to writes things can be even more complicated – MySQL buffer pool can be more efficient with more intensive writes plus group commit might be able to commit a lot of transactions with single disk write. Still, the same base logic will apply. Summary: The take away is simple – util% only shows if a device has at least one operation to deal with or is completely busy, which does not reflect actual utilization for a majority of modern IO subsystems. So you may have a lot of storage IO capacity left even when utilization is close to 100%.

So you already know what Hadoop is? Why it is used? What problems you can solve with it?

Every SQL developer uses JOINs and aggregate functions, but some may not be aware that the two interacting can return incorrect results if not handled carefully. Brett Sauve at Looker has covered the issue in this recent post - specifically, he focuses on the issue of "fanouts" - and it's an easy problem to overlook. A fanout occurs when the primary table in a query contains fewer rows than the combined table resulting from a JOIN. Because the new table contains more rows, aggregate functions like COUNT or SUM may include duplicates, throwing off the results. According to Sauve, the answer is careful ordering of JOIN tables: Herein lies a key point: to help avoid fanouts, begin your joins with the most granular table. Sauve also provides some tips to help SQL developers understand how to avoid fanouts and ensure the validity of their data. He points to three types of JOINs: 1-to-1 Many-to-1 1-to-Many Knowing which JOIN you are using can rule out the possibility of a fanout, for example, because it is not possible in a 1-to-1 or Many-to-1 JOIN. If you are using a 1-to-Many JOIN, though, Sauve has some tips there, too. For example, a messy JOIN can be cleaned up by grouping data to create a 1-to-1 relationship. Check out the full article for the rest of Sauve's tips to make sure your aggregate functions are working correctly, and if you're looking for a few more tips to keep your SQL skills sharp, you might try one of these: Yet Another 10 Common Mistakes Java Developers Make When Writing SQL 6 Simple Performance Tips for SQL SELECT Statements

The expected behaviour for volatile fields is that they should behave in a multi-threaded application the same as they do in a single threaded application. They are not forbidden to behave the same way, but they are not guaranteed to behave the same way. The solution in Java 5.0+ is to use AtomicXxxx classes however these are relatively inefficient in terms of memory (they add a header and padding), performance (they add a references and little control over their relative positions), and syntactically they are not as clear to use. IMHO A simple solution if for volatile fields to act as they might be expected to do, the way JVM must support in AtomicFields which is not forbidden in the current JMM (Java- Memory Model) but not guaranteed. Why make fields volatile? The benefit of volatile fields is that they are visible across threads and some optimisations which avoid re-reading them are disabled so you always check again the current value even if you didn't change them. e.g. without volatile Thread 2: int a = 5; Thread 1: a = 6; (later) Thread 2: System.out.println(a); // prints 5 With volatile Thread 2: volatile int a = 5; Thread 1: a = 6; (later) Thread 2: System.out.println(a); // prints 6 Why not use volatile all the time? Volatile read and write access is substantially slower. When you write to a volatile field it stalls the entire CPU pipeline to ensure the data has been written to cache. Without this, there is a risk the next read of the value sees an old value, even in the same thread (See AtomicLong.lazySet() which avoids stalling the pipeline) The penalty can be in the order of 10x slower which you don't want to be doing on every access. What are the limitations of volatile? A significant limitation is that operations on the field is not atomic, even when you might think it is. Even worse than that is that usually, there is no difference. I.e. it can appear to work for a long time even years and suddenly/randomly break due to an incidental change such as the version of Java used, or even where the object is loaded into memory. e.g. which programs you loaded before running the program. e.g. updating a value Thread 2: volatile int a = 5; Thread 1: a += 1; Thread 2: a += 2; (later) Thread 2: System.out.println(a); // prints 6, 7 or 8 This is an issue because the read of a and the write of a are done separately and you can get a race condition. 99%+ of the time it will behave as expect, but sometimes it won't/ What can you do about it? You need to use AtomicXxxx classes. These wrap volatile fields with operations which behave as expected. Thread 2: AtomicInteger a = new AtomicInteger(5); Thread 1: a.incrementAndGet(); Thread 2: a.addAndGet(2); (later) Thread 2: System.out.println(a); // prints 8 What do I propose? The JVM has a means to behave as expected, the only surprising thing is you need to use a special class to do what the JMM won't guarantee for you. What I propose is that the JMM be changed to support the behaviour currently provided by the concurrency AtomicClasses. In each case the single threaded behaviour is unchanged. A multi-threaded program which does not see a race condition will behave the same. The difference is that a multi-threaded program does not have to see a race condition but changing the underlying behaviour current method suggested syntax notes x.getAndIncrement() x++ or x += 1 x.incrementAndGet() ++x x.getAndDecrment() x-- or x -= 1 x.decrementAndGet() --x x.addAndGet(y) (x += y) x.getAndAdd(y) ((x += y)-y) x.compareAndSet(e, y) (x == e ? x = y, true : false) Need to add the comma syntax used in other languages. These operations could be supported for all the primitive types such as boolean, byte, short, int, long, float and double. Additional assignment operators could be supported such as current method suggested syntax notes Atomic multiplication x *= 2; Atomic subtraction x -= y; Atomic division x /= y; Atomic modulus x %= y; Atomic shift x <<= y; Atomic shift x >>= z; Atomic shift x >>>= w; Atomic and x &= ~y; clears bits Atomic or x |= z; sets bits Atomic xor x ^= w; flips bits What is the risk? This could break code which relies on these operations occasionally failing due to race conditions. It might not be possible to support more complex expressions in a thread safe manner. This could lead to surprising bugs as the code can look like the works, but it doesn't. Never the less it will be no worse than the current state. JEP 193 - Enhanced Volatiles There is a JEP 193 to add this functionality to Java. An example is; class Usage { volatile int count; int incrementCount() { return count.volatile.incrementAndGet(); } } IMHO there is a few limitations in this approach. The syntax is fairly significant change. Changing the JMM might not require many changes the the Java syntax and possibly no changes to the compiler. It is a less general solution. It can be useful to support operations like volume += quantity; where these are double types. It places more burden on the developer to understand why he/she should use this instead of x++; Conclusion Much of the syntactic and performance overhead of using AtomicInteger and AtomicLong could be removed if the JMM guaranteed the equivalent single threaded operations behaved as expected for multi-threaded code. This feature could be added to earlier versions of Java by using byte code instrumentation.

We had a lengthy discussion recently during code review whether scala.Option.fold() is idiomatic and clever or maybe unreadable and tricky? Let's first describe what the problem is. Option.fold does two things: maps a function f overOption's value (if any) or returns an alternative alt if it's absent. Using simple pattern matching we can implement it as follows: val option: Option[T] = //... def alt: R = //... def f(in: T): R = //... val x: R = option match { case Some(v) => f(v) case None => alt } If you prefer one-liner, fold is actually a combination of map and getOrElse val x: R = option map f getOrElse alt Or, if you are a C programmer that still wants to write in C, but using Scala compiler: val x: R = if (option.isDefined) f(option.get) else alt Interestingly this is similar to how fold() is actually implemented, but that's an implementation detail. OK, all of the above can be replaced with single Option.fold(): val x: R = option.fold(alt)(f) Technically you can even use /: and \: operators (alt /: option) - but that would be simply masochistic. I have three problems with option.fold() idiom. First of all - it's anything but readable. We are folding (reducing) over Option - which doesn't really make much sense. Secondly it reverses the ordinary positive-then-negative-case flow by starting with failure (absence, alt) condition followed by presence block (f function; see also: Refactoring map-getOrElse to fold). Interestingly this method would work great for me if it was named mapOrElse: ** * Hypothetical in Option */ def mapOrElse[B](f: A => B, alt: => B): B = this map f getOrElse alt Actually there is already such method in Scalaz, called OptionW.cata. cata. Here is whatMartin Odersky has to say about it: "I personally find methods like cata that take two closures as arguments are often overdoing it. Do you really gain in readability over map + getOrElse? Think of a newcomer to your code[...]"While cata has some theoretical background, Option.fold just sounds like a random name collision that doesn't bring anything to the table, apart from confusion. I know what you'll say, that TraversableOnce has fold and we are sort-of doing the same thing. Why it's a random collision rather than extending the contract described inTraversableOnce? fold() method in Scala collections typically just delegates to one offoldLeft()/foldRight() (the one that works better for given data structure), thus it doesn't guarantee order and folding function has to be associative. But inOption.fold() the contract is different: folding function takes just one parameter rather than two. If you read my previous article about folds you know that reducing function always takes two parameters: current element and accumulated value (initial value during first iteration). But Option.fold() takes just one parameter: current Option value! This breaks the consistency, especially when realizing Option.foldLeft() andOption.foldRight() have correct contract (but it doesn't mean they are more readable). The only way to understand folding over option is to imagine Option as a sequence with0 or 1 elements. Then it sort of makes sense, right? No. def double(x: Int) = x * 2 Some(21).fold(-1)(double) //OK: 42 None.fold(-1)(double) //OK: -1 but: Some(21).toList.fold(-1)(double) : error: type mismatch; found : Int => Int required: (Int, Int) => Int Some(21).toList.fold(-1)(double) ^ If we treat Option[T] as a List[T], awkward Option.fold() breaks because it has different type than TraversableOnce.fold(). This is my biggest concern. I can't understand why folding wasn't defined in terms of the type system (trait?) and implemented strictly. As an example take a look at: Data.Foldable in Haskell (advanced) Data.Foldable typeclass describes various flavours of folding in Haskell. There are familiar foldl/foldr/foldl1/foldr1, in Scala namedfoldLeft/foldRight/reduceLeft/reduceRight accordingly. They have the same type as Scala and behave unsurprisingly with all types that you can fold over, including Maybe, lists, arrays, etc. There is also a function named fold, but it has a completely different meaning: class Foldable t where fold :: Monoid m => t m -> m While other folds are quite complex, this one barely takes a foldable container of ms (which have to be Monoids) and returns the same Monoid type. A quick recap: a type can be aMonoid if there exists a neutral value of that type and an operation that takes two values and produces just one. Applying that function with one of the arguments being neutral value yields the other argument. String ([Char]) is a good example with empty string being neutral value (mempty) and string concatenation being such operation (mappend). Notice that there are two different ways you can construct monoids for numbers: under addition with neutral value being 0 (x + 0 == 0 + x == x for any x) and under multiplication with neutral 1 (x * 1 == 1 * x == x for any x). Let's stick to strings. If I fold empty list of strings, I'll get an empty string. But when a list contains many elements, they are being concatenated: > fold ([] :: [String]) "" > fold [] :: String "" > fold ["foo", "bar"] "foobar" In the first example we have to explicitly say what is the type of empty list []. Otherwise Haskell compiler can't figure out what is the type of elements in a list, thus which monoid instance to choose. In second example we declare that whatever is returned from fold [], it should be a String. From that the compiler infers that [] actually must have a type of [String]. Last fold is the simplest: the program folds over elements in list and concatenates them because concatenation is the operation defined in Monoid Stringtypeclass instance. Back to options (or more precisely Maybe). Folding over Maybe monad having type parameter being Monoid (I can't believe I just said it) has an interesting interpretation: it either returns value inside Maybe or a default Monoid value: > fold (Just "abc") "abc" > fold Nothing :: String "" Just "abc" is same as Some("abc") in Scala. You can see here that if Maybe Stringis Nothing, neutral String monoid value is returned, that is an empty string. Summary Haskell shows that folding (also over Maybe) can be at least consistent. In ScalaOption.fold is unrelated to List.fold, confusing and unreadable. I advise avoiding it and staying with slightly more verbose map/getOrElse transformations or pattern matching. PS: Did I mention there is also Either.fold() (with even different contract) but noTry.fold()?

We have published the results of the Eclipse Community Survey 2014. Thank you to everyone who participated in the survey this year. The complete results and data are available for anyone to download [xls] [ods]. As in other years, I think the results provide an interesting perspective on what tools software developers are using and the type of applications they are building. Here are some key highlights from the results this year: 1) Git #1 Code Management Tool. Git has finally surpassed Subversion to be the top code management tool used by software developers. A third of developers (33.3%) report they use Git as their primary code management tool compared to 30.7% using Subversion. Subversion continues to show a downward trend from previous years when it was used by more than half the developers. Of note, 9.6% claim GitHub is their primary code management tool so the prevalence of overall Git usage is becoming dominate. 2) Maven and Jenkins Key Tools. For Build and Release tools, Maven and Jenkins continue to be key tools used by developers. Of interest is the growth of Gradle from 2013 (4.5%) to 2014 (11%). 3) Top 3 Application Servers. Tomcat (32.6%), JBoss (11.8%) and Jetty (7.2%) continue to be the top 3 application servers. 4) Java 8 Adoption. Java 8 was released in March 2014 and already 9.2% of Java developers have migrated to Java 8 as their primary version of Java. 59.2% are using Java 7 but close to a quarter are using Java 6 or before. 5) Majority of Developers Use JavaScript. More and more software developers use multiple languages to develop software. Due to the Eclipse biased of the survey, Java is not surprisingly the top primary language. However, when asked what other languages developers might use, JavaScript stands out to be a popular language with a the majority of developers (56.2%) claiming it as a secondary language. 6) Developers Experimenting With Open Hardware. The Internet of Things (IoT) and Open Hardware have become important industry trends in the last couple of years. Over a third (35.7%) of software developers are spending their own personal time learning about devices like the BeagleBone, Arduino and Raspberry Pi. Thanks again to everyone who participated in the survey. I hope everyone finds the results of interest.

while writing my spring-hibernate integration post i realize that we (as java developers) have so many frameworks available which can make life easy by rapidly developing so many common things using frameworks. this ease of development is not available in java particularly when you considering ui development, we still have to struggle with old, bad looking jsp with combination of css, jquery. thank god we now have so many javascript toolkits, framework that make developers' lives easy by offering so many ready-to-use widgets. extjs is the most advanced among those client side ui frameworks. today i am going to demonstrate to you how you can leverage extjs 4 to create crud application. in the next post i will try to use the same js code with spring mvc as a backend . create a html page which include extjs library and aur books.js file. here is the extjs crud code, i have combined it in a single file. you can follow the extjs 4 mvc folder structure. ext.onready(function () { ext.define('techzoo.model.book', { extend: 'ext.data.model', fields: [ {name: 'title', type: 'string'}, {name: 'author', type: 'string'}, {name: 'price', type: 'int'}, {name: 'qty', type: 'int'} ] }); ext.define('techzoo.store.books', { extend : 'ext.data.store', model : 'techzoo.model.book', fields : ['title', 'author','price', 'qty'], data : [ { title: 'jdbc, servlet and jsp', author: 'santosh kumar', price: 300, qty : 12000 }, { title: 'head first java', author: 'kathy sierra', price: 550, qty : 2500 }, { title: 'java scjp certification', author: 'khalid mughal', price: 650, qty : 5500 }, { title: 'spring and hinernate', author: 'santosh kumar', price: 350, qty : 2500 }, { title: 'mastering c++', author: 'k. r. venugopal', price: 400, qty : 1200 } ] }); ext.define('techzoo.view.bookslist', { extend: 'ext.grid.panel', alias: 'widget.bookslist', title: 'books list - (extjs 4 crud example - @ tousif khan)', store: 'books', initcomponent: function () { this.tbar = [{ text : 'add book', action : 'add', iconcls : 'book-add' }]; this.columns = [ { header: 'title', dataindex: 'title', flex: 1 }, { header: 'author', dataindex: 'author' }, { header: 'price', dataindex: 'price' , width: 60 }, { header: 'quantity', dataindex: 'qty', width: 80 }, { header: 'action', width: 50, renderer: function (v, m, r) { var id = ext.id(); var max = 15; ext.defer(function () { ext.widget('image', { renderto: id, name: 'delete', src : 'images/book_delete.png', listeners : { afterrender: function (me) { me.getel().on('click', function() { var grid = ext.componentquery.query('bookslist')[0]; if (grid) { var sm = grid.getselectionmodel(); var rs = sm.getselection(); if (!rs.length) { ext.msg.alert('info', 'no book selected'); return; } ext.msg.confirm('remove book', 'are you sure you want to delete?', function (button) { if (button == 'yes') { grid.store.remove(rs[0]); } }); } }); } } }); }, 50); return ext.string.format('', id); } } ]; this.callparent(arguments); } }); ext.define('techzoo.view.booksform', { extend : 'ext.window.window', alias : 'widget.booksform', title : 'add book', width : 350, layout : 'fit', resizable: false, closeaction: 'hide', modal : true, config : { recordindex : 0, action : '' }, items : [{ xtype : 'form', layout: 'anchor', bodystyle: { background: 'none', padding: '10px', border: '0' }, defaults: { xtype : 'textfield', anchor: '100%' }, items : [{ name : 'title', fieldlabel: 'book title' },{ name: 'author', fieldlabel: 'author name' },{ name: 'price', fieldlabel: 'price' },{ name: 'qty', fieldlabel: 'quantity' }] }], buttons: [{ text: 'ok', action: 'add' },{ text : 'reset', handler : function () { this.up('window').down('form').getform().reset(); } },{ text : 'cancel', handler: function () { this.up('window').close(); } }] }); ext.define('techzoo.controller.books', { extend : 'ext.app.controller', stores : ['books'], views : ['bookslist', 'booksform'], refs : [{ ref : 'formwindow', xtype : 'booksform', selector: 'booksform', autocreate: true }], init: function () { this.control({ 'bookslist > toolbar > button[action=add]': { click: this.showaddform }, 'bookslist': { itemdblclick: this.onrowdblclick }, 'booksform button[action=add]': { click: this.doaddbook } }); }, onrowdblclick: function(me, record, item, index) { var win = this.getformwindow(); win.settitle('edit book'); win.setaction('edit'); win.setrecordindex(index); win.down('form').getform().setvalues(record.getdata()); win.show(); }, showaddform: function () { var win = this.getformwindow(); win.settitle('add book'); win.setaction('add'); win.down('form').getform().reset(); win.show(); }, doaddbook: function () { var win = this.getformwindow(); var store = this.getbooksstore(); var values = win.down('form').getvalues(); var action = win.getaction(); var book = ext.create('techzoo.model.book', values); if(action == 'edit') { store.removeat(win.getrecordindex()); store.insert(win.getrecordindex(), book); } else { store.add(book); } win.close(); } }); ext.application({ name : 'techzoo', controllers: ['books'], launch: function () { ext.widget('bookslist', { width : 500, height: 300, renderto: 'output' }); } } ); }); output: open the html file in any browser, the output will look similar to below. you can click to add new book record in grid. double click to any of the grid row to edit the record. view live demo

In this post we will see how we can write Javadoc comments using Markdown instead of the typical Javadoc syntax. So what is Markdown? Markdown is a plain text formatting syntax designed so that it optionally can be converted to HTML using a tool by the same name. Markdown is popularly used to format readme files, for writing messages in online discussion forums or in text editors for the quick creation of rich text documents. (Wikipedia: Markdown) Markdown is a very easy to read formatting syntax. Different variations of Markdown can be used on Stack Overflow or GitHub to format user generated content. Setup By default the Javadoc tool uses Javadoc comments to generate API documentation in HTML form. This process can be customized used Doclets. Doclets are Java programs that specify the content and format of the output of the Javadoc tool. The markdown-doclet is a replacement for the standard Java Doclet which gives developers the option to use Markdown syntax in their Javadoc comments. We can set up this doclet in Maven using the maven-javadoc-plugin. maven-javadoc-plugin 2.9 ch.raffael.doclets.pegdown.PegdownDoclet ch.raffael.pegdown-doclet pegdown-doclet 1.1 true Writing comments in Markdown Now we can use Markdown syntax in Javadoc comments: /** * ## Large headline * ### Smaller headline * * This is a comment that contains `code` parts. * * Code blocks: * * ```java * int foo = 42; * System.out.println(foo); * ``` * * Quote blocks: * * > This is a block quote * * lists: * * - first item * - second item * - third item * * This is a text that contains an [external link][link]. * * [link]: http://external-link.com/ * * @param id the user id * @return the user object with the passed `id` or `null` if no user with this `id` is found */ public User findUser(long id) { ... } After running mvn javadoc:javadoc we can find the generated HTML API documentation in target/site/apidocs. The generated documentation for the method shown above looks like this: As we can see the Javadoc comments get nicely converted to HTML. Conclusion Markdown has the clear advantage over standard Javadoc syntax that the source it is far easier to read. Just have a look at some of the method comments of java.util.Map. Many Javadoc comments are full with formatting tags and are barely readable without any tool. But be aware that Markdown can cause problems with tools and IDEs that expect standard Javadoc syntax.

Optimizing for website performance includes setting long expiration dates on our static resources, such s images, stylesheets and JavaScript files. Doing that tells the browser to cache our files so it doesn’t have to request them every time the user loads a page. This is one of the most important things to do when optimizing websites. In ASP.NET on IIS7+ it’s really easy. Just add this chunk of XML to the web.config’s element: The above code tells the browsers to automatically cache all static resources for 365 days. That’s good and you should do this right now. The issue becomes clear the first time you make a change to any static file. How is the browser going to know that you made a change, so it can download the latest version of the file? The answer is that it can’t. It will keep serving the same cached version of the file for the next 365 days regardless of any changes you are making to the files. Fingerprinting The good news is that it is fairly trivial to make a change to our code, that changes the URL pointing to the static files and thereby tricking the browser into believing it’s a brand new resource that needs to be downloaded. Here’s a little class that I use on several websites, that adds a fingerprint, or timestamp, to the URL of the static file. using System; using System.IO; using System.Web; using System.Web.Caching; using System.Web.Hosting; public class Fingerprint { public static string Tag(string rootRelativePath) { if (HttpRuntime.Cache[rootRelativePath] == null) { string absolute = HostingEnvironment.MapPath("~" + rootRelativePath); DateTime date = File.GetLastWriteTime(absolute); int index = rootRelativePath.LastIndexOf('/'); string result = rootRelativePath.Insert(index, "/v-" + date.Ticks); HttpRuntime.Cache.Insert(rootRelativePath, result, new CacheDependency(absolute)); } return HttpRuntime.Cache[rootRelativePath] as string; } } All you need to change in order to use this class, is to modify the references to the static files. Modify references Here’s what it looks like in Razor for the stylesheet reference: …and in WebForms: content/site.css") %>" /> The result of using the FingerPrint.Tag method will in this case be: Since the URL now has a reference to a non-existing folder (v-634933238684083941), we need to make the web server pretend it exist. We do that with URL rewriting. URL rewrite By adding this snippet of XML to the web.config’s section, we instruct IIS 7+ to intercept all URLs with a folder name containing “v=[numbers]” and rewrite the URL to the original file path. You can use this technique for all your JavaScript and image files as well. The beauty is, that every time you change one of the referenced static files, the fingerprint will change as well. This creates a brand new URL every time so the browsers will download the updated files. FYI, you need to run the AppPool in Integrated Pipeline mode for the section to have any effect.

Utilize this checklist to review the quality of your Java code, including security, performance, and static code analysis.