Recently I attended a local rheinJUG meeting in Düsseldorf. While the topic of the session was Eclipse e4, the night’s sponsor itemis provided some handouts on Xtext which got me very interested. The reason is that currently at work we are developing a mobile Java application (J9, CDC/Foundation 1.1 on Windows CE6) for which we needed an easy to use and reliable way for configuring navigation through the application. In a previous iteration we had – mostly because of time constraints – hard coded most of the navigational paths, but this time the app is more complex and doing that again was not really an option. First we thought about an XML based configuration, but this seemed to be a hassle to write (and read) and also would mean we would have to pay the price of parsing it on every application startup. Enter Xtext: An Eclipse based framework/library for building text based DSLs. In short, you just provide a grammar description of a new DSL to suit your needs and with – literally – just a few mouse clicks you are provided with a content-assist, syntax-highlight, outline-view-enabled Eclipse editor and optionally a code generator based on that language. Getting started: Sample Grammar There is a nice tutorial provided as part of the Xtext documentation, but I believe it might be beneficial to provide another example of how to put a DSL to good use. I will not go into every step in great detail, because setting up Xtext is Eclipse 3.6 Helios is just a matter of putting an Update Site URL in, and the New Project wizard provided makes the initial setup a snap. I assume, you have already set up Eclipse and Xtext and created a new Xtext project including a generator project (activate the corresponding checkbox when going through the wizard). In this post I am assuming a project name of com.danielschneller.navi.dsl and a file extension of .navi. When finished we will have the infrastructure ready for editing, parsing and generating code based on files like these: navigation rules for MyApplication mappings { map permission AdminPermission to "privAdmin" map permission DataAccessPermission to "privData" map coordinate Login to "com.danielschneller.myapp.gui.login.LoginController" in "com.danielschneller.myapp.login" map coordinate LoginFailed to "com.danielschneller.myapp.gui.login.LoginFailedController" in "com.danielschneller.myapp.login" map coordinate MainMenu to "com.danielschneller.myapp.gui.menu.MainMenuController" in "com.danielschneller.myapp.menu" map coordinate UserAdministration to "com.danielschneller.myapp.gui.admin.UserAdminController" in "com.danielschneller.myapp.admin" map coordinate DataLookup to "com.danielschneller.myapp.gui.lookup.LookupController" in "com.danielschneller.myapp.lookup" } navigations { define navigation USER_LOGON_FAILED define navigation USER_LOGON_SUCCESS define navigation OK define navigation BACK define navigation ADMIN define navigation DATA_LOOKUP } navrules { from Login on navigation USER_LOGON_FAILED go to LoginFailed on navigation USER_LOGON_SUCCESS go to MainMenu from LoginFailed on navigation OK go to Login from MainMenu on navigation ADMIN go to UserAdministration with AdminPermission on navigation DATA_LOOKUP go to DataLookup with DataAccessPermission from UserAdministration on navigation BACK go to MainMenu from DataLookup on navigation BACK go to MainMenu } As you can see it is a nice little language for defining coordinates in an application, meaning a specific GUI for a certain task and the possible navigation paths between them. Optionally a navigation path can be tagged to require one or more permissions to work. So for example one possible navigation path shown in the above sample is from the applications main menu, identified by the identifier MainMenu and represented in code by the com.danielschneller.myapp.gui.menu.MainMenuController class in the com.danielschneller.myapp.menu OSGi bundle to a GUI identified as DataLookup, implemented by com.danielschneller.myapp.gui.lookup.LookupController in the com.danielschneller.myapp.lookup bundle. For this path to be taken, the application must request the DataLookup navigation path and the currently logged in user be assigned the DataAccessPermission. What exactly that means is not the focus of this tutorial, suffice it to say that we somehow need to get the information contained in this specialized language into our Java application in some shape or form that can be evaluated at runtime. In the following example all information will be transformed into a HashMap based data structure. For our little mobile application this has several advantages over the XML option mentioned earlier: No XML parsing necessary on application startup, saving some performance Validation of the navigation rules ahead of time, preventing parse errors at runtime No libraries needed to access the information – by putting everything in a simple HashMap we do not have to rely on any non-standard classes whatsoever First thing I did when I started with Xtext was define a sample input file such as the one above. Then – following its general structure – I began to extract a formal grammar for it. Of course, the first draft of the sample data was not perfect, over the course of a few iterations I refined some of the syntax, but in the end this is the grammar definition I came up with. It is heavily commented to allow you to copy it out and still leave the documentation intact: grammar com.danielschneller.navi.NavigationRules with org.eclipse.xtext.common.Terminals generate navigationRules "http://com.danielschneller/fw/funkmde/navi/NavigationRules" /* * The top level entry point for the file. * "Root" is just a name as good as any, but * makes the meaning quite clear. */ Root: // first thing in the file is a "keyword", // followed by an attribute that will be // accessible as "name" later and allow // definition of an ID type of thing. 'navigation rules for' name=ID // after the keyword and "name" attribute // three sections follow, each assigned // to an attribute for later reference // (called "mappingdefs", "transitiondefs" // and "ruledefs"). // Their types are defined later in the file. mappingsdefs=Mappings transitiondefs=TransitionDefinitions ruledefs=NavigationRules // semicolon ends the definition of "Root" ; // mappings section >>>>>>>>>>>>>>>>>>>>>>>>>>>>> /* * Definition of the "Mappings" type used in * the "Root" type. */ Mappings: // first the keyword "mappings" is expected, // then an open curly 'mappings' '{' // after that a collection of "Mapping"s is // expected. The "+=" means that they will // all be collected in a collection type element // called "mappings" for future reference. // The "+" at the end means "at least one, but // more is just fine". (mappings+=Mapping)+ // finally the "Mappings" type requires a closing // curly brace. '}' // semicolon ends the definition of "Mappings" ; /* * Definition of a single "Mapping", those we are * collecting in the "mappings" attribute of the * "Mappings" type. */ Mapping: // each mapping starts with the keyword "map" // and is followed by an element of type "MappingSpec" 'map' MappingSpec ; /* * Definition of a "MappingSpec" element. This is * actually just a "parent type" for two more specific * kinds of "MappingSpec": */ MappingSpec: // no keywords are defined here, a "MappingSpec" // can be either a "PermissionMappingSpec" or a // "CoordinateMappingSpec". Any of these will be // fine where a "MappingSpec" is asked for. PermissionMappingSpec | CoordinateMappingSpec ; /* * Definition of a "PermissionMappingSpec" element. */ PermissionMappingSpec: // first the keyword "permission" is required. // then a "name" attribute is expected of type ID. // Following the name the "to" keyword is expected, // followed by a string that is stored in the "value" // attribute 'permission' name=ID 'to' value=STRING ; /* * Definition of a "CoordinateMappingSpec" element. * The definition is very similar to the "PermissionMappingSpec" * but has more attributes. */ CoordinateMappingSpec: // first the keyword "coordinate", then an ID stored as "name", // the keyword "to", followed by a string stored as "controllername", // next the keyword "in" and finally another string, memorized as // "bundleid" 'coordinate' name=ID 'to' controllername=STRING 'in' bundleid=STRING ; // <<<<<<<<<<<<<<<<<<<<<<<<<<<<< mappings section // >>>>>>>>>>>>>>>>>>>>>>>>>>>>> navigations section /* * Definition of the "TransitionDefinitions" type used in * the "Root" type. */ TransitionDefinitions: // first, this element is introduced with the "navigations" // keyword, followed by an open curly brace. 'navigations' '{' // after that a collection of "TransitionDefinition"s is // expected. The "+=" means that they will // all be collected in a collection type element // called "transitions" for future reference. // The "+" at the end means "at least one, but // more is just fine". (transitions+=TransitionDefinition)+ // the element ends with a closing curly brace '}' ; /* * Definition of a "TransitionDefinition" element. This * one is very simple. */ TransitionDefinition: // the keyword "define navigation" is required first, // then a "name" attribute of type ID is expected. 'define navigation' name=ID ; // <<<<<<<<<<<<<<<<<<<<<<<<<<<<< navigations section // >>>>>>>>>>>>>>>>>>>>>>>>>>>>> navrules section /* * Definition of the "NavigationRules" element. */ NavigationRules: // Element starts with the keywords "navrules" and // open curly. 'navrules' '{' // collection attribute called "rules", consisting // of one or more occurrences of a "Rule" element. (rules+=Rule)+ // element finishes with a closing curly keyword '}' ; /* * Definition of a "Rule" element as used in the "NavigationRules" * element. */ Rule: // first the "from" keyword, then a reference to one of the // coordinate mappings defined earlier. This time no new // definition of a coordinate is required, but one of those // that have been listed before. So the type here is put in // square brackets 'from' source=[CoordinateMappingSpec] // following the source specification, one or more "Destination" // type elements are expected, collected in a collection attribute // named "destinations" (destinations+=Destination)+ ; /* * Definition of a "Destination" type. These are collected * in a "Rule". */ Destination: // first comes an "on navigation" keyword. After that a // reference to one of the Transition elements defined // in the "navigations" section is required and stored // in the "transition" attribute. // after that follows a "go to" keyword and a reference // to a coordinate mapping, stored in the "target" attribute. // finally - as with the "destinations" collection attribute // in the "Rule" element - a "permissions" collection is // defined to store none or more (*) "PermissionReference" // elements. 'on navigation' transition=[TransitionDefinition] 'go to' target=[CoordinateMappingSpec] (permissions+=PermissionReference)* ; /* * Definition of a "PermissionReference" type. This is used * in the "permissions" collection of a "Destination". */ PermissionReference: // first, a "with" keyword is expected. After that a // "permission" attribute stores a reference to one of // the previously defined permission mappings from the // "mappings" section. 'with' permission=[PermissionMappingSpec] ; // <<<<<<<<<<<<<<<<<<<<<<<<<<<<< navrules section This is what XText can digest and create an editor plugin and outline view for. Just save this as navigationRules.xtext – when you created the XText project in Eclipse using the wizard it should have been prepared for you. Copying and pasting this into a .xtext file in Eclipse will provide you with syntax highlighting, code completion and syntax checking, making it easy to play around with grammar files. Once done, right click the .mwe2 file lying next to the grammar file in the Package Explorer view and select Run As MWE2 Workflow from the context menu. This will take a moment and generate several classes, both in the current (XText) project and the accompanying ...ui project. Next, right click the Xtext project and select Run As Eclipse Application from the context menu. This will bring up another Eclipse instance with the newly created support for navigation rules files (with a .navi suffix) installed. To try it out, just create a new project and in that a new file. Make sure its name ends in .navi. When asked, make sure to accept adding the Xtext nature to the project. You will be presented with a new, empty editor that already has an error marker in it. This is because according to our grammar definition, an empty file does not comply to all the rules we specified. Try hitting the code-completion shortcut (Ctrl-Space) twice and see what happens: The first code-completion fills in the navigation rules for part. According to the grammar this is the only valid text at the beginning of a file, so it is automatically inserted. Hitting Ctrl-Space again will tell you that now you need a Name of type ID. Just go ahead and try out the completion. It will help you create a syntactically sound navigation rules file. Notice that the Problems View tells you what is currently wrong. Also notice, that one you reach a part where references are expected by the grammar (e. g. when defining source and destination coordinates in a navigation rule) you will get suggestions based on what you entered earlier. This is what the whole sample from above looks like in the editor: While you are still fleshing out and fine tuning your grammar definitions, you will probably close this Eclipse instance and reopen it, once you repeated the Run As MWE2 Workflow steps in the main instance. In the long run I suggest you create a Feature and an Update Site project to allow easier distribution and updates of the intermediate iterations. Generating Code Now, as we have a complete Xtext DSL defined and in place let’s have a look at the Code Generation side of things. This part is completely optional: You are free to include the necessary Xtext libraries into your applications runtime (although they seem to be numerous) and just use them to dynamically load and parse .navi files on-the-fly. This would probably be a good idea if you were writing an Eclipse based application anyway. However, when targeting a very limited platform like JavaME this option is not viable. Instead we will now create a code generator that provides a transformation from the DSL syntax into more classic Java terms – specifically we will create a HashMap based data structure that carries all the same information, but in Java terms. This is a sample of what the generated output is going to look like: public class NaviRules { private Map navigationRules = new Hashtable(); // ... public NaviRules() { NaviDestination naviDest; // ========== From Login (com.danielschneller.myapp.gui.login.LoginController) // ========== On USER_LOGON_FAILED // ========== To LoginFailed (com.danielschneller.myapp.gui.login.LoginFailedController in com.danielschneller.myapp.login) naviDest = new NaviDestination(); naviDest.action = "USER_LOGON_FAILED"; naviDest.targetClassname = "com.danielschneller.myapp.gui.login.LoginFailedController"; naviDest.targetBundleId = "com.danielschneller.myapp.login"; store("com.danielschneller.myapp.gui.login.LoginController", naviDest); // ========== On USER_LOGON_SUCCESS // ========== To MainMenu (com.danielschneller.myapp.gui.menu.MainMenuController in com.danielschneller.myapp.menu) naviDest = new NaviDestination(); naviDest.action = "USER_LOGON_SUCCESS"; naviDest.targetClassname = "com.danielschneller.myapp.gui.menu.MainMenuController"; naviDest.targetBundleId = "com.danielschneller.myapp.menu"; store("com.danielschneller.myapp.gui.login.LoginController", naviDest); // ============================================================================= // ========== From LoginFailed (com.danielschneller.myapp.gui.login.LoginFailedController) // ========== On OK // ========== To Login (com.danielschneller.myapp.gui.login.LoginController in com.danielschneller.myapp.login) naviDest = new NaviDestination(); naviDest.action = "OK"; naviDest.targetClassname = "com.danielschneller.myapp.gui.login.LoginController"; naviDest.targetBundleId = "com.danielschneller.myapp.login"; store("com.danielschneller.myapp.gui.login.LoginFailedController", naviDest); // .... and so on ... } } The support class NaviDestination is omitted but is generally just a value holder struct type class. When creating the Xtext project using the wizard earlier we created a third Eclipse project, ending in ...generator. Its src folder contains three subdirectories called model, templates and workflow. Put the sample .navi file into the model directory. It will serve as the input for the generator. Create the first template Code generation is based on templates. Xtext leverages the Xpand template engine. In the templates directory create a new Xpand template using the context menu. Call it NaviRules.xpt, open it and insert the following: «REM» import the namespace defined in our DSL model «ENDREM» «IMPORT navigationRules» «REM» Define a template called "main" for elements of type "Root". The minus sign at the end takes care of not adding a newline at the end of it. «ENDREM» «DEFINE main FOR Root-» «ENDDEFINE» As there is only one instance of a Root element in a navigation rules file, this will be the main entry point - hence the name. There is no need to call it main, but it seems fitting. Now between the DEFINE and ENDDEFINE insert what is to be generated: As shown above, we need a new Java source file called NaviRules.java: ... «DEFINE main FOR Root-» «FILE "NaviRules.java"-» «ENDFILE-» «ENDDEFINE» ... Again, the contents to be generated is put in between the FILE and ENDFILE brackets. Anything not enclosed in «» will be used verbatim in the output file. So first of all, put in the static parts of the Java file. What I did was first write the source for a single navigation rule by hand, made sure it compiled and then copied over the relevant parts into the template piece by piece: ... «FILE "NaviRules.java"-» import java.util.*; public class NaviRules { public static class NaviDestination { String action; List requiredPermissions = new ArrayList(); String targetClassname; String targetBundleId; NaviDestination() {}; public final List getRequiredPermissions() { return new ArrayList(requiredPermissions); } // let Eclipse generate getters, setters, // equals and hashCode methods for this } private Map navigationRules = new Hashtable(); «ENDFILE-» ... Now, this is nothing special so far. To fill in the elements from the navigation rules DSL file put in the following: ... private Map navigationRules = new Hashtable(); public NaviRules() { NaviDestination naviDest; «REM» Iterate all elements in the "rules" collection attribute of the "ruledefs" attribute of the "Root" element. Call each iterated element (which is of type "Rule") "rule" and expand the "ruletmpl" template for it here. «ENDREM» «FOREACH ruledefs.rules AS r»«EXPAND ruletmpl FOR r»«ENDFOREACH» } ... In the class constructor we first define a local variable naviDest of the previously declared type. Then - as the comment states - the FOREACH instruction will iterate over all Rule type elements. This might not seem to be completely obvious at first. Remember at this point in the template the current scope is the "Root" element from the navigation rules file. It has an attribute called ruledefs as per the grammer definition. This attribute is of type NavigationRules which in turn has a collection attribute called rules, containing of Rule type objects. Inside the loop the current element can then be adressed by the template variable name r. The loop body (between FOREACH and ENDFOREACH) contains another Xpand instruction to expand a template called ruletmpl which will be declared next. Don't worry, even though this is a little difficult at first - switching contexts between the Java and the template scopes is made significantly easier in Eclipse, because the Xpand template editor will syntax color (static parts are blue) and also assist you with code completion inside the Xpand template parts. Ctrl-Spacing your way through it will make things more obvious than they are when reading an example. Now for the ruletmpl template. Place it below the ENDDEFINE statement belonging to the main template: ... «ENDFILE-» «ENDDEFINE» «DEFINE ruletmpl FOR Rule-» // ========== From «source.name» («source.controllername») «FOREACH destinations AS d»«EXPAND destTmpl(source) FOR d»«ENDFOREACH» // ============================================================================= «ENDDEFINE» You see the same idea used again: Static parts that get transferred into the output file 1:1 and Xpand statements that fill in data from the navigation rules definition file. In this case you see references to the attributes of the Rule element. As per the FOREACH instruction in the previous template, the one at hand will be repeated for every instance of Rule in our source file. Inside this definition the current scope is that of Rule, so with «source.name» the name attribute of the CoordinateMappingSpec object referenced as source in a Rule is taken first, then the controllername attribute likewise. Next up another FOREACH loop iterates the one or more possible Destinations of each Rule. Instead of just applying a template (destTmpl) for every Destination we also pass in the corresponding CoordinateMappingSpec stored in the source attribute of the Rule. This is then used in the following template: ... «DEFINE destTmpl(CoordinateMappingSpec source) FOR Destination-» // ========== On «transition.name» // ========== To «target.name» («target.controllername» in «target.bundleid») naviDest = new NaviDestination(); naviDest.action = "«transition.name»"; naviDest.targetClassname = "«target.controllername»"; naviDest.targetBundleId = "«target.bundleid»"; «FOREACH permissions AS p»«EXPAND permTmpl FOR p»«ENDFOREACH» store("«source.controllername»", naviDest); «ENDDEFINE» «DEFINE permTmpl FOR PermissionReference-» naviDest.requiredPermissions.add("«permission.value»"); «ENDDEFINE» In this innermost templates the attributes of the CoordinateMappingSpec objects source and target are accessed and put into place to be assigned to the members a NaviDestination Java object instance per Destination. There is only one more (very simple) template for the PermissionReference elements. With this, the Xpand file is complete. Set Up The Generator Workflow The wizard initially created a NavigationRulesGenerator.mwe2 file in the workflow folder. Open it and replace its contents with the following: module workflow.NavigationRulesGenerator import org.eclipse.emf.mwe.utils.* var targetDir = "src-gen" var fileEncoding = "Cp1252" var modelPath = "src/model" Workflow { component = org.eclipse.xtext.mwe.Reader { path = modelPath // this class has been generated by the xtext generator register = com.danielschneller.navi.NavigationRulesStandaloneSetup {} load = { slot = "root" type = "Root" } } component = org.eclipse.xpand2.Generator { metaModel = org.eclipse.xtend.typesystem.emf.EmfRegistryMetaModel {} expand = "templates::NaviRules::main FOREACH root" outlet = { path = targetDir } fileEncoding = fileEncoding } } The most interesting parts of this workflow file are the load section in the Reader component and the expand and outlet sections in the Generator component: The first one will connect a so-called slot with the Root element from our navigation rules. The second one will trigger the evaluation of the main template in the NaviRules.xpt file in the templates folder and feed any Root instances it finds in the *.navi files from the src/model (modelPath) into it. Now it is time for some actual generation. Run the generator workflow Right click the MWE2 file you just edited and select the Run As MWE2 Workflow command from the context menu. The Eclipse console will show this output: 0 [main] DEBUG org.eclipse.xtext.mwe.Reader - Resource Pathes : [src/model] 431 [main] DEBUG xt.validation.ResourceValidatorImpl - Syntax check OK! Resource: file:/Users/ds/ws/ws36_xtext/com.danielschneller.navi.dsl.generator/src/model/MyApp.navi 1013 [main] INFO org.eclipse.xpand2.Generator - Written 1 files to outlet [default](src-gen) 1014 [main] INFO .emf.mwe2.runtime.workflow.Workflow - Done. Then have a look at the newly generated contents of the src-gen source folder. If everything went alright, you should find a fresh NaviRules.java file placed there, based on the contents of your navigation rules file and the Xpand templates. Try and make some changes to the template, then re-run the workflow. You will see the changes reflected in the generated source file. Generate a second source File In the templates directory add another Xpand template file Navigation.xpt with the following content: «IMPORT navigationRules»; «DEFINE main FOR Root-» «FILE "Navigation.java"-» public final class Navigation { «FOREACH ruledefs.rules.destinations.transition.collect(e|e.name).toSet().sortBy(e|e) AS t»«EXPAND actionTmpl FOR t»«ENDFOREACH» private final String name; private Navigation(String aName) { name = aName; } public String getName() { return name; } } «ENDFILE-» «ENDDEFINE» «DEFINE actionTmpl FOR String-» /** Constant for Navigation «this» */ public static final Navigation «this» = new Navigation("«this»"); «ENDDEFINE» This is a template for a type-safe enumeration that can be used in Java 1.4 - remember I had to do this for JavaME. Notice the FOREACH loop in this case. It demonstrates that not only simple iterations are possible, but that Xpand allows more complex operations as well. In this case it will collect the names of all the navigation transitions from all the Destinations in the navigation rules. These are of type String. They are made unique by converting them to a Set datastructure and then finally sorted in their natural order. The resulting list of sorted strings is then iterated, each one - called t - is passed to the actionTmpl template. It is very simple, just placing the string itself («this») into a single line of Java source code. Of course, strictly speaking this is a rather complicated procedure to get the same information we could also have taken from the TransitionDefinitions element in the rules definition. However I think it serves as a nice example for additional Xpand capabilities. For a full description of its possibilities, have a look at the Xpand Reference in the Eclipse documentation. To use the new template, add another section to the MWE2 workflow definition: component = org.eclipse.xpand2.Generator { metaModel = org.eclipse.xtend.typesystem.emf.EmfRegistryMetaModel {} expand = "templates::Navigation::main FOREACH root" outlet = { path = targetDir } fileEncoding = fileEncoding } Running it again will produce a slightly different output, making clear that two files have been generated. This is what comes out in the src-gen folder as Navigation.java: public final class Navigation { /** Constant for Navigation ADMIN */ public static final Navigation ADMIN = new Navigation("ADMIN"); /** Constant for Navigation BACK */ public static final Navigation BACK = new Navigation("BACK"); /** Constant for Navigation DATA_LOOKUP */ public static final Navigation DATA_LOOKUP = new Navigation("DATA_LOOKUP"); /** Constant for Navigation OK */ public static final Navigation OK = new Navigation("OK"); ... More... This was just about my first experiments with Xtext. I am sure there is plenty more to be done with it. For more reading, please have a look at this very nice Getting started with Xtext tutorial by Peter Friese of Itemis. From http://www.danielschneller.com/2010/08/code-generation-with-xtext.html

There are so many differences between Agile and Waterfall that it takes a three-part series to cover it all.

There are three parts to every project, starting, finishing and everything in between. Two parts of the process are very difficult to complete, starting and finishing. This is not a tutorial on project management, as much as it is a general guide for people involved in a project. For example, lots of people have ideas. Ideas are easy because they require very little risk. But, what happens after the idea? You are supposed to start the project. However, most people stop with the idea because they “don’t have time” or even “I wouldn’t know where to begin”. Kat French explains how she does her best creative work: The super-secret, hush-hush, “I could tell you, but then I’d have to kill you” secret of how I do my best creative work. Ready? It’s called “starting.” The recipe is.. there is no recipe. This isn’t science. It’s more like alchemy. There are ingredients. Usually those ingredients have certain effects. When you put them all together and apply heat…”results may vary,” Starting does not mean that everything will go well or that you will be successful. Starting just means that you took the initiative to start, and that probably puts you ahead of the majority of workers out there. In order for a project to be successful, you have to start at some point. Most people are not good starters, they need some core foundation or baseline to start with. Some people also need the structure of a formal project management methodology or a detailed task list. The term “self-starter” has been abused by the whole recruitment/HR industry to become someone who can do their own work without significant prodding. What do you call someone who can take an idea and start a project? Some people may throw the title “entrepreneur” at that person, but it also has other meanings. The key is that this person can start something. Are you that person? One problem is that the starter may not be very good at filling in the various details of the project or finishing the project. Starters may be excited by the novelty of a project, but once you get mired in details, the novelty has worn off. By the time you are trying to finish the project, the starter is probably bored or even hates his job. Given that we know that no project is ever really done, you might be able to keep the starter happy by having them begin work on the next phase of the project or a significant new feature. At the other end of the project is completion. Starters typically do not fare well as a finisher of a project. As an example, look at the typical software development project. At the beginning of the project, there is a lot of technology research and foundation or framework code that needs to be completed. Starters love that work. At the end of a project, most of the work is in validating and correcting defects, and working with other departments to ensure deployment goes smoothly. A finisher is the person that works well juggling multiple tasks, fixing defects and managing processes to completion. Obviously, this is a very different person than the starter. The finisher loves a detailed task list as it gives them a goal. If they complete all of these tasks, it is likely that the project has reached its conclusion and the application has been deployed. However, you cannot always be really finishing a project, so how do you keep the finisher happy? Similar to the starter, you can have the finisher move from one project or feature to another. They are a nice complement to the starter in terms of the tasks to be completed. Are you a finisher? But how do you have one project look like several? In project management, a large project is broken into phases, which are really just smaller projects. If you do not have a really large project, you can create smaller projects by looking for milestones in your project. Agile methodologies take this concept to the extreme by ensuring that there is a fixed time for each iteration. In some cases, an iteration could be long enough to implement one feature. So, each feature in your product could become an iteration or a small project. So, we have talked about starting and finishing, but what about the stuff in between? Someone needs to fill in the details. I started by calling this person a filler, but that does not sound like a good name for someone. So, I will call this person an implementer. This person takes the basic infrastructure and puts the application features on top of it. They create the web forms and the code to save the data, using the frameworks provided by the starter. Most people fall into this category because it has the broadest spectrum of work. Each web page or feature may look like a new project for them. They may not require a detailed task list, depending upon experience, but they look at the requirements and fill in the details. Are you an implementer? Because most projects are full of details, the implementer has plenty of work to do. They can be moved from project to project filling in the gaps that the starter and finisher do not complete. Given that there are so many details in projects, this is where a project manager will spend a bulk of their time, managing the implementers. Implementers will also be the most diverse group of people, so management of these people could be a daunting task as well. Of course, the next question from people would be who is most valuable. For that question, I give you a quote from a Seth Godin post about linchpins: A newspaper asked me the following, which practically set my hair on fire: What inherent traits would make it easier for someone to becoming a linchpin? Surely not everyone can be a linchpin? Why not? Each of these types of people are important. What good is a starter if there is nobody there to finish? If you have a finisher, who starts the project in the right direction? Once the project is started someone needs to fill in the details, and that is not the starter or the finisher. There are some of those rare people that can take a project from start to finish, and there are others that overlap into two of the three groups. But you should be honest with yourself. What are you good at? Starting? Finishing? The stuff in between? From http://regulargeek.com/2010/06/24/are-you-a-starter-a-finisher-or-an-implementer

There are so many differences between Agile and Waterfall that it takes a three-part series to cover it all.

User stories are the substitute of formal requirements documents in an agile environment: they are short summaries of a functionality that leave space to expansion and refinement when it comes the time to implement it. Writing them it's not rocket science and it is definitely something a web developer should master. Stories are not requirements, in the sense they are not required at all: the prioritization process will choose the most important stories to implement at a given time, basing on their cost and on their value. Instead of giving a list of requirements where 90% of the features are only nice to have, the customer gets to make an informed decision over which stories should be implemented first, and can handle new requirements by adding them to the global list of stories (backlog). The typical agile estimation process is not the subject of this article, but it looks like this: asking questions to the customer generates a bunch of user stories, which go into the backlog where all the ideas about functionalities are kept. Stories are estimated in relative points or ideal time, to give an idea of their size. With the customer, the developers can choose which stories to pull from the backlog into a smaller plan (iteration or release based). if new requirements come up or a user story changes too much to be considered the same, it is put in the backlog so that the next planning process can deal with it. If it's still a priority, it will be surely included in the next iteration. How to write them A major point of user stories is their focus on the value provided to the end user and not on the technical topics related to its implementation. Technical options will be chosen to satisfy the story and to estimate its cost during the subsequent planning. User stories have usually the following overly famous form: As a [role], I [feature] so that [reason] For example, As a user, I can login into the application so that the it presents me my preferences. What is a role while writing user stories? It is the analogue of the classic Uml use case diagram role: for example it can be the customer, a user of the application, an admin, developer which uses the library you're writing. The so that part is often optional, but it should described the value provided by the feature the user story describes. The feature itself can change in development but it should conserve its original value. In our example, if we make the user login with OpenID the value is conserved even if we have thrown away our own authentication mechanism. In this sense, stories do not describe describe the how, only the what, and this particular what can change is this helps to achieve the same why (a little metaphysical definition). Keep in mind that user stories have to be testable, because they are the definition of done b: you can drink champagne only when the acceptance tests for a story are passing: consider modifying your stories if it is difficult to write automated tests for them. For instance in an application which indexed files asynchronously (and it may take a lot after the user has been returned an Indexing started message) I actually addedd a dynamic page with the last additions to the index, that is updated as the last step of the pipeline of operations, to make the story more testable. Complex stories which are unclear to test are a symptom that a refinement is needed. Where to write them The general suggestion is to write every user story on a 3x5 card because this size choice keeps the stories short and to the point. Moreover, if you write on paper you can shuffle single cards around for planning pourposes. I hate to write on paper something I may edit, so when working solo, I use txt files where a story is represented by a row. I'm currently looking for a low-footprint project management tool which does not complicate my process, but I can easily move around stories from the backlog to the release or iteration plan with vim by using dd to cut a story and P or p to paste it. This is nearly as low-tech as sheets of paper. Why web applications? Web applications adapt particularly well to iterative development and to a story-based approach. Usually a web application starts with a beta that implements the most important stories to provide basic functionalities. If a story does not gather enough success online (poor response from the users), linked stories that expand it may be delayed (left in the backlog) or cancelled, to make room for other stories that have come up. There are no problems in the client side while updating the application, as all issues with upgrading are moved to the server side (such as changes to the database schema). If the developers have access to the hosting service for the application, rolling out the result of an iteration is very easy and the users may not notice it until they start to use the new features. Compare this agility with the old MS Access applications used in the offices of half of the planet. While web applications may take over the enterprise world, legacy managerial applications are widely spread and it is difficult to substitute them in one single shot. I tried with a formal waterfall approach, and failed as the requirements were too fine-grained, and impossible to prioritize: imagine a list of an hundred of queries that can be done over the database, and no idea of which are the most used. Imagine fifty different entities which represent an outdated domain model you have to replace. How do you know where to start? Even if you can implement all these requirements, how much will it cost? An agile process is our only hope for replacing this kind of applications, and if you will someday see a PHP application generating invoices in your office, it will be in part thanks to user stories.

Steven Yi (pictured right) is a programmer and composer living in Rochester, NY. He studied music composition in college and became a programmer afterwards. He started off as a Flash and server side developer (which he did for about 7 years), and has spent the past few years at his current company doing mobile development with J2ME, Android, and iPhone, as well as server-side development with Spring, Hibernate, etc. He started learning and using Java and Swing for personal work in 2000 and has been using it since then for the development of blue, the focus of the interview that follows below. In the interview, Steven talks about the "blue" music composer, how it works, and how the NetBeans Platform and Python form the basis of this cool open-sourced Java music composer. What's "blue"? blue is a music composition environment I started in the fall of 2000. It was actually my very first Java program! At the time, I had started using the music software Csound (http://www.csounds.com) to compose, but found it slow to work with when it came to accomplishing what I was interested in musically. I had the idea to create a simple program that would have a timeline and the ability to scale musical score material in time. Fast forward many years later: in trying to solve other musical problems, and responding to feedback from the community of users, I've expanded blue's features a great deal. It now includes things like a mixer and effects system, a GUI builder tool for creating synthesizer interfaces, embedded Jython processing of musical scripts, and more. It's been quite satisfying to create a tool that can express my musical interests and to find a community of users who have found value in this program for their own musical work. Some screenshots: The Orchestra manager shows a BlueSynthBuilder instrument being edited. The "Reson 6" instrument is shown in edit mode. The BSB Object Properties panel shows the properties for the selected knob: The Score timeline shows a project using multiple parameter automations. The values automated include things like volume, panning, and a time pointer for a phase-vocoder instrument. All BlueSynthBuilder instruments, Effects, and the Mixer volume sliders can be automated: The Score timeline showing the author's composition "Reminiscences". The timeline shows multiple Python SoundObjects used. The SoundObject Editor shows the editor for the selected SoundObject in the timeline. The SoundObject Properties panel shows different properties for the selected SoundObject: The Score timeline showing a Tracker SoundObject being used. The timeline is configured to snap at every 4 beats and the time bar has been configured to show in numbers rather than time: The Score timeline showing a PianoRoll SoundObject being used. The PianoRoll is unique in that it is microtonal, meaning it can adapt the number of steps per "octave", depending on the values configured from a tuning file. In this screenshot, the scale loaded was a Bohlen-Pierce scale, which has 13-tones per tritave (octave and a half): The blue Mixer is shown docked into the bottom bar and in an open state. The interfaces for user-created Chorus and Reverb effects are shown. The interfaces were created using the same GUI builder tool that is found in the BlueSynthBuilder instrument: It's got a very special appearance. How did that come about? blue's custom look and feel started off one day when I was using my Palm PDA. I remember thinking that I enjoyed the look of the device with the backlight on, and so I wanted to recreate that kind of look for my program. Later, I modified the color scheme to tone it down in some ways, but I also introduced more colors than white and cyan to highlight secondary and tertiary features. Maybe now it is now more like Tron than it is like Palm. :) Overall, I enjoy the darker look of the application when I'm working on music. I tend to work on music when I have free time, and that is usually only late at night—I've found having a darker screen has been easier on my eyes. Also, if anyone was wondering, yes, blue is my favorite color. The blue look and feel is encapsulated in a module named "blue-plaf" and is available in the "blue" Mercurial repository (http://bluemusic.hg.sourceforge.net/hgweb/bluemusic/blue). The look and feel is quite hacked up (redoing it properly has been another item on my todo list), but it can be dropped into another application and it should work, as shown below with the CRUD Sample (which can be created from a tutorial found here): Can you explain how blue's timeline works? blue has a concept of SoundLayers and SoundObjects. SoundObjects are objects that primarily produce notes and have a start and duration. There are many different types of SoundObjects in blue and each has an editor (viewed in the SoundObject Editor TopComponent when a SoundObject is selected), and a BarRenderer, which is used to draw the content area of the bar on the timeline. A PolyObject is a special SoundObject. It consists of SoundLayers, which contain SoundObjects. The root timeline is itself just a PolyObject that you can add as many layers to as you like. You can also group individual SoundObjects into their own PolyObjects, and then use the resulting PolyObjects just like any other SoundObject on the timeline. If you double-click a PolyObject, the timeline is then reset with the timeline of the PolyObject you selected. As a result, PolyObjects allow timelines to be embedded within other timelines. If you think about how music is grouped into motives, phrases, sections, and even larger groupings, you can see how PolyObjects might represent these kinds of musical abstractions. For the component design, the ScoreTopComponent starts off with a JSplitPane to split between a SoundLayerListPanel on the left and a JScrollPane on the right. The JScrollPane has a ScoreTimeCanvas (the main timeline) in the main viewPort's view, a panel with the the time bar and tempo editor in the column header, and the corner is used to open up an extra panel to modify properties for the timeline. The JScrollPane has customized JScrollBars used to add the ± buttons that perform zooming on the timeline. There are a number of other features involved that are implemented amongst a number of classes, but the details of how viewPorts are synchronized (among other things) may be a bit too technical to discuss here. For those who are interested, the code can be viewed within the blue.ui.core.score package within the blue-ui-core module. How did blue come to find itself atop the NetBeans Platform? I first started to be interested in NetBeans IDE around the time 4.1 came out, but didn't really get into using it until the release of 5.0. At that time, I had hand-written Swing components for about 4-5 years (I don't really remember when 5.0 was released), and I found Matisse to be quite nice and began using it here and there. I had looked at the NetBeans Platform as an RCP at that time, but found it to be quite a bit to understand. However, I still kept it on my radar. Around the time 6.0 or 6.5 came out, I started to reconsider migrating to the NetBeans Platform once again. By this time, I had moved over to using NetBeans IDE full time for blue development and had been using NetBeans IDE more in general—particularly Java Web development and Ruby on Rails. One of the biggest things I found attractive about NetBeans IDE is its windowing system... and the things I read about in the platform development articles I'd seen online made me curious once again to see what the NetBeans Platform offered. I still felt that there was going to be a big learning curve to learn the NetBeans Platform, but the NetBeans Platform tutorials online were really quite helpful, as were the members of the NetBeans Platform mailing list, and there were also many more books available to help me get started. I think I ultimately spent about 6-8 months migrating blue to using the NetBeans Platform. Granted, it was a busy time in my life and I was working on this only in my spare time, so I think in the end it was a reasonable amount of time. Users have been very positive about the new blue interface and application as a whole, and I think it has been worth spending the time to use the NetBeans Platform. blue's window layout is quite unexpected for a NetBeans Platform application. By the time I had started migrating blue to the NetBeans Platform, the application was already some 7-8 years old. The interface I designed for blue in pure Swing was influenced by my experiences in using Flash, looking at other music composition environments (Digital Audio Workstations and Sequencing Programs), and evaluating the different aspects of working with Csound. Mapping the components from the Swing-based application to the NetBeans Platform was a little tricky in that I couldn't quite get the exact same design of panels as I had in pure Swing. In the end, I tried to think about where most of the components resided physically, and created TopComponents and placed them in the center, left, right, or bottom parts of the main window. I kept some of the dialogs from the old codebase as-is, but I migrated others to be TopComponents so that they could be docked, opened, or dragged out into a dialog as the user wished. In the end, the GUI is different and took a little getting used to after years of using and building the old interface, but I quickly adjusted to the changes and I think there is much greater consistency and usability now. The users have responded very positively to the general polish of the application and to being able to customize their environment. I myself have very much enjoyed being able to dock all of the windows as well as using full-screen mode, especially when I am on my netbook and composing. Excellent! What features of the NetBeans Platform are you using and what do you find to be most useful? Currently, I am using only a very small part of the NetBeans Platform. By the time I started to move my code to the NetBeans Platform, the codebase was already some 7 or 8 years old. I took the approach recommended to me on the mailing list and started off small, focusing primarily on migrating my project to using the Windows System API, the Options API, and a few other utility API's like IO and Dialogs. Having an old codebase, I found that I spent most of my time during migration just reorganizing my UI into TopComponents and working out communications between the components. I also spent time looking at API's that I had developed myself and seeing which ones could be replaced with API's provided by the NetBeans Platform. At this time, the application is still using a number of API's I wrote from the old codebase, but over time I would like to migrate more of the appplication to use the Nodes and Visual Library API's. I think migrating a codebase of this size in phases really worked out well. In the first phase, I was able to take advantage of the Window System API and have a very visible result on the application and gained a lot for usability. Also, a big part of the migration involved moving the codebase from a monolithic source tree and partitioning it into logical modules. I think there really is a great deal of benefit to working with a codebase with modular design, and that too is a very positive result of working with the NetBeans Platform. Please say something about how Jython relates to this application, how you are using it (what the benefits are), and your general opinions on Jython. I have had a Python SoundObject in blue for quite some time—I think since 2002. For me, it is one of the most important tools in blue when it comes to accomplishing what I want musically. With computer music, we have a lot of tools for what I call Common Practice computer music: PianoRolls, Pattern Editors, and Notation Objects. For computer musicians who are interested in Uncommon Practice music, the ability to use a scripting language opens up a number of ways to express musical ideas that cannot be easily conveyed using those other tools. In blue, Jython is primarily used to allow users to write scripts that will generate notes. For myself, I use Python scripts to model orchestral composition, creating Performer and PerformerGroup objects that I write in Python. I also write performance functions, usually per-project, to perform different musical material in different ways. Other users have used Python scripts in exploring things like algorithmic composition and genetic algorithms in their work. A blue project can contain any number of Python objects. The score generated by each Python object is translated and scaled in time by moving and resizing the SoundObject in the timeline. This allows a user who may want to use scripting to create musical material to also take advantage of blue's timeline to organize how the different musical objects will work together. One of the things I most appreciate about Jython (and scripting languages on the JVM in general) is that it is embeddable within a Java application. By packaging and embedding the Jython interpreter within the blue application, users can rest assured that the Python scripts they write can be interpreted anywhere that blue is installed. It's an extra assurance that their musical projects will be long-lasting, but they can still take advantage of a full programming language like Python in their work. Overall, I think that Jython is a fine piece of software and I hope that it will continue to grow and develop for years to come. Is the application open source and are you looking for code contributions and, if so, in which areas? Yes, the application is available under the GPL v2 license, and the source code can be viewed from the Mercurial repository on SourceForge at http://bluemusic.hg.sourceforge.net/hgweb/bluemusic/blue/. I am a strong proponent of open source, especially for creative work. In the same way that we can today look at and study musical works by composers of the past (like Josquin and Bach), I would like to imagine that the work composers and other artists are now creating with computers will also be open and available for study in the future. I believe that using open source software for creative work greatly helps in making musical projects available for the years ahead. I have done most of the development of blue myself, and over the years I've certainly built up a long list of things that I would like to implement. Users have also made wonderful feature requests that I would love to see in the program—but unfortunately, there are only so many hours in the day. It would certainly be nice to have others contributing code! Beyond new features, there are a number of infrastructural things that would be nice to address. The codebase is many years old, and while the application has been refactored multiple times over its lifetime, there are still some areas of the application that could be much more cleanly implemented. Also, in moving over to the NetBeans Platforms, I only really took the first steps. There are a number of components within the application that could probably be better served by migrating to using more of the NetBeans API's. For internal work, things like modifying the timeline to implement zooming to use Graphics2d and transforms, implementing a better waveform renderer for audio files, and further enhancing the instrument GUI builder are all things I'd like to see. I'd also love to get help in migrating all of the tables and trees to using the Nodes API, something that I have not yet had the time to do. It would also be nice to get the manual (currently in HTML and PDF, generated from DocBook) integrated into the application as JavaHelp, but this is another thing that I have had to postpone due to lack of time. For features, some interesting things I'd love to see are a Notation SoundObject, a separate graphical instrument builder using the Visual Library API, and a Sampler instrument. There's also a sound drawing SoundObject, enhancements to existing SoundObjects, and more I'd love to see moving forward. Maybe someone will find these kinds of things interesting and will take a look at blue's code sometime! Thanks Steven and happy music making with blue!

Today, CollabNet announced that it has acquired Danube Technologies, a company that develops Scrum project management software and conducts Scrum certification training. Founded in 2000, Danube was a small business with about 30 employees before the acquisition. Despite their small size, Danube is one of the leading Scrum training organizations in the world and they are the authors of the popular ScrumWorks software for implementing Scrum. CollabNet says they want to make ScrumWorks a part of their cloud-based developer toolset. ScrumWorks and the commercial version, ScrumWorks Pro, will continue to be sold separately from Team Forge, Collabnet's core ALM platform. In the short term, CollabNet will link the two platforms by enabling defect reports and commits tracked in TeamForge to update ScrumWorks. The backend repositories of TeamForge and ScrumWorks will remain separate and they will continue to be sold separately. CollabNet says a combined solution will be available in the second quarter of this year. ScrumWorks Pro's Enhanced Burndown Chart CollabNet will focus on making ScrumWorks available on it's cloud-based infrastructure in order to allow distributed teams to use the product. CollabNet CEO Bill Portelli believes that Scrum has become "the de facto method of managing software projects." Although many companies, including Collabnet, have branded themselves as agile, they are usually not providers of purpose-built agile tools (like Rally Software and VersionOne). Today, CollabNet can say without a doubt that they are a provider of true agile development tools (even though ScrumWorks is focused on one methodology). Before the Danube acquisition, CollabNet simply had an agile template on top of their TeamForge platform, which was not originally designed for agile processes. CollabNet was best known instead for being one of the first open source tool vendors to become profitable. The company is well known for being co-founded by Brian Behlendorf, the co-founder of Apache, and for creating the popular source control/config management tool, Subversion. With the acquisition of Danube, CollabNet could become very successful by expanding into the agile realm. Requirements management is the one area where the combined TeamForge/ScrumWorks portfolio is still lacking. The terms of the deal were not disclosed, but Danube CEO and co-founder Laszlo Szalvay says that the Danube team "couldn't be happier" with the terms. Laszlo's brother and Danube co-founder Victor Szalvay is going to become the CTO of CollabNet's new Scrum Business unit. You can see DZone's interview with Laszlo Svalvay at the Agile 2009 conference.

There are at least four methods that can be used in different combinations to make the process of setting up a complete development environment a lot less painful.

Chris Bohme is the chief software architect at Pinkmatter Solutions – a small, specialized software development company in South Africa. Pinkmatter has been working with a company called Paterva for the past few years to build Maltego - a tool for data visualization, reconnaissance and intelligence gathering. Maltego is used by law enforcement and intelligence agencies, network security professionals and large corporates to discover and analyze information. In a nutshell, how does Maltego work? Maltego models information as entities (e.g., persons, e-mail addresses) and relationships between them. Relationships are discovered by running pluggable functions (called transforms) on the entities. For example, when running a social network transform on my e-mail address, one would discover my Facebook and LinkedIn profiles. Out of the box, Maltego ships with over 150 transforms that mainly relate to open source intelligence. However, an organization using Maltego user can easily create their own transforms that run on their internal data. The concept of transforms makes data gathering very quick and easy which is one of the aspects that sets it apart from some of its competitors like Analyst Notebook, which has been the de-facto tool for investigation and intelligence analysis. Why and how did you choose to use the NetBeans Platform as the basis of this application? We have actually been using the NetBeans Platform at Pinkmatter since 2002, back in the days of NetBeans 3.2, when the NetBeans Platform was not really separate from the IDE and the only real documentation for NetBeans Platform users was the source code. Back then Pinkmatter was building a network security management tool we called “Palantir”, which was never released but which would later form the basis framework for Maltego. (Ironically one of Maltego’s competitors is now made by a company called Palantir Tech.) I was using Forte (Sun’s customized version of NetBeans) as my IDE for Java development and realized that I would need very similar features in Palantir – global selection management, runtime composition (i.e., modules), copy/paste/undo/redo, auto-update, property grid, window manager, system palette etc. So I began reading through the sources and building Palantir as a NetBeans module while trying to remove as much of the IDE parts as possible. I immediately fell in love with its design and complexity (yes, complexity – no matter how long you have been using the NetBeans Platform, there is something new you can learn every day) – but there was a definite beauty to it and I knew that following its architecture guidelines would save me from the certain “spaghetti-death” to which all large UI applications I had seen thus far were doomed from the start. What are the main advantages of the NetBeans Platform to you? On a personal level, working with the NetBeans Platform early on in my developer career has shaped my mindset around application design. As such, the NetBeans Platform source code was one of my most influential teachers when it comes to API design and architecture of large complex applications. I started looking for similar patterns in the frameworks I was building using other programming languages and it has helped me identify designs that are “right” and those that are “wrong”. (When it comes to API design I believe that “truth, like beauty, is not a matter of opinion” :-) ) On the level of Maltego, I think the benefits are fairly obvious – there is a platform that comes with lots “free stuff” right out of the box. And hey, the best thing is, someone else improves, fixes and supports all this free stuff while you can focus on your specific problem domain. If I were to rephrase the question to read “what in the NetBeans Platform couldn’t I live without?” – well, it would be the features related to runtime composition. The fact that components can be registered declaratively (for example in layer files) and are added as modules that get loaded at runtime shapes the overall design and maintainability and is something a modern application cannot do without. As Maltego matures, instead of removing the dependency on some NetBeans APIs and replacing them with our own, we tend to use more and more of what the NetBeans Platform (and even the IDE) has to offer. This is a very good indication to me that a) NetBeans Platform was the right choice to build Maltego on and b) that the evolution of the NetBeans Platform is in line with the needs of its users (well, at least for us). Continue to part 2 of this interview... Were there things that pleasantly surprised you while working with the NetBeans Platform? There were many.... but let’s start with backward compatibility. A lot of the Palantir code from 2002 can still run in NetBeans 6 – that is 3 major versions and 8 years later! – not a small feat to achieve for an API designer. As another example, for the upcoming 3.0 of Maltego we redesigned our underlying information model to allow a user to model entities with a multitude of properties. We needed to allow the user to configure these using many kinds of weird and wonderful type editors... and actually the good old PropertySheet works well for that, can be highly customized and takes up very little screen real estate. In general I am amazed every time how efficiently NetBeans can handle so many modules (and merged layer files)! What could be improved? Well, I have this gripe with the wizard framework. Although sufficient for the IDE, there is a lot to be desired from wizards when used in other applications. How about re-using wizard panels for editing something in a dialog (panels as tabbed panes for example)? Or quick and dirty mechanisms to disable the Cancel button or intercept it to cancel a background thread? (I know, I know, stop complaining, Chris, and contribute something of that sort – yes... one day when Maltego has grown up and I am no longer working nights.) But in the end I think that in spite of all the great efforts that have been made, documentation is still a limiting factor when it comes to the adoption and effective use of the NetBeans Platform. There are a number of really good books, blogs and tutorials, however, I feel there is a need for something like “An Architect’s Guide for Designing Applications for the NetBeans Platform” – something that focuses more on core design decisions that have to be made before getting started. For example, “how is your global selection management to work?” and “what mechanisms does the NetBeans Platform provide for that?” Any tips or tricks for other NetBeans Platform developers? Read every book that has ever been published about the NetBeans Platform. Read and take note of tips published on blogs – you might not need them today but in 6 months time you will remember that there is a smart way to do something. I check planetnetbeans.org every day for interesting articles. Keep a copy of the NetBeans Platform sources around (you can download them in a handy ZIP file and don’t even have to do a checkout). Whenever there is something that you don’t understand or that seemingly does not work, grep the sources for the relevant classes. Don’t feel you have to make use of NetBeans APIs all the time. Sometimes it makes sense to just use a JTable instead of creating a Node implementation with OutlineView. As that component gets more full featured, you can always refactor it and replace it with a suitable View. The default lookup is your friend! Finish this sentence: "If I had known..." Actually, if I had known that it is possible (and easy) to replace the default implementation of ContextGlobalProvider I would have more hair left on my head! (Before I read Tim’s blog entry, activating a TopComponent would amount to changing the global selection – something that is not valid for all applications – and boy did I struggle...) What's the future of the application? We are close to releasing Maltego 3.0 – the next big milestone in the life of our beloved baby. This release brings many new features with it, not least of all a slick new look (thanks to some of the beautiful work done by the likes of Gunnar Reinseth, Mikael Tollefsen and Kirill Grouchnikov): Our ultimate vision is to evolve Maltego into an autonomous information monitoring system – something like an IDS (intrusion detection system), but for information. The threats to organizations (or governments) on the internet are no longer constrained to attacks on their network infrastructure (the origin of the term IDS) but information about them, their competitors or employees floating around on the internet can seriously harm them. Think of it as a highly customizable, intelligent Google Alert, which is fed from the internet as well as private, internal databases. Subsequent releases will bring us closer to that vision with geo-spatial data, time base analyses and live, real time data feeds.

It’s housecleaning time again, and like last time, I stumbled across an article I wrote back in 2006 that I don’t believe ever reached publication (at least, I don’t think it did…how am I expected to remember what I did in 2006?). For the most part, I’ve left it in its original state, except that I removed the Agile Manifesto and 12 supporting principles. There are easily enough found on the Agile Manifesto website, and the article is long enough without this duplication. The wordle at right shows the most common words used in this document. Here, in it’s otherwise unadulterated glory, is Agile: A New Era of Software Development. Agile: A New Era of Software Development Embrace Change Writing code is easy, but developing software is hard. While syntax errors are common, their severity pales in comparison to the logic flaws inherent in many software systems. The difficulty in software development is not writing code or applying a certain technology stack. Instead, the challenge lies in the specification and design of the software itself. Therein lies the essential complexity of software development, an idea introduced by Frederick Brooks in his 1987 article titled, “No Silver Bullet” [Brooks]. The most difficult aspect of software development is deciding what, exactly, needs to be built. There is certainly evidence backing this claim. The original Chaos Report shows the top three impediments to a successful development effort are lack of user input, incomplete requirements and specifications, and changing requirements and specifications [CHAOS]. No other activity, if done incorrectly, stands to compromise the system more than incorrect requirement specifications. It might not be so difficult were software a concrete entity, existing in a world where we could easily visualize it’s structure and behavior, allowing us to more reliably assess and share the impact of change. But software is a highly conceptual, invisible construct. It is considered infinitely malleable by those not intimately familiar with the conceptual complexity of it’s structure and behavior. The contractor building your home would look at you with incredulous disbelief if you suggested that the house he has 90% complete no longer met your needs, and you asked that he move walls. Or imagine how ridiculous it would sound to suggest that a new third floor be inserted to a 100 story skyscraper. Physicists labor on with firm belief that there exist an underlying set of unifying principles to serve as guidance. Or at least, there are laws of physics that we hold to be true. There are no such rules or principles that guide software development. We are left with the imagination and dreams of our clients, and they demand and deserve rapid response to change. We have made valiant attempts at conformity. Ceremonial processes attempting to define standardized activities that guide the development process have failed, however. We cannot define detailed up-front requirements specifications and expect them to survive the development lifecycle intact. We cannot establish an initial design of the conceptual construct and expect the structure to go unscathed throughout the process of construction. Software development is an error prone human activity involving experts with varying backgrounds and skills who must come together and attempt to communicate uniformly, working as a team toward a common goal. While tools and process do help, we must also accept that change is expected. We cannot treat change as evil. Instead, the tools and process used must allow us to accommodate change, treating it as an inherent part of software development. Changing requirements is a rule of our game. The software we develop must be malleable and adaptive to change, and the process we use must embrace change. We often draw comparisons between software development and various manufacturing processes. As Larman points out, however, manufacturing is a predictive process [Larman]. Herein lies one of the greatest differences between software development and the manufacturing processes to which we often draw comparisons. Manufacturing is a repeatable activity, with high rates of near-identical creation where a consistent product is produced in assembly line fashion. Little change is expected, making it possible to reliably estimate and predict the outcome. Software development is much more like new product development, where evolutionary specifications and adaptive planning is necessary to deal with the many unknowns that lie ahead. Agile Principles In early 2001, a small group of industry experts converged in Utah to discuss alternatives to heavy, document driven software development methods. Emerging from this meeting was the Agile Manifesto, a symbolic proclamation endorsing the virtues of a lighter, more flexible, people-oriented approach to software development, giving birth to the agile software development movement. (Since this is already a long article, I’ve snipped the manifesto and principles, which were included in the original version. If you’re interested, you can view the manifesto and its 12 principles on the Agile Manifesto website.) The ideas behind these 12 principles are simple, and contain no hidden messages. Of course, there are different techniques embodied within various agile processes that support these principles. The one certainty is that agile teams definitely work differently from their less agile peers. They recognize there is one end goal - to create a working, functional software product. With that in mind, they work very closely with project stakeholders throughout the development lifecycle, knowing it is the stakeholders who possess the knowledge the system must embody. Agile teams work very hard to deliver working software iteratively and incrementally, and they adopt techniques representative of that ideal. Agile project managers tend to favor intense communication and collaboration over heavy documentation. Empowering team members to make decisions enables responsiveness to change. Facilitating and negotiating requirements scope provides important feedback, helping plan future iterations, where each iteration produces a deliverable that can be shared with clients and stakeholders. Instead of forcing the team to follow a predictive project plan, agile project managers are more opportunistic. They prioritize features based on stakeholder feedback, and make adjustments as the iterations progress. Concurrent and parallel activities are favored over a phased approach. Agile project managers tend to guide the team instead of manage the team, and strongly discourage unnecessary overhead. Agile developers work with a similar set of goals, knowing functional software must be delivered early and often. They work judiciously to grow a code base built upon a solid foundation, where each day represents a step forward. They integrate frequently, and do not tolerate failed builds. A rich suite of tests provide the courage necessary to respond to change when the need arises. They avoid the notion of code ownership, empowering other developers to make improvements to any component of the software product. A common misconception is that agile processes discourage all documentation. This is untrue. Agile processes discourage unnecessary documentation, favoring collaboration as the preferred technique. Instead of using documentation to drive communication, agile processes favor face-to-face communication. Documents are encouraged by agile processes, so long as the need is immediate and significant. Transitioning to Agile Agile software development is based upon the fundamental premise that we must drive and respond to change quickly. The Agile Manifesto and 12 supporting principles serve this premise well. Advocates of agility claim speedier delivery of software, software with more business value, increased team productivity, higher quality systems, and a more enjoyable development experience. I believe each of these to hold true. Agile teams not only welcome change, they are able to respond to change at all levels of development. A project manager might discuss a changing requirement with a business client, empower a business analyst to schedule a meeting with the client to discuss further details, while a developer assesses the impact of change knowing she has the courage to accommodate the request because of the rich suite of unit tests in place. Saying you’ll be more responsive to change and creating an environment that embraces change are separate beasts, however. Practicing agility is hard work, especially if your team is accustomed to more traditional approaches. As with many things new and unfamiliar, some resistance will no doubt arise by those who aren’t fully convinced. Agile projects differ greatly from their less agile counterparts, and skeptics will have many opportunities to express their discontent. As someone experimenting with agility, you may even have doubts. But don’t be discouraged, and give your agile transition the time it deserves. One of the most significant changes you may experience is a feeling that you’ve been thrust into a chaotic nightmare. I doubt it’s unusual to feel this way. You’ve lost the security of detailed requirements specification and user sign-off. You are writing code without the comfort of knowing exactly what your stakeholders want. The detailed plans that have served as your security blanket on past projects no longer exist. And the celebrations accompanying completion of your various phase milestones are gone. Of course, these were all false comforts anyway. Stakeholders always changed their minds. Your detailed requirements and plans were outdated as quickly as they were completed. Instead, you’re now working in shorter iterations with vague requirements. Initial releases early in the lifecycle may be completely thrown away. Your first few weeks may seem wasted, with little or no valuable artifacts produced. Naysayers will immediately come forward and cite the lack of progress. Previously, those first few weeks or months were spent producing very detailed requirement specifications and beautiful design models. But don’t give up yet. In that previous world, you were only delaying risk and postponing integration, avoiding the most difficult aspect of software development until the end of the lifecycle. Now you’re attacking risk early, prioritizing features, and working hard to develop a functional piece of software as early as possible. Progress may not be at breakneck speeds, but you are learning a tremendous amount about the requirements of the system, and your velocity is sustainable. Additionally, you are also performing a number of other important lifecycle activities. Depending on the level of ceremony and bureaucracy within your organization, you will experience varying degrees of success when adopting agile techniques. As with any new technology adoption, it’s best to phase the transition. Some agile techniques are easier to adopt than others, and some serve as valuable catalysts to adopting additional techniques in the future. Don’t attempt to completely redefine how you work. It’s relatively easy to phase the agile transition, and you’ll want to adopt those principles that offer you the greatest initial reward. For instance, if you’re struggling to produce quality software at a consistent rate, implementing a continuous integration strategy will help you frequently verify the quality of your work. In addition to the comfort of knowing you have a product always in a functional state, the ability to share the product with clients using functional demos and prototypes will tighten the feedback loop and offer valuable insight to the client’s perception of the software. In a number of cases, I’ve found this to be valuable in identifying subtle requirements that can be difficult to identify in other requirements elicitation venues. Empirical Evidence In recent years, there has been a significant amount of research comparing agile development methods to their waterfall counterpart. In Agile and Iterative Development: A Manager’s Guide, Craig Larman illustrates the advantage of agile development through detailed analysis of multiple studies[Larman]. The compilation of his results are illustrated below. A study by Alan MacCormack at Harvard Business School explored whether evolutionary development techniques yielded better results than the waterfall model. The study included applications ranging from application software to embedded systems, with median values of nine developers spanning a 14 month development cycle. A key conclusion of the study, in which 75% of participants used agile techniques compared to 25% using waterfall, explained releasing software earlier, rather than later, contributed to a lower defect rate and higher productivity. There was little evidence showing that a detailed design specification resulted in a lower defect rate, however, reviews with peers did help in reducing the rate of defects. The study found that iterative and agile practices have a significant impact on defect and productivity factors, as indicated by the following points. Releasing a system with 20% of the functionality complete is associated with a decrease in the defect rate of 10 defects per month per million lines of code as compared to waiting to release a product until 40% of the functionality is complete, and an increase in productivity of eight more lines of source code per person-day. Continuous Integration, the idea of integrating and testing code as it is released to your source code repository, resulted in a decrease in the defect rate of 13 defects per month per million lines of code, and an increase in productivity of 17 lines of source code per person-day. The study also found four practices that were consistently used by the most successful development teams. The first two are deeply embedded in the ideals of agile software development. Releasing early and often to project stakeholders, using an iterative lifecycle. Continuous integration, with daily builds including regression testing. Teams with broad experience delivering multiple projects. Careful attention to modular and loosely coupled, componentized architectures. In a separate study [Shine], 88% of organizations cited improved productivity when using agile methods, and 84% cited improved productivity. 49% stated that the cost of development was less when using agile methods. Additionally, 83% claimed increased business satisfaction and 26% claimed significantly better satisfaction. Another study by Boehm and Papaccio [Boehm] discovered that a typical project experiences a 25% change in requirements, while yet another [Johnson] showed that 45% of features were never used. There have also been many research efforts devoted exclusively to the analysis of waterfall methods. Below is a summary of these findings, taken from a variety of studies. Scope management related to detailed up-front requirements was a significant contributing factor of failure [Thomas]. The U.S. Department of Defense (DoD), when following a waterfall lifecycle, experienced a 75% failure rate [Jarzombek]. This resulted in the DoD adopting a more iterative and agile approach. On a study including 400 waterfall projects, only 10% of the code was deployed. Only 20% of code deployed was used. The main factors included changing and misunderstood requirements [Cohen]. As these studies clearly illustrate, there is significant evidence showing that agile and iterative techniques offer significant advantages over the waterfall model of development. In fact, for larger projects, the statistics supporting agility were even more pronounced. Conclusion There are a variety of agile processes available to choose from, and each abide by the spirit of the manifesto and it’s 12 supporting principles. The agile movement and it’s supporters recognize that software development is a human (though not always humane) activity. Instead of forcing process on people, agile methods allow process conformance to people. Good people, working toward a common goal, can achieve great things will little ceremonial process, assuming you give them an environment that empowers them. Solid empirical evidence backs this claim. And if the quality of people is in question, it’s doubtful that any process will produce success. References [Alliance]. The Agile Alliance. Manifesto for Agile Software Development. 2001. http://www.agilemanifesto.org [Boehm]. Boehm, B, and Papaccio, P. Understanding and Controlling Software Costs. IEEE Transaction on Software Engineering. October 1988. [Brooks]. Brooks, Frederick. No Silver Bullet: Essence and Accidents of Software Engineering. 1987. [CHAOS]. The Standish Group International, Inc. The CHAOS Report. 1995. [Cohen]. Cohen, D., Larson, G., and Ware, B. Improving Software Investments through Requirements Validation. IEEE 26th Software Engineering Workshop. 2001. [Jarzombek]. Jarzombek, J. The 5th Annual JAWS S3 Proceedings. 1999. [Johnson]. Johnson, J. Keynote speech, XP 2002, Sardinia, Italy. 2002. [Larman]. Larman, Craig. Agile and Iterative Development: A Manager’s Guide. Addison-Wesley, 2004. [MacCormack]. MacCormack, A. Product-Development Practices That Work. MIT Sloan Management Review. 2001. [Shine]. Corporate Report. Agile Methodologies Survey Results. Shine Technologies Pty Ltd. Victoria, Australia. 2003. [Thomas]. Thomas, M. IT Projects Sink or Swim. British Computer Society Review. 2001. From http://techdistrict.kirkk.com

Always in search of the absolute minimum of ceremony, my last team "discovered" a useful agile practice that takes 60 seconds from start to end: the ROTI Meeting.After every meeting, on the way out the door, draw a diagonal line on the whiteboard with the labels 0, 2, and 4. Each person in turn gives a number on how the meeting performed as a "Return on Time Invested" and the person with the marker draws in the rating. Here is the rating scale we used: 0 = "I'd have been better off making a Starbuck's run. Complete waste of time" 1 = "You really should have let me stay at my desk and code" 2 = "This was an OK meeting. About as valuable as if I'd been coding" 3 = "Surprisingly, this was more valuable than if I'd been writing code" 4 = "Wow, this meeting saved me tons of time. Thank goodness I didn't skip it to code" And then each person answers the same question, "What could be done to improve your number by one point?" To do this in 60 seconds means there is no discussion. The feedback is what it is; no debating, no fixing problems, and no hurt feelings. ROTI meetings create tacit, organization knowledge that can be acted upon by team members in the future. It drives a team towards less meetings (almost always a good thing), pushes team members to be more respectful of each others time and expertise, and influences meeting organizers to craft more succinct, on topic, and meaningful gatherings. It takes only 60 seconds so you might as well try it a few time! ... and now the historical details. ROTI analysis is nicely described in Esther Derby's great book "Agile Retrospectives". The practice in the context of iteration retrospectives takes more lie 5 to 10 minutes. Our team found ROTI to be so effective in retrospectives that we shortened it and held one at the end of every meeting. The actual ROTI scale is a bit more formal than what we created: 0 - Lost Principle: No Benefit Received for Time Invested Break-Even: 1 - 2- Received Benefit Equal to Time Invested High Return on Investment 3 - 4 - Received Benefit Greater than Time Invested Lastly, ROTI charts are covered in detail a few other places as well. For a mere 60 second investment, this practice is worth trying on your team. From http://hamletdarcy.blogspot.com

Many a time, in a Java EE application, besides the user-triggered transactions via the UI (e.g. from the JSF), there's a need for a mechanism to execute long running jobs triggered over time, e.g., batch jobs. Although in the EJB specs there's a Timer service, where Session Beans can be scheduled to run at intervals through annotations as well as programmatically, the schedule and intervals to execute the jobs have to be pre-determined during development time and Glassfish does not provide the framework and the means to do that out-of-the-box. So it is left to the developer to code that functionality or to choose a 3rd party product to do that. In one of my previous projects using a different application server, I implemented a scheduler module for the application. So with that experience, I will discuss in this article how to create a simple scheduler called SchedulerApp in NetBeans IDE 6.8 that can be deployed in Glassfish v3. The example comes with a framework and the JSF2 PrimeFaces-based UI to schedule and manage (CRUD) your batch jobs implemented by Stateless Session Beans without having to pre-determine the time and interval to execute them during development time. Below is the Class Diagram to give you an overview of the application: Through this exercise, I also hope that you will have a better understanding of the Timer Service in the EJB specs and how you can use it in your projects. Note: If you cannot get your copy running, not to worry, you can get a working copy here. Tutorial Requirements Before we proceed, make sure you review the requirements in this section. Prerequisites This tutorial assumes that you have some basic knowledge of, or programming experience with, the following technologies. JavaServer Faces (JSF) with Facelets Enterprise Java Beans (EJB) 3/3.1 esp. the Timer Service Basic knowledge of using NetBeans IDE will help to reduce the time required to do this tutorial Software needed for this Tutorial Before you begin, you need to download and install the following software on your computer: NetBeans IDE 6.8 (Java pack), http://www.netbeans.org Glassfish Enterprise Server v3, https://glassfish.dev.java.net PrimeFaces Component Library, http://www.primefaces.org Notes: The Glassfish Enterprise Server is included in the Java pack of NetBeans IDE, however, Glassfish can be installed separately from the IDE and added later into Servers services in the IDE. A copy of the working solution is included here if needed. Creating the Enterprise Projects The approach for developing the demo app, SchedulerApp, will be from the back end, i.e., the artifacts and services needed by the front-end UI will be created first, then working forward to the User Interface, i.e., the Ajax-based Web UI will be done last. The first step in creating the application is to create the necessary projects in NetBeans IDE. Choose "File > New Project" to open the New Project Wizard. Under Categories, select Java EE; under Projects select Enterprise Application. Click Next. Select the project location and name the project, SchedulerApp, and click Next. Select the installed Glassfish v3 as the server, and Java EE 6 as the Java EE Version, and click Finish. The above steps will create 3 projects, namely SchedulerApp (Enterprise Application project), SchedulerApp-ejb (EJB project), and SchedulerApp-war (Web project). Creating the Session Beans Before creating the necessary session bean classes, let's look at one of the main classes, JobInfo, which will be heavily used in the application both at the front-end and back. Basically this is a Value Object class that stores information required to configure the timer. Below is an abstract of the class: package com.schedulerapp.common; public class JobInfo implements java.io.Serializable { private static SimpleDateFormat sdf = new SimpleDateFormat("MM/dd/yyyy"); private static SimpleDateFormat sdf2 = new SimpleDateFormat("MM/dd/yyyy HH:mm:ss"); private String jobId; private String jobName; private String jobClassName; private String description; //Details required by the SchedulerExpression private Date startDate; private Date endDate; private String second; private String minute; private String hour; private String dayOfWeek; private String dayOfMonth; private String month; private String year; private Date nextTimeout; public JobInfo() { this("", "", "java:module/"); } public JobInfo(String jobId, String jobName, String jobClassName) { this.jobId = jobId; this.jobName = jobName; this.jobClassName = jobClassName; this.description = ""; //Default values, everyday midnight this.startDate = new Date(); this.endDate = null; this.second = "0"; this.minute = "0"; this.hour = "0"; this.dayOfMonth = "*"; //Every Day this.month = "*"; //Every Month this.year = "*"; //Every Year this.dayOfWeek = "*"; //Every Day of Week (Sun-Sat) } //Getter and Setter methods for the above attributes... /* * Expression of the schedule set in the object */ public String getExpression() { return "sec=" + second + ";min=" + minute + ";hour=" + hour + ";dayOfMonth=" + dayOfMonth + ";month=" + month + ";year=" + year + ";dayOfWeek=" + dayOfWeek; } @Override public boolean equals(Object anotherObj) { if (anotherObj instanceof JobInfo) { return jobId.equals(((JobInfo) anotherObj).jobId); } return false; } @Override public String toString() { return jobId + "-" + jobName + "-" + jobClassName; } } Notice the class holds the information about the job and its schedule. Create the above class in the EJB project, SchedulerApp-ejb with the package name, com.schedulerapp.common. After creating this class, we are ready to create the session beans. Creating the BatchJob Session Beans In this demo, we will be creating THREE batch jobs, namely: BatchJobA, BatchJobB and BatchJobC, where each is a Stateless Session Bean that implements a Local Interface, BatchJobInterface. The Interface will have a method, executeJob(javax.ejb.Timer timer), so each of the batch job session bean will need to implement it and this becomes the starting point for the batch jobs. Let's proceed to create them and you will see what I mean. In the Projects window, right-click on the SchedulerApp-ejb project and select "New > Session Bean..." In the New Session Bean dialog, specify the EJB Name as BatchJobA, the package as "com.schedulerapp.batchjob", Session Type as Stateless and select Local for Create Interface option Notice 2 files are created: BatchJobA (Implementation class) and BatchJobALocal (Local Interface). Here I want to rename the Interface so that it has a generic name like BatchJobInterface In the project view, navigate to the BatchJobALocal file. Right-click on the item and select "Refactor > Rename...", and change the name to BatchJobInterface. Open the renamed file, BatchJobInterface in the editor, and add the method: @Local public interface BatchJobInterface { public void executeJob(javax.ejb.Timer timer); } Notice the file, BatchJobA becomes errorneous after the above is performed. Open the file, BatchJobA and you should see the error hint (lightbulb with exclamation icon) on the left side of the editor. Click on the icon and select "Implement all abstract methods" and edit the file so that it looks like this: @Stateless public class BatchJobA implements BatchJobInterface { static Logger logger = Logger.getLogger("BatchJobA"); @Asynchronous public void executeJob(Timer timer) { logger.info("Start of BatchJobA at " + new Date() + "..."); JobInfo jobInfo = (JobInfo) timer.getInfo(); try { logger.info("Running job: " + jobInfo); Thread.sleep(30000); //Sleep for 30 seconds } catch (InterruptedException ex) { } logger.info("End of BatchJobA at " + new Date()); } } As you can see, the executeJob method does nothing but just sleeps for 30 sec to simulate a long running job. And because of that, it is made an asynchronous method thru the @Asynchronous annotation so that it doesn't block the calling Session Bean. Notice also that the JobInfo object is extracted from the Timer object so that you have the information to execute your job. We will see later how the JobInfo object got into the Timer object. We will next create the other 2 batch job session beans: BatchJobA and BatchJobB using the Copy/Paste and Refactor features of NB6.8. In the project view, navigate to the file, BatchJobA. Right-click on the item and select "Copy" In the same view, right-click the package, "com.schedulerapp.batchjob" and select "Paste > Refactor Copy..." In the Copy Class dialog, enter "BatchJobB" for the New Name field and click on the Refactor button. Notice the new Session Bean, BatchJobB is created with a few easy clicks of a button. The only thing to change in the new class is the print statements, where "BatchJobA" will be changed to "BatchJobB". Repeat the above steps to create BatchJobC session bean. So we now have THREE batch job session beans: BatchJobA, BatchJobB and BatchJobC that implements the Local Interface, BatchJobInterface. We will next create the last Session Bean for this project. Creating the Job Session Bean Here, we will create the Job Session Bean whose main responsibility is to provide the necessary services to the front-end UI to manage (CRUD) the jobs and also provide the timeout method for the TimerService. In the Projects window, right-click on the SchedulerApp-ejb project and select "New > Session Bean..." In the New Session Bean dialog, specify the EJB Name as JobSessionBean, the package as "com.schedulerapp.ejb", Session Type as Stateless and leave Create Interface unchecked, i.e. no Interface (New in EJB 3.1), and click Finish. Open the newly created file, JobSessionBean in the editor and edit the content so that it looks like the following: @Stateless @LocalBean public class JobSessionBean { @Resource TimerService timerService; //Resource Injection static Logger logger = Logger.getLogger("JobSessionBean"); /* * Callback method for the timers. Calls the corresponding Batch Job Session Bean based on the JobInfo * bounded to the timer */ @Timeout public void timeout(Timer timer) { System.out.println("###Timer <" + timer.getInfo() + "> timeout at " + new Date()); try { JobInfo jobInfo = (JobInfo) timer.getInfo(); BatchJobInterface batchJob = (BatchJobInterface) InitialContext.doLookup( jobInfo.getJobClassName()); batchJob.executeJob(timer); //Asynchronous method } catch (NamingException ex) { logger.log(Level.SEVERE, null, ex); } catch (Exception ex1) { logger.severe("Exception caught: " + ex1); } } /* * Returns the Timer object based on the given JobInfo */ private Timer getTimer(JobInfo jobInfo) { Collection timers = timerService.getTimers(); for (Timer t : timers) { if (jobInfo.equals((JobInfo) t.getInfo())) { return t; } } return null; } /* * Creates a timer based on the information in the JobInfo */ public JobInfo createJob(JobInfo jobInfo) throws Exception { //Check for duplicates if (getTimer(jobInfo) != null) { throw new DuplicateKeyException("Job with the ID already exist!"); } TimerConfig timerAConf = new TimerConfig(jobInfo, true); ScheduleExpression schedExp = new ScheduleExpression(); schedExp.start(jobInfo.getStartDate()); schedExp.end(jobInfo.getEndDate()); schedExp.second(jobInfo.getSecond()); schedExp.minute(jobInfo.getMinute()); schedExp.hour(jobInfo.getHour()); schedExp.dayOfMonth(jobInfo.getDayOfMonth()); schedExp.month(jobInfo.getMonth()); schedExp.year(jobInfo.getYear()); schedExp.dayOfWeek(jobInfo.getDayOfWeek()); logger.info("### Scheduler expr: " + schedExp.toString()); Timer newTimer = timerService.createCalendarTimer(schedExp, timerAConf); logger.info("New timer created: " + newTimer.getInfo()); jobInfo.setNextTimeout(newTimer.getNextTimeout()); return jobInfo; } /* * Returns a list of JobInfo for the active timers */ public List getJobList() { logger.info("getJobList() called!!!"); ArrayList jobList = new ArrayList(); Collection timers = timerService.getTimers(); for (Timer t : timers) { JobInfo jobInfo = (JobInfo) t.getInfo(); jobInfo.setNextTimeout(t.getNextTimeout()); jobList.add(jobInfo); } return jobList; } /* * Returns the updated JobInfo from the timer */ public JobInfo getJobInfo(JobInfo jobInfo) { Timer t = getTimer(jobInfo); if (t != null) { JobInfo j = (JobInfo) t.getInfo(); j.setNextTimeout(t.getNextTimeout()); return j; } return null; } /* * Updates a timer with the given JobInfo */ public JobInfo updateJob(JobInfo jobInfo) throws Exception { Timer t = getTimer(jobInfo); if (t != null) { logger.info("Removing timer: " + t.getInfo()); t.cancel(); return createJob(jobInfo); } return null; } /* * Remove a timer with the given JobInfo */ public void deleteJob(JobInfo jobInfo) { Timer t = getTimer(jobInfo); if (t != null) { t.cancel(); } } } Take note of the followings in the above code: Timer Service is made available thru Resource Injection near the top of the class The callback method for the timers created is timeout thru the use of the @Timeout annotation Notice how the JobInfo object gets into the timer thru the TimerConfig object in the createJob method Notice how the Batch Job session beans are being lookup and accessed in the timeout method. The job class name will be the Portable JNDI name provided by the user in the UI later At this point, we are done with the EJB project, and will now move on to the Web project. Creating the Web UI using JSF 2.0 with PrimeFaces At the time of writing this tutorial, there are not many choices of Ajax-based frameworks that works with JSF 2.0 as it is still quite new. But I have found PrimeFaces to be the most complete and suitable for this demo as it has implemented the dataTable UI component and it seems to be the easiest to integrate into the NetBeans IDE. Preparing the Web project to use JSF 2.0 and PrimeFaces Before creating the web pages, ensure the JavaServer Faces framework is added to the Web project, SchedulerApp-war. In the Project view, right-click on the Web project, SchedulerApp-war, and select Properties (last item). Under the Categories items, select Frameworks, and ensure the JavaServer Faces is added to the Used Frameworks list: Before we are able to use PrimeFaces components in our facelets, we need to include its library in NetBeans IDE and set up a few things. Download the PrimeFaces library (primefaces-2.0.0.RC.jar) from http://www.primefaces.org/downloads.html [13] and store it somewhere on the local disk. To allow future projects to use PrimeFaces, I chose to create a Global library in NetBeans for PrimeFaces. Select "Tools > Libraries" from the NetBeans IDE main menu. In the Library Manager dialog, choose "New Library" and provide a name for the library, e.g. "PrimeFaces2". With the new "PrimeFaces2" library selected, click on the "Add JAR/Folder..." button and select the jar file that was downloaded earlier and click OK to complete: Next, we need to add the newly created library, PrimeFaces2 to the Web project: Select the Web project, SchedulerApp-war, from the Project window, right-click and select "Properties". Under the Libraries category, click on the "Add Library..." button (on the right), and choose the PrimeFaces2 library and click OK to complete: Because we will be using Facelets in our demo, we will update the XHTML template in NetBeans so that all the XHTML files created subsequently will already have the required namespaces and resources needed for the development. Choose "Tools > Templates" from the NetBeans menu. In the Template Manager dialog, select "Web > XHTML" and click the "Open in Editor" button. Edit the content of the file so that it looks like this: <#assign licenseFirst = ""> <#include "../Licenses/license-${project.license}.txt"> TODO write content Lastly, we need to add the following statements in the web.xml file of the Web project for the PrimeFaces components to work properly: Faces Servlet /faces/* *.jsf Resource Servlet org.primefaces.resource.ResourceServlet Resource Servlet /primefaces_resource/* com.sun.faces.allowTextChildren true At this point, we are done setting up and configuring the environment for PrimeFaces to work in NetBeans. In the sections below, we will create the JSF pages to present the screens to perform the CRUD functions. To achieve this, we will be creating THREE web pages: JobList - listing of all the active Jobs/Timers created in a tabular form JobDetails - view/update/delete the selected Job JobNew - create a new Job Creating the Backing Beans for the JSF pages Before creating the actual JSF pages, we first need to create the backing beans that provides the properties and action handlers for the JSF pages (XHTML). Here we will create TWO backing beans: JobList - RequestScoped backing bean for the Job Listing page JobMBean - SessionScoped backing bean for the rest of the JSF pages Steps to create the beans: In the Project view, right-click on the Web project, SchedulerApp-war, and select "New > JSF Managed Bean...", specify JobList as the Class Name, "com.schedulerapp.web" as the Package Name, and the scope to be request Repeat the steps to create the second backing bean, name it JobMBean and set the scope to be session instead. Edit the class, JobList, so that it looks like this: @ManagedBean(name = "JobList") @RequestScoped public class JobList implements java.io.Serializable { @EJB private JobSessionBean jobSessionBean; private List jobList = null; /** Creates a new instance of JobList */ public JobList() { } @PostConstruct public void initialize() { jobList = jobSessionBean.getJobList(); } /* * Returns a list of active Jobs/Timers */ public List getJobs() { return jobList; } } Edit the class, JobMBean, so that it looks like this: @ManagedBean(name = "JobMBean") @SessionScoped public class JobMBean implements java.io.Serializable { @EJB private JobSessionBean jobSessionBean; private JobInfo selectedJob; private JobInfo newJob; /** Creates a new instance of JobMBean */ public JobMBean() { } /* * Getter method for the newJob property */ public JobInfo getNewJob() { return newJob; } /* * Setter method for the newJob property */ public void setNewJob(JobInfo newJob) { this.newJob = newJob; } /* * Getter method for the selectedJob property */ public JobInfo getSelectedJob() { return selectedJob; } /* * Setter method for the selectedJob property */ public String setSelectedJob(JobInfo selectedJob) { this.selectedJob = jobSessionBean.getJobInfo(selectedJob); return "JobDetails"; } /* * Action handler for back to Listing Page */ public String gotoListing() { return "JobList"; } /* * Action handler for New Job button */ public String gotoNew() { System.out.println("gotoNew() called!!!"); newJob = new JobInfo(); return "JobNew"; } /* * Action handler for Duplicate button in the Details page */ public String duplicateJob() { newJob = selectedJob; newJob.setJobId(""); return "JobNew"; } /* * Action handler for Update button in the Details page */ public String updateJob() { FacesContext context = FacesContext.getCurrentInstance(); try { selectedJob = jobSessionBean.updateJob(selectedJob); context.addMessage(null, new FacesMessage(FacesMessage.SEVERITY_INFO, "Success", "Job successfully updated!")); } catch (Exception ex) { Logger.getLogger(JobMBean.class.getName()).log(Level.SEVERE, null, ex); context.addMessage(null, new FacesMessage(FacesMessage.SEVERITY_ERROR, "Failed", ex.getCause().getMessage())); } return null; } /* * Action handler for Delete button in the Details page */ public String deleteJob() { jobSessionBean.deleteJob(selectedJob); return "JobList"; } /* * Action handler for Create button in the New page */ public String createJob() { FacesContext context = FacesContext.getCurrentInstance(); try { selectedJob = jobSessionBean.createJob(newJob); context.addMessage(null, new FacesMessage(FacesMessage.SEVERITY_INFO, "Sucess", "Job successfully created!")); return "JobDetails"; } catch (Exception ex) { Logger.getLogger(JobMBean.class.getName()).log(Level.SEVERE, null, ex); context.addMessage(null, new FacesMessage(FacesMessage.SEVERITY_ERROR, "Failed", ex.getCause().getMessage())); } return null; } } Now, we have all the services and properties ready to be used by the JSF pages. Creating the JSF pages Finally, we are ready to create the THREE JSF pages: JobList, JobDetails and JobNew. In the Project view, right-click on the Web project, SchedulerApp-war, and select "New > XHTML...", specify JobList as the File Name. Note: If the item "XHTML..." doesn't appear in your menu list, select "New > Others..." instead, then in the New File dialog, select Web under Categories and you should be able to see the XHTML file type on the right. Repeat the above step for JobDetails and JobNew. Edit the file, JobList.xhtml to look like this: Job List Edit the file, JobDetails.xhtml to look like this: Help Edit the file, JobNew.xhtml to look like this: Help At this point, we are done with all the coding, and it's now time to verify the results. Perform a "Clean and Build" of the project and deploy it to the Glassfish v3 server. Testing the application Here, we will do a simple test to verify that the application is working. We will schedule 3 jobs as follows: Job 1 - run BatchJobA every 2 minutes (just to make sure we see the job running) Job 2 - run BatchJobB everyday at 11pm Job 3 - run BatchJobC every Sunday at 1am Steps to create the jobs: Go to the listing page, http://localhost:8080/SchedulerApp-war/JobList.jsf and you should see the following screen: Click on the "New Job" button below the table. Enter the details for Job 1 as follows and click on the "Create" button Click on the "Duplicate" button below to create a new Job using the current information. Enter the details for Job 2 as follows and click on the "Create" button Click on the "Duplicate" button below to create a new Job using the current information. Enter the details for Job 3 as follows and click on the "Create" button At this point, we are done creating the jobs, click on the "Back" button to see the listing. The Job List page should consists of 3 jobs that was created in the above steps Things to Note The Portable JNDI syntax for accessing the Session Beans: BatchJobA, BatchJobB and BatchJobC The "*" in the text fields represents "Every", see Java EE 6 Tutorial for details You should be able see in the log file, server.log, that BatchJobA now runs every 2 minutes The timers(jobs) are persistent, i.e. they will survive server restarts. Try restarting ther server and view the Job list again Try out the other functions of the CRUD and schedule your own jobs to see it in action. Summary Congratulations! You now have a simple scheduler to schedule your long running jobs in your application. With this framework and the GUI, you can have the flexibility and full control over the jobs you want to manage without having to pre-determine the time and interval to run them during Design and Development phase. Although the timers are persistent, the server may remove them when changes, such as new deployments, are detected. As such, you can further extend the scheduler to persist information in the database in a more dynamic and complex environment, e.g., a cluster. Good luck and have fun using the Scheduler. If you cannot get your copy running, not to worry, you can get a working copy here. See Also For other related resources, see the following: Develop Java EE 5 application with Visual JSF, EJB3 and JPA Securing Java EE 6 application with JEE Security and LDAP How to Create a Java EE 6 Application with JSF 2, EJB 3.1, JPA, and NetBeans IDE 6.8

In 1979 a group of western dignitaries visited Japan to learn more about the manufacturing models that had been applied to great success. Konosuke Matsushita, the president of Matsushita Corporation (Panasonic, Legend, Technics etc.), opened his talk with the famous statement…. “We are going to win and the industrial west is going to lose: there’s nothing much you can do about it, because the reasons for your failure are within yourselves. Your firms are built on the Taylor model; even worse, so are your heads. For you, the essence of management is getting the ideas out of the heads of the bosses and into the hands of labour. We are beyond the Taylor model.”. This leads to two questions: What did Matsushita-san mean by this bold statement?... and why did his visitors deserve such a warm welcome?! To understand Matsushita-san’s point we need to take a brief look at the history of management science. Prior to the Industrial Revolution in the 1830’s, businesses were small-scale, intimate affairs, consisting of a limited number of individuals. To look at the management of large numbers of people prior to this point requires the study of governments and armies. During the early 1800’s technological advances led to the rise of larger industrial enterprises. Factories emerged to produce goods at a larger scale and at lower costs than traditional cottage industries. Cue the entrance of Frederick Winslow Taylor to our story. Taylor was a mechanical engineer who was fascinated with industrial efficiency. He is regarded as being one of the founding fathers of management science and wrote a book on ‘Scientific Management’1. Taylor’s industrial models separated ‘working’ from ‘doing’; he believed that it was the role of management to determine the ‘one best way’ to perform the work… “It is only through enforced standardisation of methods, enforced adoption of the best implements and working conditions and enforced cooperation that this faster work can be assured. And the duty of enforcing the adoption of standards and enforcing this cooperation rests with management alone.”. When Henry Ford set about his mission to revolutionise mass transportation in the early 1900’s, he turned to the latest management thinking to make his dream a reality. Ford created a business of such scale and effectiveness that it must have seemed to his competitors and peers analogous to turning up to the proverbial knife fight with an F-14. He progressed from building a handful of vehicles in 1909 to over 500,000 units just six years later, inventing all of the technology he needed to do it along the way. The success of Henry Ford, and later Alfred Sloan at General Motors, led to a cascade in management thinking. The Taylor model that Ford and Sloan had applied to such great success became the archetypal model for the western corporation in the 20th century. Cut to 1947… The Second World War has ended and Japanese industry has been decimated by two atomic bombs: one at Hiroshima, the other at Nagasaki. The allies, keen to support the redevelopment efforts in Japan, send over a party of management consultants to help with the work required to rebuild industry. Amongst them is William Edwards Deming, a statistician and management theorist whose philosophies had been largely ignored at home. Deming, in contrast to Taylor, believed that ‘thinking’ and ‘doing’ should not be separated, and further, employees should be encouraged and empowered to make decisions on how work should be performed. “All anyone asks for is a chance to work with pride.”. While Deming had been largely ignored in the US, the Japanese got religion. In particular organisations like Toyota and Matsushita built organisational philosophies around empowerment, teamwork and collaboration. They went from some of the worst performing businesses in the country to the strongest. By the late 1970’s world governments were looking to these emerging giants to understand what made them so successful, and in particular, so resilient. It was at this point in 1979 that Matsushita-san delivered his famous speech to western industrial leaders keen to learn the secrets of Japan’s success. It was not until 1991 that the world began to understand the power of the management systems that had been developed in Japan. A book2 was published following a study by MIT’s automotive industry research programme. The book studied the history of the automotive industry and the rise and rise of the mighty Toyota Motor Corporation; the term used to describe Toyota’s secret sauce? Lean Thinking. What Konosuke Matsushita, Toyota’s Taichi Ohno and their contemporaries understood was that the key to the success of their businesses didn’t lie in their tools, techniques, or processes, but was the result of the management philosophies that underpinned their corporations. They thought about their businesses as socio-technical systems and because of this created organisations that encouraged the right behaviours throughout. So, what does this have to do with IT? Firstly we have to recognise that IT is failing. Standish Group estimate that $85B to $145B is spent every year on failed and cancelled IT projects, and that 60% to 70% of all projects either fail outright or are considered troubled (time, cost, scope issues)3. This is a repeated pattern; we can change the country, the industry, the people and the business; the data shows a similar pattern – the problem is systemic. To solve this kind of systemic problem we need to investigate the system more closely and understand the ‘games’ that are being played out within our IT divisions. But what are the components of our IT ecosystem? When we lift the hood, there are a few areas of focus for us to investigate: people, structure, process, culture and technology. The first thing that we may notice about our corporate IT divisions is how little they differ to the Taylorised models Henry Ford and Alfred Sloan built their businesses around 100 years ago. They are structured around functional silo’s, management philosophies are command and control and empowerment is just a word that appears on corporate mouse mats. They are structured for an industrial paradigm in an information age. To make matters worse, most IT managers aspire to create self-managed teams, high levels of collaboration, innovation and continuous improvement. Many have little appreciation that the management models that they enforce, often the only ones that they know, are the very things that are preventing them from achieving the results that they dream of. So how do we change things? Firstly, it’s important to understand the differences in the two management philosophies, as the contrast is stark. For example, where Taylor’s ‘scientific management’ teaches us that managers should manage people, systems management theory teaches us that managers should manage processes. ‘Scientific management’ advocates for maximising the utilisation of our people and machinery, ‘systems management theory’ teaches us to ruthlessly eliminate waste. Although the transition is anything but easy, the results, at least so far, appear impressive. At a recent conference, Jeff Smith, CIO of Suncorp’s 2000 person IT division, estimated that they had increased throughput by over 40% whilst at the same time reducing net operating costs by over 20%4. Similarly, the BBC’s David Joyce announced in a recent article that they had reduced the time taken to engineer a software feature by over 50%5. These organisations are re-engineering many of the components of the IT taxonomy. By taking Agile software principles and introducing statistical control and scheduling techniques from Lean Thinking, teams are radically improving the efficiency and throughput of software delivery processes. They aren’t stopping here though. Product development is being refined to ensure the teams aren’t ‘building the wrong things at speeds previously thought impossible’. Planning and governance processes are being simplified to support responsiveness and adaptability in the business. Even the organisational structure itself isn’t sacred, with some of the more progressive IT divisions moving away from a top-down, hierarchical design, towards a systems based, bottom-up model, removing organisational silo’s to increase collaboration and introduce a stronger customer focus. The real change for these organisations isn’t in the structure, processes or tools of course, but in something much more subtle and complex: the way they think. Changing 100 years of western management thinking is not a simple task but industrial models just don’t cut it in an information age. Deming taught us that the processes and structures we create as leaders always produce exactly the results they are designed to produce; the system always works perfectly. In IT we have created approaches that fail (or have difficulties) 60% to 70% of the time. It’s our responsibility as leaders to change the system. This leads to the title of the article. The most common excuse I hear for avoiding change and improvement in IT leadership is that we’re not Japanese and we don’t build cars. I hear this excuse every day and it misses the point. Lean Thinking, and the management paradigm that underpins it, Systems Management Theory, focus on changing the role of leadership; it knows no national or industrial bounds, and this has been proven time again over the last 30 years, from manufacturing to healthcare. IT leadership is once again lagging behind the management curve. To re-enforce the point even further, a recent article in the Harvard Business Review6 asked some of the worlds leading academic and industrial business thinkers for the big ticket changes required in western management thinking over the next 10 years. Retraining managerial minds in systems thinking appeared in it’s top 25. Also in there was reducing the pull of the past, eliminating the pathologies of formal hierarchy and reconstructing management’s philosophical foundations. This is nothing new – management science has pointed towards collaboration, teamwork and trust for over 30 years. But mouthing the words is easy; Systems Management Theory gives us the tools to go execute. Introducing this paradigm shift, although not an easy journey, has certainly been proven achievable. Agile software development methods, based on the Toyota Production System, help us to quickly introduce Lean concepts to our software development operations. Statistical control techniques can help us improve and refine them. Recently, Lean concepts have helped scale these working-level techniques to the enterprise by borrowing philosophies, tools and techniques to solve historic problems with structure, budgeting and governance in top-down, command and control organisations. And middle management are proving willing to change, and even lead the charge, given the right leadership, support and opportunities. Driving change is hard. I often compare being a CIO to the job of a grounds keeper in a cemetery; there are a lot of people underneath you but no-one is listening. Of course, we don’t have to strive to improve the problem; I’ll leave the last word to Deming himself… “It’s not necessary to change. Survival is not mandatory.”. 1. The Principles of Scientific Management: Frederick Winslow Taylor. 2. The Machine That Changed The World: Womack, Jones, Roos. 3. Standish Chaos Reports 2000 to 2009. 4. Agile Australia (http://www.agileaustralia.com/video.html) 5. David Joyce (http://leanandkanban.wordpress.com) 6. Harvard Business Review, February 2009: Moonshots for Managers

My goal is to compare Sonatype Nexus and JFrog Artifactory,the two leading open source Maven repository managers.

This is the second part of a two-part article introducing Jeff De Luca’s Feature Driven Development (FDD) process. In particular, we are looking at how FDD differs from Scrum and eXtreme Programming-inspired approaches when it comes to working with larger teams and projects. In the first part we briefly introduced the ‘just enough’ upfront activities that FDD uses to support the additional communication that inevitably is needed in a larger project/team. In the second part of the article we cover how FDD leverages the results of those upfront activities within the highly iterative, self-managing, organized-chaos that is the delivery engine room of an FDD project. The Engine Room: Delivering Frequent, Tangible Working Results Once there is an initial overall model (FDD Process #1), an initial overall features list (FDD Process #2), and an initial overall plan (FDD Process #3) in place, an FDD project is ready to start delivering the required software feature by feature. Peter Coad, the Chief Architect on the original FDD project used the phrase ‘Deliver frequent, tangible, working results’ as a mantra to impress upon people the idea of delivering real, completed, client-valued function as often as possible. Scrum and eXtreme Programming do this using fixed length iterations of a calendar month or 2-4 weeks. FDD is different. Each Chief Programmer (lead developer) runs a series of iterations, each of which is normally a matter of a few days, and never longer than two weeks. At the start of each of these iterations, each Chief Programmer selects the next few features that make sense to implement from the backlog of feature sets (activities) that were assigned to him or her in FDD Process #2. The Chief Programmer leads the development of these features through FDD processes #4 and #5, Design by Feature (DBF) and Build By Feature (BBF). Note that iterations through the DBF/BBF processes are not fixed length, and Chief Programmers do not synchronize the start and end of their iterations with each other. In addition, the DBF/BBF processes are always executed as a pair (FDD describes them as two separate processes rather than one combined process for psychological reasons). FDD Process #4: Design By Feature After selecting the features for the iteration, a Chief Programmer needs to form their feature team. Yes, feature teams are formed and disbanded for each iteration through the DBF/BBF process pair. Using the knowledge gained from the modeling process (FDD Process #1), the Chief Programmer identifies the domain classes that are likely to be involved in this iteration, and forms his or her feature team from the owners of those classes. In practice, this means: a feature team is small, typically 3 to 5 people, because features are small. By definition, a feature team comprises of all the class owners who need to modify their classes in the development of the features during that iteration. There is no need to wait for members of other teams to change code. Therefore, there are all the benefits of code ownership and a sense of collective ownership too. Class owners may find themselves a member of multiple feature teams at the same time. This does not happen as frequently as might be supposed because iterations are so short – days not weeks. When it does, it is not a big problem in practice. Chief Programmers work together to resolve any problematic conflicts and, with care, most developers can manage the demands of occasionally belonging to more than one feature team for a short time. Once formed, the Chief Programmer facilitates the collaborative analysis and design of the features for that iteration. Depending on the complexity, this may involve the team walking through the requirements in detail with a domain expert, and studying any existing relevant documents. It also involves agreeing on the interactions and other details that need to be added to the model to support the new features. The final step in the DBF part of the iteration is to review the design. For simple features, this may be a brief sanity check of the design held within the feature team. For more significant features, the Chief Programmer will typically involve other Chief Programmers or class owners so that they are aware and can comment on the impact of the proposed design. For small team projects, the object models are frequently small enough for individual or pairs of developers to create good designs while writing tests for a particular feature or user story. For larger projects, this is not necessarily the case and designs created purely by considering the tests a feature or user story must pass are more likely to be brittle and require significant refactoring. The DBF process in FDD ensures that the overall model also guides the design, helping to maintain its ‘conceptual integrity’ [Brooks]. FDD Process #5: Build By Feature The Build by Feature (BBF) part of the iteration involves the team members coding up the features, testing them at both unit level and feature level, and holding a code inspection before promoting the completed features into the project's regular build process. Testing FDD expects developers to unit test their code. It expects feature teams to test their features. FDD is not overly concerned with how this is achieved. Projects and feature teams are free to adopt the testing tools, frameworks, and level of formality and completeness that are most appropriate. FDD does not mind if tests are written before or after code. What FDD mandates, is that the feature team deliver code that has been appropriately tested and inspected. Only once the new features have passed testing and inspection is the source code allowed into the build process. Code Inspections Most people want to know why FDD mandates code inspections, especially those that have endured sitting through hours of boring, unproductive, ego-polishing/demolishing, point-scoring sessions that formed so-called code reviews, inspections or walkthroughs. The reason FDD mandates code inspections is that research has shown time and again that when done well, inspections find more defects and different kinds of defects than testing [McConnell]. Not only that but by examining the code of the more experienced, knowledgeable developers on the team and having them explain the idioms they use, less experienced developers learn better coding techniques. In addition, knowing that their code will be inspected and not be allowed in the build unless it conforms to the agreed standards encourages developers to pay more attention to conforming to those standards. One of the benefits of working in feature teams is that the whole feature team is on the hot seat during an inspection, not just one individual. This removes much of the intensity and anxiety inherent in inspecting one individuals work. The Chief Programmer decides on the level of formality of each inspection depending on the complexity and impact of the features developed in that iteration. Where the code has little or no impact outside the feature team, an inspection will usually only involve the feature team inspecting each other’s work. Where there is significant impact the Chief Programmer pulls in other Chief Programmers and developers to both verify the code and communicate the impact of the new features. eXtreme Programming acknowledges inspections as a ‘best practice' but promotes pair programming as the logical conclusion of applying this practice. Pair programming is obviously better than individual developers delivering code without any form of inspection. However, while FDD neither mandates nor forbids pair programming, a more-traditional inspection is: fresh eyes looking at the code, catching bad assumptions made by the coder/s a Chief Programmer present to ensure the techniques passed on are good. After all, developers can just as easily teach each other bad habits as well as good habits. a change of pace for developers, a chance to step away from the keyboard and mouse for a short while. With the wide availability of automated source code formatting and static analysis tools, code inspections can now be shorter, concentrating on the logic and coding idioms involved and not getting bogged down in nit-picking such as alignment of braces, etc. The Build FDD assumes some sort of regular build process. Some teams build weekly, others daily and others continuously. FDD avoids mandating any particular build regime. This enables the project team to apply the most applicable. If a continuous integration environment makes sense, then the team is free to employ the best there is. Progress Reports Agile projects like highly visible progress information. FDD projects are no exception. In fact, because larger projects frequently have higher profiles within an organization, presenting meaningful, accurate, timely project information appropriately at the different levels of leadership/management is even more important. Conventionally, FDD projects track the development of each feature through its DBF/BBF iteration against six milestones: domain walkthrough, design, design inspection, coding, testing and inspection, and promoted to build. For each feature, Chief Programmers record the actual date a milestone is reached. Tracking each feature through these six milestones enables the project to keep an eye on how much work is 'in progress'. Too many features at a particular milestone indicate a process problem. Those promoting Kanban and other Limited Work In Progress methods have formalized this idea to strictly define what is meant by 'too many' for each of their development iteration milestones/statuses. They then refuse to move an item to a new milestone/status if the limit on the number of items at that status has been reached. This forces a team to keep items moving forward through the process [Kanban]. FDD is not so formal, leaving the Chief Programmers and Development Manager to keep an eye informally on the amount of work in progress. The Big Wallchart, Burn-Down/Up Charts, Etc For general visibility of progress within a project, the team typically lists all the features in the project complete with their owning Chief Programmer, feature team members, and the dates of each milestone achieved on a suitable wall. In addition, features can be colored to show if they are started, in-progress, completed or blocked. This allows people to stand back from the wall and get a good visual feel for the overall status of the project. They can then walk up to the wall to zoom in on particular areas and activities in more detail. Recording the date each milestone is achieved enables a team to produce burn-down or burn-up charts analogous to those produced in Scrum and XP. Chief Programmers and Project Managers can determine from these if the underlying rate of feature completion is increasing, decreasing, or stable, etc. One of the best ways to achieve this is to have the Chief Programmers regularly (typically once a week) communicate progress to either the project manager or someone dedicated to the task. That person then produces whatever roll-up and burn-down charts desired. Having an administrative person, the equivalent of the Tracker role in eXtreme Programming, perform these report formatting duties frees the Chief Programmers to spend more time on making progress rather than formatting reports about it. Parking Lot Charts For reporting to senior management, the level of individual features is often too granular. Here, FDD projects typically use a graphical report format that known as the Parking Lot chart. In a Parking Lot chart, each group of ‘parking lots’ represents one of the subject areas from the features list. Each parking lot represents one of the activities within that subject area, and displays the name of that set of features, the number of features within it, and the percentage of those features that have been completed (typically both in text and using a progress bar). The parking lots are also colored to indicate whether the features in that activity have been started, completed, or have significant blockages. The FDD parking lot format has become so popular that Mike Cohn included it in his book, Agile Planning and Estimating [Cohn]. (click for larger image) Figure 1: Example Parking Lot Chart Conclusion Feature-Driven Development combines the key advantages of other popular agile approaches with model-centric techniques and other best practices that scale to much larger teams and projects. It defines three upfront activities that provide a conceptual and management framework within which a larger-than-usual agile team can add functionality to the software, feature by feature. It is also just as applicable for smaller teams tackling non-trivial problem domains where it is worth spending just a little time to sketch a map of the journey before dashing off down the agile coding highway. Even if you and your team decide not to adopt FDD as a whole, understanding why FDD is the way it is, can provide insight into scaling traditional agile approaches beyond small, largely independent teams. Finally, I would like to say thank you to Serguei Khramtchenko and Mark Lesk at Nebulon for their corrections and suggestions incorporated in this article. References [Brooks] Frederick P. Brooks, Jr., The Mythical Man-Month, Addison Wesley [Cohn] Cohn, Agile Planning and Estimating, Prentice-Hall PTR [FDD] FDD Community Site, www.featuredrivendevelopment.com/ [Kanban] The home of Kanban software development, www.limitedwipsociety.org/ [McConnell] McConnell, Code Complete, Microsoft [Nebulon] The Latest FDD Processes available from www.nebulon.com/articles/fdd/latestprocesses.html [Palmer-1] Palmer, Felsing, A Practical Guide to Feature-Driven Development, Prentice Hall PTR

One nice feature in JUnit 4 is that of Parameterized Tests, which let you do data-driven testing in JUnit with a minimum of fuss. It's easy enough, and very useful, to set up basic data-driven tests by defining your test data directly in your Java class. But what if you want to get your test data from somewhere else? In this article, we look at how to obtain test data from an Excel spreadsheet. Parameterized tests allow data-driven tests in JUnit. That is, rather than having different of test cases that explore various aspects of your class's (or your application's) behavior, you define sets of input parameters and expected results, and test how your application (or, more often, one particular component) behaves. Data-driven tests are great for applications involving calculations, for testing ranges, boundary conditions and corner cases. In JUnit, a typical parameterized test might look like this: @RunWith(Parameterized.class) public class PremiumTweetsServiceTest { private int numberOfTweets; private double expectedFee; @Parameters public static Collection data() { return Arrays.asList(new Object[][] { { 0, 0.00 }, { 50, 5.00 }, { 99, 9.90 }, { 100, 10.00 }, { 101, 10.08 }, { 200, 18}, { 499, 41.92 }, { 500, 42 }, { 501, 42.05 }, { 1000, 67 }, { 10000, 517 }, }); } public PremiumTweetsServiceTest(int numberOfTweets, double expectedFee) { super(); this.numberOfTweets = numberOfTweets; this.expectedFee = expectedFee; } @Test public void shouldCalculateCorrectFee() { PremiumTweetsService premiumTweetsService = new PremiumTweetsService(); double calculatedFees = premiumTweetsService.calculateFeesDue(numberOfTweets); assertThat(calculatedFees, is(expectedFee)); } } The test class has member variables that correspond to input values (numberOfTweets) and expected results (expectedFee). The @RunWith(Parameterzed.class) annotation gets JUnit to inject your test data into instances of your test class, via the constructor. The test data is provided by a method with the @Parameters annotation. This method needs to return a collection of arrays, but beyond that you can implement it however you want. In the above example, we just create an embedded array in the Java code. However, you can also get it from other sources. To illustrate this point, I wrote a simple class that reads in an Excel spreadsheet and provides the data in it in this form: @RunWith(Parameterized.class) public class DataDrivenTestsWithSpreadsheetTest { private double a; private double b; private double aTimesB; @Parameters public static Collection spreadsheetData() throws IOException { InputStream spreadsheet = new FileInputStream("src/test/resources/aTimesB.xls"); return new SpreadsheetData(spreadsheet).getData(); } public DataDrivenTestsWithSpreadsheetTest(double a, double b, double aTimesB) { super(); this.a = a; this.b = b; this.aTimesB = aTimesB; } @Test public void shouldCalculateATimesB() { double calculatedValue = a * b; assertThat(calculatedValue, is(aTimesB)); } } The Excel spreadsheet contains multiplication tables in three columns: The SpreadsheetData class uses the Apache POI project to load data from an Excel spreadsheet and transform it into a list of Object arrays compatible with the @Parameters annotation. I've placed the source code, complete with unit-test examples on BitBucket. For the curious, the SpreadsheetData class is shown here: public class SpreadsheetData { private transient Collection data = null; public SpreadsheetData(final InputStream excelInputStream) throws IOException { this.data = loadFromSpreadsheet(excelInputStream); } public Collection getData() { return data; } private Collection loadFromSpreadsheet(final InputStream excelFile) throws IOException { HSSFWorkbook workbook = new HSSFWorkbook(excelFile); data = new ArrayList(); Sheet sheet = workbook.getSheetAt(0); int numberOfColumns = countNonEmptyColumns(sheet); List rows = new ArrayList(); List rowData = new ArrayList(); for (Row row : sheet) { if (isEmpty(row)) { break; } else { rowData.clear(); for (int column = 0; column < numberOfColumns; column++) { Cell cell = row.getCell(column); rowData.add(objectFrom(workbook, cell)); } rows.add(rowData.toArray()); } } return rows; } private boolean isEmpty(final Row row) { Cell firstCell = row.getCell(0); boolean rowIsEmpty = (firstCell == null) || (firstCell.getCellType() == Cell.CELL_TYPE_BLANK); return rowIsEmpty; } /** * Count the number of columns, using the number of non-empty cells in the * first row. */ private int countNonEmptyColumns(final Sheet sheet) { Row firstRow = sheet.getRow(0); return firstEmptyCellPosition(firstRow); } private int firstEmptyCellPosition(final Row cells) { int columnCount = 0; for (Cell cell : cells) { if (cell.getCellType() == Cell.CELL_TYPE_BLANK) { break; } columnCount++; } return columnCount; } private Object objectFrom(final HSSFWorkbook workbook, final Cell cell) { Object cellValue = null; if (cell.getCellType() == Cell.CELL_TYPE_STRING) { cellValue = cell.getRichStringCellValue().getString(); } else if (cell.getCellType() == Cell.CELL_TYPE_NUMERIC) { cellValue = getNumericCellValue(cell); } else if (cell.getCellType() == Cell.CELL_TYPE_BOOLEAN) { cellValue = cell.getBooleanCellValue(); } else if (cell.getCellType() ==Cell.CELL_TYPE_FORMULA) { cellValue = evaluateCellFormula(workbook, cell); } return cellValue; } private Object getNumericCellValue(final Cell cell) { Object cellValue; if (DateUtil.isCellDateFormatted(cell)) { cellValue = new Date(cell.getDateCellValue().getTime()); } else { cellValue = cell.getNumericCellValue(); } return cellValue; } private Object evaluateCellFormula(final HSSFWorkbook workbook, final Cell cell) { FormulaEvaluator evaluator = workbook.getCreationHelper() .createFormulaEvaluator(); CellValue cellValue = evaluator.evaluate(cell); Object result = null; if (cellValue.getCellType() == Cell.CELL_TYPE_BOOLEAN) { result = cellValue.getBooleanValue(); } else if (cellValue.getCellType() == Cell.CELL_TYPE_NUMERIC) { result = cellValue.getNumberValue(); } else if (cellValue.getCellType() == Cell.CELL_TYPE_STRING) { result = cellValue.getStringValue(); } return result; } } Data-driven testing is a great way to test calculation-based applications more thoroughly. In a real-world application, this Excel spreadsheet could be provided by the client or the end-user with the business logic encoded within the spreadsheet. (The POI library handles numerical calculations just fine, though it seems to have a bit of trouble with calculations using dates). In this scenario, the Excel spreadsheet becomes part of your acceptance tests, and helps to define your requirements, allows effective test-driven development of the code itself, and also acts as part of your acceptance tests. From http://weblogs.java.net/blog/johnsmart

Feature-Driven Development (FDD) is one of the agile processes not talked or written about very much. Often mentioned in passing in agile software development books and forums, few actually know much about it. However, if you need to apply agile to larger projects and teams, it is worthwhile taking the time to understand FDD a little more The natural habitat of Scrum and XP-inspired approaches is a small team of skilled and disciplined developers. It remains a significant challenge to scale these approaches to larger projects and larger teams. Some have been successful but many have struggled. Feature-Driven Development (FDD) invented by Jeff De Luca is different. While just as applicable for small teams, Jeff designed FDD from the ground up to work for a larger team. Larger teams present different challenges. For example, a small team of disciplined and highly skilled developers by definition is likely to succeed regardless of which agile method they use. In contrast, it is unrealistic to expect that everyone in a larger team is equally skilled and disciplined. For this and other reasons, FDD makes different choices to Scrum and XP in a number of areas. In the first part of this two-part article, we briefly introduce the ‘just enough’ upfront activities that FDD uses to support the additional communication that inevitably is needed in a larger project/team. In the second part of the article, we cover how the highly iterative delivery part of FDD differs from Scrum and XP-inspired approaches. Iteration Zero:Getting Set to Deliver Most experienced agile teams are familiar with the concept of an iteration zero, a relatively short period for a team to put in place what they need to start delivering client-valued functionality in subsequent iterations. Despite general acceptance within the agile community that some form of iteration zero is a pragmatic necessity on most projects, neither Scrum nor eXtreme Programming formally have much to say about it. In contrast, an FDD project is organized around five 'processes', of which the first three can be considered roughly the equivalent of iteration zero activities. FDD does not use the term, iteration zero. It calls these three ‘processes’ initial project-wide activities. Each of the FDD processes is described so that it can be printed, in a typical-sized font, on no more than two sides of letter-sized paper. The most recent versions of the FDD processes are available from the FDD section of the Nebulon website, but very briefly an FDD project: … starts with the creation of a domain object model in collaboration with Domain Experts. Usinginformation from the modeling activity, and from any other requirements activities that have taken place, the developers go onto create a features list. Then a rough plan is drawn up and responsibilities assigned. Now we are ready to repeatedly take small groups of features through a design and build iteration that lasts no longer than two weeks and is often much shorter, sometimes only a matter of hours...[Palmer-1] FDD Process #1: Develop an Overall Model For many who have escaped from the perils of large, upfront analysis and design phases to the freedom and discipline of Scrum and eXtreme Programming-inspired approaches, the idea of developing a domain object model at the start of a project is controversial. In FDD, however, the building of an object model is not a long, drawn-out, activity performed by an elite few using expensive CASE tools. The modelers do not format the resulting model into a large document and throw it over the wall for developers to implement. Instead, building an initial object model in FDD is an intense, highly iterative, collaborative and generally enjoyable activity involving ‘domain and development members under the guidance of an experienced object modeler in the role of Chief Architect' [Nebulon]. FDD Process #1 describes the tasks and quality checks for executing this work, and while not mandatory, the object model is typically built using Peter Coad's modeling in color technique (modeling in color needs an introductory article all of its own [Palmer-2]). The idea is for both domain and development members of the team to gain a good, shared understanding of the problem domain. It is important that everyone understands the key problem domain concepts, relationships, and interactions. In doing so, the team as a whole learn to communicate with each other and start to establish a shared vocabulary, what Eric Evans calls a Ubiquitous Language [Evans]. The object model developed at this point concentrates on breadth rather than depth; depth is added iteratively through the lifetime of the project. The model is, therefore, a living artifact. Throughout the project, the model becomes the primary vehicle around which the team discusses, challenges, and clarifies requirements. FDD Process #2: Build a Features List With the first activity being to build an object model, some may conclude FDD is a model-driven process. It is not. While the model is central to the process, an FDD project is like a Scrum or eXtreme Programming project in being requirement-driven. Small, client-valued requirements referred to as features drive the project; the model merely helps guide. Formally, FDD defines a feature as a small, client-valued function expressed in the form:

When I was trying to search for "leaders vs. rulers" on Google, I found many references to governments, royalty, and the military, throughout history. But the strange thing is that none of the articles seemed to distinguish between leaders and rulers. As if leaders and rulers are the same kind of people. They are not. Leaders Last week I was reading the book Tribes, by Seth Godin. In his book Seth says that never in history has it been so easy for anyone to be a leader. These days, with the use of social media, each of us is able to attract our own followers. And on Twitter, this is exactly what we're doing (quite literally). Seth explains that a crowd becomes a tribe when it has a leader that the people are following out of their own free will. And the interesting thing is that people can follow different leaders for different causes. In software projects it is the same. Some people can take the lead on an architectural level, while some have the lead on a functional level. Still others may be the first ones to turn to when people need advice about tools or processes. A complex system does not need a single leader. In fact, I believe a cross-functional team functions best when it has multiple leaders, each with his own area(s) of interest. Rulers In social systems the rulers are of an entirely different breed. While leaders use the power of attraction to convince people what to do, rulers use the power of authority to tell people what to do. Ruling people's lives is the very purpose of the ruler's job. With ruling comes law-making, enforcement and sanctioning, also called the trias politica (legislature, executive, judiciary). Unfortunately, rulers have gotten a bit of a bad name over the centuries. (Much of it deserved, by the way.) But ruling isn't all that bad. Laws, enforcement and sanctions are necessary evils, and in many social systems rulers can peacefully co-exist with leaders. For example: in any football (or soccer) match you will find leaders (one in each team) and rulers (the referees). They all play their parts in making the game work for everyone. Are managers rulers? There's no doubt in my mind that managers are rulers. They are (usually) the only ones with the authority to hire and fire people, and to place them in (or remove them from) teams or departments. They are able to tell people what software to use, what clothes to wear, and how much to pay for a place at the parking lot. Are managers leaders? This is a more interesting question. Lots of management book have been trying hard to turn managers into leaders. The last one I read was Good to Great, by Jim Collins. In his book Jim listed a 5-level hierarchy: Level 5 Executive: Builds enduring greatness through a paradoxical blend of personal humility and professional will. Level 4 Effective Leader: Catalyzes commitment to and vigorous pursuit of a clear and compelling vision, stimulating higher performance standards. Level 3 Manager: Organizes people and resources toward the effective and efficient pursuit of pre-determined objectives. Level 2 Contributing Team Member: Contributes individual capabilities to the achievement of group objectives and works effectively with others in a group setting. Level 1 Highly Capable Individual: Makes productive contributions through talent, knowledge, skills, and good work habits. The problem I have with Jim's hierarchy is that it suggests a linear progression to "higher" levels (where a leader is on a "higher" level than a manager). This doesn't fit with my observations of how social networks operate. In a software project, or any other social network, there can be many leaders, each with his or her own goals and desires. Some are taking initiatives for better architectures, some are leading the way to better user interface design, and some are guiding their followers towards better customer service, better processes, better software tools, or better coffee. To be a leader is not the next step for managers It is the manager's job to give room to leaders There are thousands of leaders on Twitter, and they all have their own huge numbers of followers. But who are the managers of Twitter? Only Evan Williams, Biz Stone and Jack Dorsey are. It's their platform. It's their game. They are the referees, making the laws, enforcing them, and sanctioning, while thousands of leaders and tribes are running around trying to score. Sure, it's ok when managers are trying to be leaders. Nothing wrong with that. Evan, Biz and Jack have a large number of followers themselves too. But they don't have the largest tribes. Managers are on top of things, but they are not on top. Rulers don't need to have the largest tribes themselves. Being a great ruler is hard enough already. If you think you need to be a great leader too, you're just making it hard for yourself. Referees contribute to great football/soccer games by being great rulers. They don't attempt to lead. It's not their job. They are in charge, but they are not the ones with the biggest egos. In his presentation Step Back from Chaos Jonathan Whitty shows that managers are often not the hubs in a social network. It's the informal leaders in a network through which most of the communication flows. It's the managers' job to make sure that leadership is cultivated, and that the emerging leaders are following the rules. So, you can be a leader, or you can be a ruler. And if you're exceptionally talented, perhaps you can be both. Which one will you be?

Agile Client is a NetBeans Platform application developed by Northrop Grumman in partnership with the Defense Information System Agency (DISA). It brings the war fighter a 3-D common operational picture (COP) workstation designed for greater efficiency and mission effectiveness. This efficiency is empowered by its ability to be installed and upgraded on demand. Its mission effectiveness is permitted by the ability to tailor the user’s application with only the capabilities and data they need for their specific mission. The interview below is with Charlie Black, who runs the team using the NetBeans Platform at Northrop Grumman. Hi Charlie, who are you and what do you do? My name is Charlie Black and I have worked for Northrop Grumman since 1998. Over that time I have been working as a software engineer on a Global Command and Control System (GCCS) program supporting data fusion, dissemination and display technologies. I am currently running a team that uses the NetBeans Platform for a product called Agile Client, which is now part of GCCS. Team: Greg Gibsen, Charlie Black, Brice Bingman and James Maloney What are the technical specifics of Agile Client? The Agile Client uses several commercial off the shelf (COTS) products that make up its core feature set: It uses the NetBeans Platform, which provides the services common to almost all large desktop applications: window management, menus, settings and storage, an update manager, and file access. For map and visual rendering, Agile Client uses NASA’s World Wind product, enabling the retrieval of geospatial data via open standards that are embraced by the international community. For providing data services, Agile Client uses Gemstone GemFire, which provides ultra low-latency distributed caching with high availability and no data loss. Here's a screenshot of Agile Client in action: Agile Client is also integrated with GCCS COP Collaboration. This allows one or more war fighters to work simultaneously on a single map. They can collaboratively create GCCS Overlays, share files and communicate by instant messaging and Voice over IP. This is all done with the Extensible Messaging and Presence Protocol (XMPP), which is a set of open XML technologies for presence and real-time communication. What are the two or three things that you are happiest about, relating to this application? Since the NetBeans Platform is based on standard Java UI technologies, we are able to integrate existing Java windows directly, without any issues. This has accelerated our development substantially. Using Sun UI guidelines and reusable components in any new development, we have actually made an application that is embraced by our customers as looking modern. Underneath this application is the NetBeans Platform. Why? The main reason for going with a rich client platform such as the NetBeans Platform was the module system. It will help with our deployment of a large scale application on an enterprise level. I can envision a future where there will be hundreds of modules deployed in the application. Then, using the update center, we can keep those applications up to date. How did you find out about it? When? Why did you start using it? We have known about NetBeans for a long time. However it was used mainly as a developer tool. It wasn’t until NetBeans 6.0 that we started using it as our platform for basing new work on. What are the main things you gained from the NetBeans Platform? Using the NetBeans Platform, we decreased our time to develop our application by using existing window components. Also, the automatic updates are integral to our final application. In the past, it would have taken months or even years to get a patch to our end users. That said, the most significant gain has been the community! Which APIs have you used? Which ones are your favorites and why? We actually use several API sets, with just about every one enabled. The Windows API is incredible, as that is what the end user sees that allows them to tailor their display. With the customization of toolbars, the end-user can make their own ad-hoc workflow. The other API that we have been impressed with is the Nodes API, providing a view to our layers on the 3D globe. What could be improved about the NetBeans Platform? In our community OSGi is a big item, so if NetBeans could formally handle them for module deployment that would be a monumental win. Imagine a module of business intelligence that can be deployed in Glassfish and in the NetBeans Platform. I realize that some work has already been done in this area, but to see it on an official roadmap would be remarkable. After that, the Properties window could use something to make it more appealing. We are thinking of dropping our own Properties window in favor of the NetBeans implementation. Overall, we have been really happy with the NetBeans Platform and the features it provides. Do you have some tips and tricks for a complete newbie, who is getting started with the NetBeans Platform? When new team members join, I customarily distribute to them a book on the NetBeans Platform. For further aid, I point them to the NetBeans Developer Wiki. There is an amazing amount of information available, detailing not only what the NetBeans Platform can do for you, but what can you do with the NetBeans Platform as well. If more assistance is needed, I recommend just asking the community. Someone can usually help in answering any level of questions. How have customers responded to this application and what are your plans for its future? Our customers are very pleased with the application and they are moving forward with plans to base future work on the Agile Client. Internally, we have made Agile Client with the goal of releasing portions back to the community. However, before that can be done, we have to figure out what that means from our customer’s standpoint.

The concept of a software factory describes a practical work-product approach to governing an efficient service factory - a software engineering-based approach to defining, developing, testing, deploying, and operating functional services and automated business processes. All services follow a similar lifecycle of analysis, followed by design, development, deployment, and ongoing management. Because the service creation process is repetitive, a production engineering approach to automating software development can be used. The Production Engineering method required a significant effort up front, creating a specialized production or assembly line that can then mass-produce the product efficiently and in quantity In effect, we are building a Services Factory: much of the purpose of SOA governance is to define how that factory can operate most effectively. In the following excerpt from the book "SOA Governance: Achieving and Sustaining Business and IT Agility" [REF-1] we will take a look specifically at service development lifecycle control points. This article authored by Clive Gee and Robert Laird and published in SOA Magazine on Feb 23, 2009 Introduction While most organizations have some form of a system development lifecycle (SDLC), the nature of creating shared services is best guided by an SDLC with sufficient governance control points to ensure quality of service. This article discusses and explains the key concepts of a governance control point, as applied specifically to the service development lifecycle. Service Development Lifecycle Control Points Most organizations already have some type of system development lifecycle (SDLC) and a methodology that is used to perform development, although we often see in practice a lack of enforcement of that approach across different business units, and even if a set of best practices, standards, policies, and patterns has been defined, they are not always enforced. Effectively enforcing best practices and a consistent SDLC provides a reasonable entry point for real governance, while not being a huge stretch from what is already being performed via the SDLC. At the same time, if the governance maturity level of the organization can be increased to the degree that it is able to govern the SDLC, the organization is then in a much better position to proceed to the next phase of the SOA governance cycle and create program and organization governance. The danger here for even initial attempts at SOA governance is that often some key individuals view the imposition of any process or governance as being something that might apply to other people but not to them personally. For them, it's an over-engineered, useless exercise that just gets in the way of meeting their own deadlines. So, many governance processes are simply bypassed, or they're followed in a less than an enthusiastic manner. The main reason for this is that governance is imposed from the outside and the execution is onerous. What would happen if governance were mostly automated, easy, and added value to the development process and actually helped with project deadlines? Would the skeptics be more willing to take the medicine if it genuinely eased their pain? To adequately govern the SDLC, there is a need to establish measurements, policy, standards, and control mechanisms to enable people to carry out their governance roles and responsibilities as efficiently as possible, without introducing overly bureaucratic procedures. Governance of the SDLC may be characterized by the sorts of decisions that need to be made at certain "control points" within the process of services development. A control point is a decision checkpoint that provides an opportunity to measure adherence to the established processes, whether you are on track to meet the targets and goals you have established, and then decide whether the way the processes are executed or managed needs adjusting. Knowing what decisions involved in the process are critical, when to make them, and understanding what measurements are needed to monitor those processes are all essential aspects of governance. Certain activities within a process may be associated with a control point. At the end of each identified activity, there is a control point at which the governance function decides whether the program is ready to move to the next activity. Each of these milestones is a control point. At its essence, the governance of the SDLC provides a way to identify control points and to define the governance rules. At each control point, it is necessary to identify the following: • The roles for who does what at the control point • The policies to be applied at the control point • Measurements at each control point that should be applied and collected for later governance vitality actions • The proof of compliance records to be created and archived A control point will be created where there is a demonstrated advantage weighing the standardization and efficiency provided versus the time, effort, and possible project delay. The control point enables SOA governance the opportunity to ascertain progress, to communicate this progress, to forecast efforts for subsequent phases of the SDLC based on scope and issues found, to review and report compliance, and to facilitate the injection of expertise and qualified review of the artifacts, process, or decisions made by the development team. Control points don't have to consist of huge formal meetings. Services and most automated business processes are smaller entities than projects, and there are many more of them. Therefore, the existing governance approach has to be streamlined or it might grind to a halt. We've found in practice that effective control point reviews can be made during regular - typically weekly - sessions of a subset of the SOA enablement. A real productivity aid in performing these control point reviews is the use of previously completed checklists, signed off by one or more senior professionals as certification that one or more tasks has been completed successfully, and that the service, process, or other work product is fit for purpose and ready in all respects for the next task in the development process. These checklists should be viewed as contracts between different experts in the service development process. The most important part of the checklist is the signature block to show who exercised approval authority; people tend to be careful about the quality of anything that carries their personal reputation with it. Another productivity aid is the use of automated tooling. As much of the governance control point as possible should be automated. This aids in better near real-time feedback to the developers and provides an easy method to recheck work that has been updated. In addition, human beings are busy and will tend to apply governance in an inconsistent manner. Machines are consistent but not usually as flexible as needed. The combination of the two provides an optimal governance mix. Let's look at the control points needed to govern a generic development lifecycle, at least at a high level. Figure 1 represents a "governance dashboard" monitoring a typical SDLC with an eye toward the key concepts and the points where they must be addressed by SOA governance. Figure 1: Software Development Lifecycle Governance Dashboard As mentioned previously, we need a streamlined process that can handle the large number of services and automated processes that we need to implement to have real impact on business agility and flexibility. However, that streamlined process must not sacrifice the quality of governance just because of the need for extra speed. That would be an unacceptable trade-off. Some organizations deal with highly regulated processes that have mission-critical or life-critical products and need to apply highly formal, auditable governance to manage the risks involved. Other organizations have processes with lower associated risks that can be more lightly regulated. We have found in practice that the same governance process can handle both these extremes perfectly well. If there is a need for stricter governance, it can be met with tighter policies at the control points together with more stringent policy enforcement and compliance measurement. If less-strict governance is more appropriate, the same process can be used with less restrictive policies, fewer audits, and lower levels of checklist signoff required. Even within a single organization, different processes may require different styles of governance. Some processes, such as service certification, require stricter governance than other processes, such as solution architecture. Different organizational cultures require different levels of autonomy in decision making. Good governance requires good judgment. First, let's update our Figure 1 with the location of these control points so that you have a visual representation in mind as you read their descriptions. Figure 2 shows where the control points occur in that development cycle. Figure 2: Software Development Lifecycle with SOA Control Points Here are descriptions of these control points. Business Requirements and Service Identification Control Point For an SOA approach, there is an emphasis on creating services that provide agility and reuse for the business. This first business requirements and service identification control point consists of a high-level review to determine that services are being identified in accordance with services selection and prioritization policies. This first business requirements and service identification control point should address the following types of questions: • Business goals. What are the business goals that the business seeks to attain and how do we measure the benefits or progress toward the business goals via key performance indicators (KPIs)? • Do the requirements as we currently understand them clearly support those goals, and do they align with an existing "business heat map"? • Are those requirements sufficiently understood and agreed to? Are they presented in a form such as use cases, business process models, sequence diagrams, or class diagrams that are consistent with the SOA development approach? • How do we provide traceability of the requirements so that we can ascertain that those requirements have been met during the development process? Have those requirements been entered into an enterprise-wide requirements and business rules catalog? Is there any conflict with existing entries in that catalog? • Which of those requirements could be translated into good candidate services, either because they represent functionality that may be needed by multiple consumers or that might be needed for process automation? Which requirements could be better supported by deploying applications, automated processes, or manual processes? • Where we have identified candidate services, have we identified potential consumers, and determined whether any of them have specific requirements that should be considered? • Given finite IT resources, what development priority should we assign? Is ownership of any new candidate IT asset defined, and is outline funding available for its development? Solution Architecture Control Point Different IT developers and groups, if left to make all design decisions on their own, would invariably use completely different platforms, coding languages, tools, styles, methods, and techniques. This variation adds cost and complexity to the ability of the business to make future changes, and makes future maintenance very hard and costly. Further, it reduces the reliability, stability, and interoperability of the organization's IT assets. We have seen this problem at many organizations that we have visited. Simply put, the purpose of the solution architecture control point is to prevent that expensive multiplicity of approaches from occurring ever again. Essentially, any proposed IT artifact that makes it past this control point is part of the IT build plan. For the area of solution architecture, the governance should control for a series of criteria the following: • Do the proposed standards, policies, and reference architectures - the solution architecture - identify the standards, policies, and design patterns to be followed in the service implementation? This will include reference architectures, platform standards for hardware, and software-usage standards. • Have any reusable assets been identified and assessed for suitability? Has the service sourcing policy been followed? • Have the nonfunctional requirements been identified and assessed? This includes the number of transactions per time unit, a busy hour analysis, the service performance required, presentation access to the service functionality, data managed by the service, space required for the installation of the service, and any dependency and configuration requirements. Governance must validate that all these are considered and addressed. • Governance must validate that all security policies are being considered and addressed. • Governance must validate that all legal and regulatory policies are considered and addressed. • By this stage in the development of IT assets such as services or automated processes, the technical IT staff involved should have a pretty good idea about the complexity of the tasks involved, and the probable level of resources required to complete development. Should development of the asset be confirmed, the scope reduced, or the asset abandoned? Service Specification Control Point A service specification should be created for each service whose development has been approved. Best practices for service design must integrate both an IT and business perspective for the design of the interface and the responsibilities of each service. Because the service specification is, in effect, the organization's face to business partners, customers, and other stakeholders, the service externals - those details of a service that are to be made public - become an important part of the overall business design. The design should take into account the requirements of all potential service consumers (within reason), and be created at a granularity that maximizes business value. For the area of service identification and specification, the governance should control for a series of criteria the following: • Does the service identified make sense, is at the right granularity, and is not duplicating an existing service? • Does the service specification follow all SOA standards and policies? • Does the service specification follow the messaging model? If not, should an exception be granted? Service Design Control Point After the service solution architecture has been turned over to the design team, a number of design elaboration decisions must be made. Collectively, these form the service internals - a set of design models, notes, and advice that will guide the service developers as they create and test the service code. For the area of service design, the governance should control for a series of criteria the following: • Has a service architect confirmed that the design should be able to meet the nonfunctional and functional requirements for this service? • Have the service designer and data architect agreed that the service can be made to conform to the signature (that is, inputs and outputs) described in the service externals? • If a service is wrapping an existing or planned application, are the necessary interfaces to that application well defined and stable (that is, won't change if a new version of that application is installed)? • Have the monitoring metrics (for example, usage, quality of service [QoS] levels) been established? • In the case of automated processes, have the monitoring requirements been defined and planned? • In the case of long-running automated processes, have all the necessary actions to handle recovery from process errors or technical failures been addressed? • Is the overall quality and level of completeness of the service specification package good enough that the service developers or process developers can complete development without further input? Service Build Control Point After the service design has been turned over to the service build team, a number of implementation decisions need to be made before development of the code or executable model. In the interests of consistency and quality, we strongly recommend the use of code walkthrough reviews, where peers (that is, other service developers or process developers) review the work in progress and offer constructive criticism. The service build control point is effectively the last of these code walkthroughs, and should be performed with slightly more formality than the others. Questions that should be addressed include the following: • Was the asset coded in accordance with the design? • Does the code follow the accepted coding standards? • Have all the associated artifacts (for example, load libraries, metadata files, resources) been defined to create a transportable build? Have the versions of each of those artifacts been checked to see that there are no version conflicts with services already in production? Service Test Control Point Service testing is different from testing complete IT solutions or applications. Because services and automated processes do not have their own user interface, it is not possible to perform user acceptance testing directly on services or automated processes. Code frameworks or specialized tools are needed to exhaustively test services and automated processes thoroughly to avoid uncovering problems during later formal user acceptance testing when the rest of the IT solution that uses those services or processes has been completed. SOA governance must ascertain that the services test is being performed in a manner conducive to a services approach, and that exhaustive functional and nonfunctional tests have been passed before releasing any SOA asset to production. The service test team must create and use the right service test environment with tools and data to affect a comprehensive test. This should include the following: • Using the optimum set of service test tools and frameworks. • The use of an automated build and test environment that can enable fast changes of the tested software and regression testing. This environment must closely resemble the production environment. • A load/stress test tool to test nonfunctional requirements, specification, creation, and loading of realistic but artificial test data. • A test management reporting tool to keep management apprised of the testing status. • Trace the test case to the original user requirements. Service Certification and Deployment Control Point The objectives of the deployment are to migrate the services to the production environment while minimizing client downtime and impact on the business. This process is subject to many errors if performed manually. It is vital that the correct version of the services be deployed and that any deployment binding with other services and applications be performed quickly and correctly. Areas for governance to validate include the following: • The use of a tool that automates the deployment and back-out process. • Final certification checks have been made against the services to verify compliance with all policies and standards and being able to demonstrate that what was tested matches not only the requirements but what was delivered, and that no corrective changes made during testing have invalidated other test results. • IT operations have completed acceptance testing and have formally accepted the asset, signifying their confidence in being able to operate it within the terms of the QoS specified for it. • The service registrar and business service champion have reviewed the service description in the service registry and approved it. Certification of a service or automated process is a formal "passing out" ceremony, and granting of certification should signify that the SOA enablement team is happy for their reputation to be associated with the performance of the new asset. Service Vitality Control Point Service vitality takes place periodically as part of SOA governance to check up on and update the governance processes, procedures, policies, and standards in reaction to the results of the real world. This involves examining any and all lessons learned in any of the SOA planning, program control, development, or operations activities. It also includes such things as comments and feedback from all stakeholders and an examination of any common patterns (for example, common exemption requests or common reasons for failure to pass one or more control points) that need remedial action. Metrics in the efforts required for each stage of the development process can show trends that indicate improvements or declines in their vitality. A formal service vitality control point review should be conducted every three to six months to determine whether the SOA transition remains on track, and whether the level and style of governance is optimal. Individual service or automated processes should be reviewed every 6 to 12 months. Usage data of all versions of each service can determine any "stale" versions that can be deprecated or deleted, and whether the deployment options taken and decisions on who should own and who should access each service are optimal. Conclusion We have focused in this extract on SOA Governance service development control points as a method to create a software engineering capability of a service factory. The factory is a production line for services. All services pass through a common, repeatable series of development, deployment and management steps. Quality and governance is built-in throughout the entire process. References [REF-1] "SOA Governance: Achieving and Sustaining Business and IT Agility" by William A. Brown, Robert G. Laird, Clive Gee, Tilak Mitra (IBM Press, ISBN 0137147465, Copyright 2009 by International Business Machines Corporation. All rights reserved.) This article was originally published in The SOA Magazine (www.soamag.com), a publication officially associated with "The Prentice Hall Service-Oriented Computing Series from Thomas Erl" (www.soabooks.com). Copyright ©SOA Systems Inc. (www.soasystems.com)