MapDB is a pure Java database, specifically designed for the Java developer. The fundamental concept for MapDB is very clever yet natural to use: provide a reliable, full-featured and “tune-able” database engine using the Java Collections API. MapDB 1.0 has just been released, this is the culmination of years of research and development to get the project to this point. Jan Kotek, the primary developer for MapDB, also worked on predecessor projects (JDBM), starting MapDB as an entire from-scratch rewrite. Jan’s expertise and dedication to low-level debugging has yielded excellent results, producting an easy-to-use database for Java with comparable performance to many C-based engines. What sets MapDB apart is the “map” concept. The idea is to leverage the totally natural Java Collections API – so familiar to Java developers that most of them literally use it daily in their work. For most database interactions with a Java application, some sort of translator is required. There are many Object-Relational Mapping (ORM) tools to name just one category of such components. The goal has always been in the direction of making it natural to code in objects in the Java language, and translate them to a specific database syntax (such as SQL). However, such efforts have always come up short, adding complexity for both the application developer and the data architect. When using MapDB there is no object “translation layer” – developers just access data in familiar structures like Maps, Sets, Queues, etc. There is no change in syntax from typical Java coding, other than a brief initialization syntax and transaction management. A developer can literally transform memory-limited maps into a high-speed persistent store in seconds (typically changing just one line of code). A MapDB Example Here is a simple MapDB example, showing how easy and intuitive it is to use in a Java application: // Initialize a MapDB database DB db = DBMaker.newFileDB(new File("testdb")) .closeOnJvmShutdown() .make(); // Create a Map: Map myMap = db.getTreeMap(“testmap”); // Work with the Map using the normal Map API. myMap.put(“key1”, “value1”); myMap.put(“key2”, “value2”); String value = myMap.get(“key1”); ... That’s all you need to do, now you have a file-backed Map of virtually any size. Note the “builder-style” initialization syntax, enabling MapDB as the agile database choice for Java. There are many builder options that let you tune your database for the specific requirements at hand. Just a small subset of options include: In-memory implementation Enable transactions Configurable caching This means that you can configure your database just for what you need, effectively making MapDB serve the job of many other databases. MapDB comes with a set of powerful configuration options, and you can even extend the product to make your own data implementations if necessary. Another very powerful feature is that MapDB utilizes some of the advanced Java Collections variants, such as ConcurrentNavigableMap. With this type of Map you can go beyond simple key-value semantics, as it is also a sorted Map allowing you to access data in order, and find values near a key. Not many people are aware of this extension to the Collections API, but it is extremely powerful and allows you to do a lot with your MapDB database (I will cover more of these capabilities in a future article). The Agile Aspect of MapDB When I first met Jan and started talking with him about MapDB he said something that made a very important impression: If you know what data structure you want, MapDB allows you to tailor the structure and database characteristics to your exact application needs. In other words, the schema and ways you can structure your data is very flexible. The configuration of the physical data store is just as flexible, making a perfect combination for meeting almost any database need. They key to this capability is inherent in MapDB’s architecture, and how it translates to the MapDB API itself. Here is a simple diagram of the MapDB architecture: As you can see from the diagram, there are 3 tiers in MapDB: Collections API: This is the familiar Java Collections API that every Java developer uses for maintaining application state. It has a simple builder-style extension to allow you to control the exact characteristics of a given database (including its internal format or record structure). Engine: The Engine is the real key to MapDB, this is where the records for a database – including their internal structure, concurrency control, transactional semantics – are controlled. MapDB ships with several engines already, and it is straightforward to add your own Engine if needed for specialized data handling. Volume: This is the physical storage layer (e.g., on-disk or in-memory). MapDB has a few standard Volume implementations, and they should suffice for most projects. The main point is that the development API is completely distinct from the Engine implementation (the heart of MapDB), and both are separate from the actual physical storage layer. This offers a very agile approach, allowing developers to exactly control what type of internal structure is needed for a given database, and what the actual data structure looks like from the top-level Collections API. To make things even more extensible and agile, MapDB uses a concept of Engine Wrappers. An Engine Wrapper allows adding additional features and options on top of a specific engine layer. For example, if the standard Map engine is utilized for creating a B-Tree backed Map, it is feasible to enable (or disable) caching support. This caching feature is done through an Engine Wrapper, and that is what shows up in the builder-style API used to configure a given database. While a whole article could be written just about this, the point here is that this adds to MapDB’s inherent agile nature. By way of example, here is how you configure a pure in-memory database, without transactional capabilities: // Initialize an in-memory MapDB database // without transactions DB db = DBMaker.newMemoryDB() .transactionDisable() .closeOnJvmShutdown() .make(); // Create a Map: Map myMap = db.getTreeMap(“testmap”); // Work with the Map using the normal Map API. myMap.put(“key1”, “value1”); myMap.put(“key2”, “value2”); String value = myMap.get(“key1”); ... That’s it! All that was needed was to change the DBMaker call to add the new options, everything else works exactly the same as in the example shown earlier. Agile Data Model In addition to customizing the features and performance characteristics of a given database instance, MapDB allows you to create an agile data model, with a schema exactly matching your application requirements. This is probably similar to how you write your code when creating standard Java in-memory structures. For example, let’s say you need to lookup a Person object by username, or by personID. Simply create a Person object and two Maps to meet your needs: public class Person { private Integer personID; private String username; ... // Setters and getters go here ... } // Create a Map of Person by username. Map personByUsernameMap = ... // Create a Map of Person by personID. Map personByPersonIDMap = ... This is a very trivial example, but now you can easily write to both maps for each new Person instance, and subsequently retrieve a Person by either key. Another interesting concept with MapDB data structures are some key extensions to the normal Java Collections API. A common requirement in applications is to have a Map with a key/value, and in addition to finding the value for a key to be able to perform the inverse: lookup the key for a given value. We can easily do this using the MapDB extension for bi-directional maps: // Create a primary map HTreeMap map = DBMaker.newTempHashMap(); // Create the inverse mapping for primary map NavigableSet> inverseMapping = new TreeSet>(); // Bind the inverse mapping to primary map, so it is auto-updated each time the primary map gets a new key/value Bind.mapInverse(map, inverseMapping); map.put(10L,"value2"); map.put(1111L,"value"); map.put(1112L,"value"); map.put(11L,"val"); // Now find a key by a given value. Long keyValue = Fun.filter(inverseMapping.get(“value2”); MapDB supports many constructs for the interaction of Maps or other collections, allowing you to create a schema of related structures that can automatically be kept in sync. This avoids a lot of scanning of structures, makes coding fast and convenient, and can keep things very fast. Wrapping it up I have shown a very brief introduction on MapDB and how the product works. As you can see its strengths are its use of the natural Java Collections API, the agile nature of the engine itself, and the support for virtually any type of data model or schema that your application needs. MapDB is freely available for any use under the Apache 2.0 license. To learn more, check out: www.mapdb.org.

Explore different message brokers, and discover how these important web technologies impact a customer's backlog of messages, and cluster/data performance.

This is the second in a series of posts on the new “Red Deer” (https://github.com/jboss-reddeer/reddeer) open source testing framework for Eclipse. In the previous post in this series, we introduced Red Deer, learned how to install it into Eclipse, examined some of its cool features, and built and ran a sample test program from scratch. One of the challenges in creating effective automated tests is in making the tests self-sufficient enough to be able to set up their required operation environment, and robust enough to be able to determine whether that operating environment has been set up correctly. In the first post in this series, we took a quick look at Red Deer’s implementation of Requirements classes. In this post, we’ll take a more detailed look at Requirements, including how Red Deer supports your creating custom Requirements. The Case for Automated Test Requirements Let’s start by setting the context for why test programs need requirements. It’s often the case that a set of automated tests runs unattended and all the tests fail, not due to a bug in the software under test, but due to a broken or incomplete test environment. When we refer to a Red Deer “requirement,” we’re talking about actions that must be performed, or objects that must be created, before a test can be run. Examples of these requirements are having a user account defined or a connection to a database created and verified. What makes using Red Deer requirements different from your creating a less formal set of requirements with the @BeforeClass annotation provided by JUnit, is that if requirements are not met, then the test in question is not run. This can save you a lot of test execution time and test failure debugging time. Red Deer requirements are implemented in the RedDeerSuite. A test that makes use of requirements is must be run with a RedDeerSuite suite and annotated with @RunWith(RedDeerSuite.class) OOTB Red Deer Requirements As we saw in the first post in this series, Red Deer currently provides OOTB (out of the box) predefined requirements that enable you to clean out your current workspace and open a perspective. Using these requirements is simple. All you have to do is to add these import statements to your Red Deer test programs: import org.jboss.reddeer.eclipse.ui.perspectives.JavaBrowsingPerspective; import org.jboss.reddeer.requirements.cleanworkspace.CleanWorkspaceRequirement.CleanWorkspace; import org.jboss.reddeer.requirements.openperspective.OpenPerspectiveRequirement.OpenPerspective; And, we also have to add a reference to org.jboss.reddeer.requirements to the required bundle list in our example’s MANIFEST.MF file. And finally, add these annotations to the test program: @CleanWorkspace @OpenPerspective(JavaBrowsingPerspective.class) What if you want to define your own custom requirements? Let’s move on and examone how Red Deer supports that too. Different Ways to Implement New Red Deer Requirements Red Deer supports (4) ways to implement new requirements. We’ll look at them in order of their relative complexity: Simple Requirements Requirements with Parameters Requirements with Property Based Configuration Requirements with a Custom Schema In order to examine how Red Deer supports implementing new requirements, we’ll actually create some new requirements in Red Deer source code. In order to do this, we’ll have to download a copy of Red Deer’s source code. To perform this download, navigate to your desired directory and enter this command: git clone https://github.com/jboss-reddeer/reddeer.git And then, import Red Deer into eclipse as a set of existing Maven projects: If you navigate to the top level of the directory into which you downloaded the Red Deer source code, you’ll see this: What you want to do is to select all of the Red Deer projects. After you press the “Next>” key, Eclipse will import all the Red Deer packages as maven projects. (This may take a few minutes.) OK, now we can move on to creating some new requirements. We’ll start with the simplest of the (4) types, a simple requirement. Implementing a Simple Requirement A simple requirement consists of (2) parts: a “fulfilling” class that provides the code executed when the requirement is invoked, and an annotation that references that fulfilling class. As an illustration, let’s look at the skeleton “AdminUserRequirement” provided with your Red Deer download. This requirement is intended to serve as an example for implementing a full requirement to ensure that an admin-level user is defined before an attempt is made to run a test. The source file you want to look for is: /plugins/org.jboss.reddeer.examples/src/org/jboss/reddeer/junit/annotation/simple/AdminUserRequirement.java While it’s a small file, it’s a full example. It’s worthwhile examining it line-by-line: package org.jboss.reddeer.junit.annotation.simple; import java.lang.annotation.ElementType; import java.lang.annotation.Retention; import java.lang.annotation.RetentionPolicy; import java.lang.annotation.Target; import org.jboss.reddeer.junit.requirement.Requirement; import org.jboss.reddeer.junit.annotation.simple.AdminUserRequirement.AdminUser; /** * Admin user test requirement * @author lucia jelinkova * */ public class AdminUserRequirement implements Requirement { @Retention(RetentionPolicy.RUNTIME) @Target(ElementType.TYPE) public @interface AdminUser { } public boolean canFulfill() { // return true if you can connect to the database return true; } public void fulfill() { // create an admin user in the database if it does not exist yet } public void setDeclaration(AdminUser declaration) { // no need to access the annotation } } The important elements in this file are: Line 17 - @Retention - Specifies how the marked annotation is stored—Whether in code only, compiled into the class, or available at runtime through reflection. Line 18 - @Target - Marks another annotation to restrict the types of Java elements to which the the annotation can be applied Line 20 - AdminUser interface - This defines the object type used by the defined requirement. Line 23 - canFulfill method - In a fully written requirement this method will include the code to determine if the requirement can be met (or “fulfilled”). This method is set to always return a value of true. Line 32 - fulfill method - And here is the code that will be executed if the canFulfill method returns a value of true. For an example of the corresponding annotation in action, let’s look at the test program that is included with the fulfilling class. The test program is here: /plugins/org.jboss.reddeer.examples/src/org/jboss/reddeer/junit/annotation/simple/AdminUserTest.java This test program is also very short as it is a skeleton. The outline is there, but the specific logic that to implement the AdminUser requirement is left as an “exercise for the reader.” package org.jboss.reddeer.junit.annotation.simple; import org.jboss.reddeer.junit.runner.RedDeerSuite; import org.jboss.reddeer.junit.annotation.simple.AdminUserRequirement.AdminUser; import org.junit.Test; import org.junit.runner.RunWith; @RunWith(RedDeerSuite.class) @AdminUser /** * Test with AdminUser requirement * @author lucia jelinova * */ public class AdminUserTest { @Test public void test(){ // put test logic here } } The @AdminUser annotation on line NN tells the whole story. When this annotation is executed, the fulfilling class is invoked and if the “canFulfill()” method returns true, the test is executed. If the method returns false, then the test is not executed. Let’s run this test and see what happens. First, locate the AdminUserTest.java file in the eclipse Navigator view: Then, right-click and specify that it be executed as a JUnit test: And, not surprisingly, here’s the successful output from the test: Before we move on, let’s modify the canFulFill() method to return a false value, and rerun the test. The results look like this: 22:11:04.923 INFO [main][RequirementsRunnerBuilder] Found test class org.jboss.reddeer.junit.annotation.simple.AdminUserTest 22:11:04.924 INFO [main][RequirementsBuilder] Creating requirements for test class org.jboss.reddeer.junit.annotation.simple.AdminUserTest 22:11:04.925 DEBUG [main][RequirementsBuilder] Found requirement class org.jboss.reddeer.junit.annotation.simple.AdminUserRequirement for annotation interface org.jboss.reddeer.junit.annotation.simple.AdminUserRequirement$AdminUser 22:11:04.927 INFO [main][Requirements] Requirement class org.jboss.reddeer.junit.annotation.simple.AdminUserRequirement can be fulfilled: false 22:11:04.927 INFO [main][RequirementsRunnerBuilder] All requirements cannot be fulfilled, the test will NOT run So, this time, the requirement was not met and the test was not run. Note that the requirement did the work for us. We did not have to write a lot of new code to determine if the requirement had been met to decide whether or not to run the test. That’s all well and good for a simple requirement. But what about if we want to make the requirement a bit more flexible by enabling us to pass it a parameter? Let’s look at that next. Implementing a Requirement with Parameters In order to implement a requirement that accepts one or more parameters, we have to make two additions to the simple requirement that we just examined. First, we have to use a different requirement definition. The code that we want to look at this time is here: /plugins/org.jboss.reddeer.examples/src/org/jboss/reddeer/junit/annotation/advanced/UserRequirement.java The file looks like this: package org.jboss.reddeer.junit.annotation.advanced; import java.lang.annotation.ElementType; import java.lang.annotation.Retention; import java.lang.annotation.RetentionPolicy; import java.lang.annotation.Target; import org.jboss.reddeer.junit.requirement.Requirement; import org.jboss.reddeer.junit.annotation.advanced.UserRequirement.User; /** * Parameterized requirement with parameter name * @author vpakan * */ public class UserRequirement implements Requirement { @Retention(RetentionPolicy.RUNTIME) @Target(ElementType.TYPE) public @interface User { String name(); } private User user; public boolean canFulfill() { // return true if you can connect to the database return true; } public void fulfill() { System.out.println("Fulfilling reuirement User with name: " + user.name()); // create an admin user in the database if it does not exist yet } public void setDeclaration(User user) { this.user = user; } } The important difference between this class and the original AdminUserRequirement that we examined a moment ago is: Line 20 - The interface “User” now defines a String parameter “name” and on line NNN here the User object is defined. Second, we have to change the declaration of the requirement in the test program. The test program that we’ll look at this time is here: /plugins/org.jboss.reddeer.examples/src/org/jboss/reddeer/junit/annotation/advanced/UserTest.java Finally, our test program for this requirement looks like this: package org.jboss.reddeer.junit.annotation.advanced; import org.jboss.reddeer.junit.runner.RedDeerSuite; import org.jboss.reddeer.junit.annotation.advanced.UserRequirement.User; import org.junit.Test; import org.junit.runner.RunWith; @RunWith(RedDeerSuite.class) @User(name="admin") /** * Test with parameterized requirement User * @author lucia jelinkova * */ public class UserTest { @Test public void test(){ // put test logic here } } The interesting line in this test is: Line 8 - @User(name="admin") - Where we set the value of the “name” parameter. When we run the UserTest as a JUnit test, we see this output: 20:46:03.554 INFO [main][RequirementsRunnerBuilder] Found test class org.jboss.reddeer.junit.annotation.advanced.UserTest 20:46:03.555 INFO [main][RequirementsBuilder] Creating requirements for test class org.jboss.reddeer.junit.annotation.advanced.UserTest 20:46:03.556 DEBUG [main][RequirementsBuilder] Found requirement class org.jboss.reddeer.junit.annotation.advanced.UserRequirement for annotation interface org.jboss.reddeer.junit.annotation.advanced.UserRequirement$User 20:46:03.558 INFO [main][Requirements] Requirement class org.jboss.reddeer.junit.annotation.advanced.UserRequirement can be fulfilled: true 20:46:03.558 INFO [main][RequirementsRunnerBuilder] All requirements can be fulfilled, the test will run 20:46:03.575 INFO [main][RedDeerSuite] RedDeer suite created 20:46:03.584 INFO [main][Requirements] Fulfilling requirement of class org.jboss.reddeer.junit.annotation.advanced.UserRequirement Fulfilling requirement User with name: admin 20:46:03.585 DEBUG [main][RequirementsRunner] Injecting fulfilled requirements into test instance 20:46:03.587 INFO [main][RequirementsRunner] Started test: test(org.jboss.reddeer.junit.annotation.advanced.UserTest)20:46:03.588 INFO [main][RequirementsRunner] Finished test: test(org.jboss.reddeer.junit.annotation.advanced.UserTest) While it makes requirements more flexible when you are able to add parameters to their definition, it is still limited as a solution as you have to handle the individual parameters one by one. Fortunately, Red Deer also supports defining test configurations in your own custom XML schemas. Defining Complex Configurations - Two Approaches Red Deer supports two different approaches to defining complex configurations. You can either: Define the configuration as a set of (key=value) properties. If you choose this approach, you will have to also define setter methods for each property in your requirement’s fulfilling class. Create a custom XML schema. If you choose this approach, you will have to create a configuration object in your test code and then inject that object into your requirement. Regardless of which approach you choose, you store the configuration data in either a single XML file, or directory of XML files and then pass those files to your test program by defining this JVM argument when you run your test programs: -Dreddeer.config=/home/path/to/file/or/directory Let’s examine each of these approaches in detail. We’ll start with the properties based approach. Requirements with a Property Based Configuration The first thing we have to do to use a property based configuration is to define the properties. We’ll do this in an an XML file that complies with the Red Deer requirements XSD schema file. You can view the XSD here: http://cloud.github.com/downloads/jboss-reddeer/reddeer/RedDeerSchema.xsd The code for this example is here: /jboss/local/reddeer_fork/reddeer/plugins/org.jboss.reddeer.examples/src/org/jboss/reddeer/junit/configuration/simple And - here’s our properties file. Note that the requirement defined in this file contains two properties: name and ip (IP address). Let’s now expand on the “UserRequirement” example that we defined a few minutes ago. What we want to be able to do is to remove hardcoded requirements data from the source code and instead define that data in set of properties. To use this requirements.xml file, we have to make some changes to the UserRequirement.java class. package org.jboss.reddeer.junit.annotation.simple; import java.lang.annotation.ElementType; import java.lang.annotation.Retention; import java.lang.annotation.RetentionPolicy; import java.lang.annotation.Target; import org.jboss.reddeer.junit.requirement.Requirement; import org.jboss.reddeer.junit.annotation.simple.UserRequirement.User; import org.jboss.reddeer.junit.requirement.PropertyConfiguration; /** * Admin user test requirement * @author lucia jelinkova */ public class UserRequirement implements Requirement , PropertyConfiguration { @Retention(RetentionPolicy.RUNTIME) @Target(ElementType.TYPE) public @interface User { } private String name; private String ip; public boolean canFulfill() { // return true if you can connect to the database return true; } public void fulfill() { System.out.println("Fulfilling User requirement with\nName: " + name + "\nIP: " + ip); // create an admin user in the database if it does not exist yet } @Override public void setDeclaration(User user) { // annotation has no parameters no need to store reference to it } public void setName(String name) { this.name = name; } public void setIp(String ip) { this.ip = ip; } public String getName() { return name; } public String getIp() { return ip; } } The important changes are the addition of this import statement at line 8: import org.jboss.reddeer.junit.requirement.PropertyConfiguration And the addition of the implement clauses for the Requirement (with a type of User), and the PropertyConfiguration (so that the properties can be read) at line 15: public class UserRequirement implements Requirement , PropertyConfiguration And addition of the setter methods for the name and ip properties. Finally, here is the updated test program: package org.jboss.reddeer.junit.annotation.simple; import org.jboss.reddeer.junit.runner.RedDeerSuite; import org.jboss.reddeer.junit.annotation.simple.UserRequirement.User; import org.junit.Test; import org.junit.runner.RunWith; import org.jboss.reddeer.junit.requirement.inject.InjectRequirement; @RunWith(RedDeerSuite.class) @User /** * Test with AdminUser requirement * @author lucia jelinova * */ public class UserTest { @InjectRequirement private UserRequirement userRequirement; @Test public void test(){ System.out.println("The test is running"); System.out.println(userRequirement.getName()); // put test logic here } } What’s new in the test program is the addition of the import statement for the requirement injection: import org.jboss.reddeer.junit.requirement.inject.InjectRequirement; And the code to define and inject the UserRequirement: @InjectRequirement private UserRequirement userRequirement; When we run the test, we have to reference the configuration file via a Java VM argument . This means that we must define a new “run configuration” that is based on the JUnit run configuration provided in Eclipse and provide the VM argument that references the configuration file: In our example, the -Dreddeer.config VM argument is defined as: -Dreddeer.config=/jboss/local/reddeer_fork/reddeer/plugins/org.jboss.reddeer.examples/src/org/jboss/reddeer/junit/annotation/simple/reddeer.xml To execute the test, right-click on the UserTest class, and select the run configuration we just created: And, the test generates this test output in the console: 22:40:50.988 INFO [main][RedDeerSuite] Creating RedDeer suite... 22:40:50.990 INFO [main][SuiteConfiguration] Looking up configuration files defined via property reddeer.config=/jboss/local/reddeer_fork/reddeer/plugins/org.jboss.reddeer.examples/src/org/jboss/reddeer/junit/annotation/simple/reddeer.xml 22:40:50.991 INFO [main][SuiteConfiguration] Found configuration file /jboss/local/reddeer_fork/reddeer/plugins/org.jboss.reddeer.examples/src/org/jboss/reddeer/junit/annotation/simple/reddeer.xml 22:40:50.992 INFO [main][RedDeerSuite] Adding suite with name reddeer.xml to RedDeer suite 22:40:51.012 INFO [main][RequirementsRunnerBuilder] Found test class org.jboss.reddeer.junit.annotation.simple.UserTest 22:40:51.025 INFO [main][RequirementsBuilder] Creating requirements for test class org.jboss.reddeer.junit.annotation.simple.UserTest 22:40:51.027 DEBUG [main][RequirementsBuilder] Found requirement class org.jboss.reddeer.junit.annotation.simple.UserRequirement for annotation interface org.jboss.reddeer.junit.annotation.simple.UserRequirement$User 22:40:51.027 DEBUG [main][PropertyBasedConfigurator] Setting property based configuration to requirement class org.jboss.reddeer.junit.annotation.simple.UserRequirement 22:40:51.031 DEBUG [main][XMLReader] Reading configuration for class org.jboss.reddeer.junit.internal.configuration.entity.PropertyBasedConfiguration 22:40:51.827 DEBUG [main][PropertyBasedConfigurator] Configuration successfully set 22:40:51.828 INFO [main][Requirements] Requirement class org.jboss.reddeer.junit.annotation.simple.UserRequirement can be fulfilled: true 22:40:51.828 INFO [main][RequirementsRunnerBuilder] All requirements can be fulfilled, the test will run 22:40:51.865 INFO [main][RedDeerSuite] RedDeer suite created 22:40:51.874 INFO [main][Requirements] Fulfilling requirement of class org.jboss.reddeer.junit.annotation.simple.UserRequirement Fulfilling User requirement with Name: USERS_ADMINISTRATION IP: 127.0.0.1 22:40:51.875 DEBUG [main][RequirementsRunner] Injecting fulfilled requirements into test instance 22:40:51.876 INFO [main][RequirementsRunner] Started test: test reddeer.xml(org.jboss.reddeer.junit.annotation.simple.UserTest) 22:40:51.876 INFO [main][RequirementsRunner] Started test: test reddeer.xml(org.jboss.reddeer.junit.annotation.simple.UserTest) The test is running USERS_ADMINISTRATION 22:40:51.878 INFO [main][RequirementsRunner] Finished test: test reddeer.xml(org.jboss.reddeer.junit.annotation.simple.UserTest) 22:40:51.878 INFO [main][RequirementsRunner] Finished test: test reddeer.xml(org.jboss.reddeer.junit.annotation.simple.UserTest) Requirements with a Custom Schema The fourth and final approach to defining new requirements is to create a custom XML schema. This is the most complex approach, but it also provides you with the most flexibility as you can more easily share requirements in multiple configuration files. Also, this approach can protect you against forgetting to define properties in the configuration files by designating specific properties as required XML elements. To use this approach, you create a custom XML schema, then you create a configuration object in the test programs, and inject that object into your requirement. The configuration details are defined in an XML file and accessed through JAXB annotations. Let’s take a look at an example. The code for this example is available in Red Deer here: /plugins/org.jboss.reddeer.examples/src/org/jboss/reddeer/junit/configuration/advanced In order to use a custom XML schema, you need a custom schema. In this example, the schema is defined in a local file: /plugins/org.jboss.reddeer.examples/src/org/jboss/reddeer/junit/configuration/advanced/RedDeerRequirements.xsd This example schema is fairly simple, but it provides the flexibility needed for the example to define a test configuration of key=value pairs in the context of testruns and requirements. Also, the schema enforces the “required” setting for the requirement name. The configuration for requirement is defined in an XML requirement configuration file, the format of which complies with the custom schema: USERS_ADMINISTRATION 127.0.0.1 1111 In order to make use of this configuration, the Requirement class must instantiate a “UserConfiguration” object for the requirement. The UserRequirement class implements the org.jboss.reddeer.junit.requirement.CustomConfiguration interface with and specifies a type of UserConfiguration to enable the use of custom configurations: package org.jboss.reddeer.junit.configuration.advanced; import java.lang.annotation.ElementType; import java.lang.annotation.Retention; import java.lang.annotation.RetentionPolicy; import java.lang.annotation.Target; import org.jboss.reddeer.junit.requirement.CustomConfiguration; import org.jboss.reddeer.junit.requirement.Requirement; import org.jboss.reddeer.junit.configuration.advanced.UserRequirement.User; /** * User requirement using configuration from custom xml file * @author lucia jelinkova * */ public class UserRequirement implements Requirement, CustomConfiguration { @Retention(RetentionPolicy.RUNTIME) @Target(ElementType.TYPE) public @interface User { String name(); } private User user; private UserConfiguration userConfiguration; public boolean canFulfill() { // return true if you can connect to the database return true; } public void fulfill() { System.out.println("fulfiling requirement User with\nName: " + user.name() + "\nDB name: " + userConfiguration.getDbName() + "\nPort: " + userConfiguration.getPort() + "\nIP: " + userConfiguration.getIp()); // create an admin user in the database if it does not exist yet } public void setDeclaration(User user) { this.user = user; } public Class getConfigurationClass() { return UserConfiguration.class; } public void setConfiguration(UserConfiguration config) { this.userConfiguration = config; } } The UserConfiguration object (see line 25) is used by the org.jboss.reddeer.junit.requirement.CustomConfiguration class to provide the values for the requirement. The UserConfiguration definition (see below) maps the requirement as defined in the elements defined in the requirement XML file. package org.jboss.reddeer.junit.configuration.advanced; import javax.xml.bind.annotation.XmlElement; import javax.xml.bind.annotation.XmlRootElement; /** * Stores user requirement configuration loaded from custom xml file * @author lucia jelinkova * */ @XmlRootElement(name="user-requirement", namespace="http://www.jboss.org/NS/user-schema") public class UserConfiguration { private String dbName; private String ip; private String port; public String getIp() { return ip; } @XmlElement(namespace="http://www.jboss.org/NS/user-schema") public void setIp(String ip) { this.ip = ip; } public String getPort() { return port; } @XmlElement(namespace="http://www.jboss.org/NS/user-schema") public void setPort(String port) { this.port = port; } public String getDbName() { return dbName; } @XmlElement(name="db-name", namespace="http://www.jboss.org/NS/user-schema") public void setDbName(String dbName) { this.dbName = dbName; } } Note the getter and setter methods in the class definition. These methods make use of JAXB annotations to access the configuration element values. The test program looks largely the same as the test programs that we’ve used in the earlier examples. (It’s a nice characteristic of Red Deer tests in that since the “heavy lifting” is performed by the Red Deer harness, the tests can be kept simple, and therefore kept easy to maintain.) package org.jboss.reddeer.junit.configuration.advanced; import org.jboss.reddeer.junit.runner.RedDeerSuite; import org.jboss.reddeer.junit.configuration.advanced.UserRequirement.User; import org.junit.Test; import org.junit.runner.RunWith; /** * User test using configuration from custom xml file * Set VM parameter -Dreddeer.config to point to directory with requirements.xml file * -Dreddeer.config=${project_loc}/src/org/jboss/reddeer/junit/configuration/advanced * @author lucia jelinkova */ @RunWith(RedDeerSuite.class) @User(name="admin") public class UserTest { @Test public void test(){ // put your test logic here } } When the program is run, the console shows that the requirement was successfully met: 21:26:25.075 INFO [main][RedDeerSuite] Creating RedDeer suite... 21:26:25.077 INFO [main][SuiteConfiguration] Looking up configuration files defined via property reddeer.config=/jboss/local/reddeer_fork/reddeer/plugins/org.jboss.reddeer.examples/src/org/jboss/reddeer/junit/configuration/advanced/requirements.xml 21:26:25.077 INFO [main][SuiteConfiguration] Found configuration file /jboss/local/reddeer_fork/reddeer/plugins/org.jboss.reddeer.examples/src/org/jboss/reddeer/junit/configuration/advanced/requirements.xml 21:26:25.078 INFO [main][RedDeerSuite] Adding suite with name requirements.xml to RedDeer suite 21:26:25.084 INFO [main][RequirementsRunnerBuilder] Found test class org.jboss.reddeer.junit.configuration.advanced.UserTest 21:26:25.087 INFO [main][RequirementsBuilder] Creating requirements for test class org.jboss.reddeer.junit.configuration.advanced.UserTest 21:26:25.089 DEBUG [main][RequirementsBuilder] Found requirement class org.jboss.reddeer.junit.configuration.advanced.UserRequirement for annotation interface org.jboss.reddeer.junit.configuration.advanced.UserRequirement$User 21:26:25.089 DEBUG [main][CustomConfigurator] Setting custom configuration to requirement class org.jboss.reddeer.junit.configuration.advanced.UserRequirement 21:26:25.090 DEBUG [main][CustomConfigurator] Configuration object associated with requirement class org.jboss.reddeer.junit.configuration.advanced.UserRequirement is class org.jboss.reddeer.junit.configuration.advanced.UserConfiguration 21:26:25.090 DEBUG [main][XMLReader] Reading configuration for class org.jboss.reddeer.junit.configuration.advanced.UserConfiguration 21:26:25.782 DEBUG [main][CustomConfigurator] Configuration successfully set 21:26:25.832 INFO [main][Requirements] Requirement class org.jboss.reddeer.junit.configuration.advanced.UserRequirement can be fulfilled: true 21:26:25.832 INFO [main][RequirementsRunnerBuilder] All requirements can be fulfilled, the test will run 21:26:25.911 INFO [main][RedDeerSuite] RedDeer suite created 21:26:25.921 INFO [main][Requirements] Fulfilling requirement of class org.jboss.reddeer.junit.configuration.advanced.UserRequirement fulfiling requirement User with Name: admin DB name: USERS_ADMINISTRATION Port: 1111 IP: 127.0.0.1 21:26:25.922 DEBUG [main][RequirementsRunner] Injecting fulfilled requirements into test instance 21:26:25.923 INFO [main][RequirementsRunner] Started test: test requirements.xml(org.jboss.reddeer.junit.configuration.advanced.UserTest) 21:26:25.924 INFO [main][RequirementsRunner] Started test: test requirements.xml(org.jboss.reddeer.junit.configuration.advanced.UserTest) 21:26:25.925 INFO [main][RequirementsRunner] Finished test: test requirements.xml(org.jboss.reddeer.junit.configuration.advanced.UserTest) 21:26:25.925 INFO [main][RequirementsRunner] Finished test: test requirements.xml(org.jboss.reddeer.junit.configuration.advanced.UserTest) Before we move on, let’s try introducing an error in the XML test configuration and then see how Red Deer can trap that error. I don’t know about you, but avoiding typos is sometimes difficult for me. Let’s “inadvertently” remove the (required) name for the requirement. And rerun the test. This time, the console output shows: ERROR [main][XMLReader] cvc-complex-type.4: Attribute 'name' must appear on element 'user:user-requirement'. And the Junit output shows: org.jboss.reddeer.junit.configuration.RedDeerConfigurationException: Xml configuration is not valid. Recap In this post, we examined the (4) ways in which Red Deer supports creating your own custom test configurations. These methods range from simple requirements that optionally include parameters, to more complex requirements that can be defined in external XML files, either as key=value pairs or in a custom schema, that can be be shared between multiple test cases. It’s often the case that automated test runs can fail not because of bugs in software under test, but because the environment required by the test was properly initialized. Red Deer, by providing multiple approaches to create custom requirements helps you to ensure that your test failures can be more easily debugged and configuration errors are detected. What’s Next? In the next post in this series, we’ll take a look at how Red Deer makes creating new tests from scratch easier through its keystroke recorder feature. References https://github.com/jboss-reddeer/reddeer/wiki/Requirements http://www.oracle.com/technetwork/articles/javase/index-140168.html (JAXB)

I find it helpful to consider three questions when choosing a sprint goal: Why do we carry out the sprint? How do we reach its goal? And how do we know that the goal has been met? My sprint goal template therefore consists of three main parts: the actual goal, the method employed to reach the goal, and the metrics to determine if the goal has been met. It additionally provides a header section that allows you to state to which product and sprint the goal belongs, as the picture below shows. You can download the template as a PDF from romanpichler.com/tools/sprint-goal-template/ or by clicking on the image below. The template above has grown out of my experience of working with Scrum for more than ten years, and it is inspired by the scientific method and Lean Startup. Let’s have a look at the template sections in more detail. The Goal Section The goal section states why it is worthwhile to undertake the sprint. Examples are: Test an assumption about the user interaction and learn what works best for the user, for instance: “Will users be willing to register before using the product features?” Address a technical risk such as: “Does the architecture enable the desired performance?” Release a feature, for instance: “Get the reporting feature for general release.” The sprint goal hence differs from listing the user stories that should be implemented. It communicates the reason for carrying out the work, and it provides a motivation for running the sprint. The sprint goal should be shared: The product owner and the development team should believe that working towards the goal is the right thing to do. To choose the right sprint goal I find it helpful to consider the amount of uncertainty present. In the early sprints, addressing risks and testing assumptions allows me to learn about what the product should look like and do and how it is built. Once the key risks and critical assumptions have been dealt with, I like to focus on completing and optimising features, as the following picture shows: The Method Section This section addresses the question of how the goal is met. The default Scrum answer is simple: Create a (potentially shippable) product increment using the high-priority product backlog items, and demo it to the stakeholders in the sprint review meeting. But writing software and employing a product demo are not always the best methods to achieve the goal! A paper prototype can be good enough to test a visual design idea or an assumption about the user interaction, for instance. What’s more, other methods such as carrying out a usability test or releasing software to run an A/B test may well be more effective than a product demo. You should therefore carefully choose the right method and state it in this section. But don’t stop there. Determine the test group, the people who should provide feedback and data. Who these individuals are depends on the sprint goal: If you are validating an assumption about the visual design, the user interaction or the product functionality, then you probably want to collect feedback and data from the users. But if you are addressing a technical risk, then users may not be able to help you. Consider inviting a senior developer or architect from another team instead. Stating the test group clarifies who “the stakeholders” are, who is required to provide feedback so that the right product is developed. The Metrics Section The metrics section communicates how you determine if the goal has been met. Which metrics you use depends on the method chosen. For a product demo, you may state that at least two thirds of the stakeholders present should respond positively to the new feature, for instance; for a usability test, at least three of the five testers are complete the task successfully in less than a minute; and for the release of a new feature, you might say that at least 80% of the users use the new functionality at least once within five days after launching the feature. Whichever metrics you choose, make sure that they allow you to understand if and to which extent you have met the goal. The Header Section The header section consists of the two subsections “Product” and “Sprint”. They simply allow you to state which product and which sprint the goal belongs to. Customise this section according to your needs. If you work for an agencies or an IT solution provider, you could replace “Product” with “Project”, for instance. User Stories and the Sprint Goal You may be wondering how the template relates to the user stories. Let me first reiterate that your sprint goal should differ from your user stories. The goal explains the why it is a good idea to carry out the sprint an implement the stories. The user stories enable you to reach the goal. It’s a common mistake to confuse the two. To connect the template and the stories you have two options: You can state the relevant user stories in the template’s method section, or you can list them separately on the sprint backlog, as the following picture illustrates. In the picture above, the sprint goal is stated on the left to the sprint backlog, which lists the user stories and the tasks required to meet the goal in form of a task board. Learn more You can learn more about choosing effective sprint gaols and applying the sprint goal template by attending my Certified Scrum Product Owner training course. I have written in more detail about sprint planning in my book “Agile Product Management with Scrum”. Please contact me for onsite and virtual product owner training.

"And let Reform her columns roll. With thunder peal, and lightning flash..." - Ignis, "The Genius of Liberty" Vol III No. 2 Introduction In the past couple of articles we've seen how a Kanban board is able to help in the attainment of transparency and the stabilization of an agile team. Today we'll see if we can resolve one of the most common queries that result from this usage: how does a team decide which columns should appear on the board for tracking the progress of work items? The simplest case...and why it may not be enough When we set up a Kanban board in the last article, there were only three columns - or to use the correct term, "stations". These were a "Backlog" station (essentially a "To Do" list of work that has not yet been started), a station for showing which work is "In Progress", and a finally a station for representing the work that has been completed. You can't get much simpler than that, and it begs the question as to why you would want to make it more complicated. In practice however, there are at least two situations in which this minimalist approach will be found wanting: A team isn't cross-trained, and its members effectively work in skill silos. Consequently we can expect dependencies between team members, some of whom may become blocked while waiting for others to complete their part of the work. The incurral of this wasted time will not be apparent if it is all considered to be "work in progress". For example, the team may be split into developers and testers, and bottlenecks may arise as work passes between them. We may need to break Work In Progress down into further stations in order to expose this waste more fully. Bottlenecks arise due to constraints in the workflow, which is a different problem. In this situation a team might be fully cross-trained, and none of its members become blocked waiting for another. Rather, waste arises because the work itself is inefficiently staged. This often happens with activities like development, review, and test. For example if two people are required for a review, but only one is needed for development and test, then a bottleneck may well occur. Work will build-up awaiting review due to contention for these resources and the value of the investment in effort will start to depreciate. Again the incurral of this waste will not be apparent if it is all considered to be "work in progress". More stations are needed to expose it. Adding further stations These then are the two key things to consider when choosing additional stations. We're out to expose waste caused by work silos, or by the inappropriate staging of activities. Either of these can introduce constraints and become the source of bottlenecks in a value stream. Sometimes blockages can occur due to a dependency on something that must be done outside of the team. When this happens, it implies that the team are not fully in control of their own process, and consequently are unable to meet their own Definition of Done. They don't have all of the skills or resources needed. This is a problem and a contra-indication to agile practice. If it happens it's essential to make the dependency clear so that it can be challenged and removed. We may therefore choose to have an "externally blocked" column on the board to expose problems of this nature. It isn't really a station, because it doesn't represent a state in which value is added. Rather, it shows that an item has stalled within the value stream and that the team are not in a position to provide remedy. Another option is to place a red, day-glo sticky note on the ticket highlighting the seriousness of the problem. This is a clear signal that an impediment has occurred...that is to say, in Lean-Kanban terminology, it is an andon flag. In this case the flag shows that a major blockage has arisen and needs resolving. Challenging the boundaries Now we need to turn our attention to the boundaries of the board. There are two principal areas we should look at. Firstly, on the leftmost side of the board, we can see the work that a team inducts into its "Backlog" prior to actioning it. Secondly, on the rightmost side, we can see the work that the team considers to be "Done". These two boundaries are very often a source of waste. To understand why, just consider how backlogs are often allowed to grow without effective limit, and at how completed work may be permitted to accumulate in a "Done" column. These stations may not represent work in progress as far as the team is concerned, but it would be foolish to deny that they are batches too. After all, they are still part of the value stream. They represent inventory that is depreciating in value, or relevance, until something useful is done with it all. It behooves us to query the waste that is incurred, and to ask how the size of these batches may be constrained. Specifically: How can work be inducted into a backlog with minimal accumulation and delay? How can value be delivered to consumers as soon as work is completed? In short, what can be done to "lean" these process boundaries, so that inventory in the team's part of the value stream enters and exits in a "just-in-time" fashion? We can answer these questions by improving transparency still further. This can mean the refinement of the "Backlog" and "Done" columns into other, more finely-grained stations. For example, work might be building up in a Product Backlog because it is not being triaged appropriately, or perhaps because acceptance criteria are insufficiently well defined. We might be able to expose these problems by replacing a backlog with "Triaged", "Accepted", and "Ready" stations. At the other side of the board, completed work may be building up in the "Done" column because a release cannot yet be made. Additional stations such as "System Integrated", "In User Acceptance", and "Awaiting Release" could add clarity here. Removing stations The simple, 3 column board we started has now exploded into a behemoth of perhaps ten columns or more. This may seem like an excessively complex structure for a workflow and a casual observer may criticize it for being fundamentally unagile. After all, inventory should either be work in progress by an agile team, or it will be awaiting their attention or have already been completed. The criticism is a valid one but we need to bear one thing in mind: these stations are there to expose problems. Only once transparency has been attained can we hope to provide remedy. The bottlenecks, along with the diagnostic stations we added to reveal them, can then be removed. Conclusion Knowing how many "columns" to include on an agile board, and what they should be, is something of a black art to many agile teams. In this article we've looked at the issues involved in making this decision. The board of a fully cross-trained team should be elegant in its simplicity, but when problems arise we must be prepared to do some digging in order to root out their causes.

This is the first in a series of posts on the new “Red Deer” (https://github.com/jboss-reddeer/reddeer) open source testing framework for Eclipse. In this post, we’ll introduce Red Deer, and take a look at the some of the advantages that it offers by building a sample test program from scratch. Some of the features that Red Deer automated offers are: An easy to use, high-level API for testing standard Eclipse components Support for creating custom extensions for your own applications A requirements validation mechanism to assist you in configuring complex tests Eclipse Tooling to Assist in Creating new Projects A record and playback tool to enable you to quickly create automated tests An integration with Selenium for testing web based applications Support for running tests in a Jenkins CI environment Note that as of this writing, Red Deer is in an incubation stage. The current release is at level 0.5. The target date for the 1.0 release of Red Deer is late 2014. But, as a community-based, open source project, now is a great time to try Red Deer and make suggestions or even contribute code! A Look at Red Deer’s Architecture The Red Deer project itself is comprised of utilities and the API that supports the development and execution of automated tests. The API (the parts of the above diagram that are enclosed in dashed line boxes) can be thought of as having three layers: The top layer consists of extensions to Red Deer’s abstract classes or implementations for Eclipse components such as Views, Editors, Wizards, or Shells. For example, if you are writing tests for a feature that uses a custom Eclipse View, you can extend Red Deer’s View class by adding support for the specific functions of the feature. The advantage that this API layer gives you is that your test programs do not have to focus on manipulating the individual UI elements directly to perform operations. Your programs can instead instantiate an instance of an Eclipse component such as a View, and then use that instance’s methods to perform operations on the View. This layer of abstraction makes your test programs easier to write, understand, and maintain. The middle layer consists of the Red Deer implementations for SWT UI elements such as: Button, Combo, Label, Menu, Shell, TabItem, Table, ToolBar, Tree. This API layer supports the API’s higher level by providing the building blocks for the API’s Views, Editors, Shells, and WIzards. This middle layer of the API also provides Red Deer packages that enable your tests to enforce requirements, so that necessary setup tasks are performed before a test is run. The bottom layer consists of Red Deer packages that support the execution of tests such as: Conditions, Matchers, Widgets, Workbench, and Red Deer extensions to JUnit. What Makes Red Deer different from other Tools? A Layer of Abstraction The top-most layer of the API enables you to instantiate Eclipse UI elements as objects, and then manipulate them through their methods. The resulting code is easier to read and maintain, instead of being brittle and subject to failures when the UI changes. For example, for a test that has to open a view and press a button, without Red Deer, the test would have to navigate the top level menu, find the view menu, then the view type in that menu, then find the view open dialog, then locate the “OK” button, etc. Your test would have to spend a lot of time navigating through the UI elements before it could even begin to perform the test’s steps. With Red Deer, the code to open a view (in this case, the servers view) is simply: ServersView view = new ServersView(); view.open(); Furthermore, within that ServersView, your test program can perform operations on the View through methods which are defined in the view (and are incidentally also well debugged by the Red Deer team), instead of having to explicitly locate and manipulate the UI elements directly. For example, to obtain a list of all the servers, instead of locating the UI tree that contains the server list, and extracting that list of servers into an array, your Red Deer program can simply call the “getServers()” method. Likewise, the code to open a PackageExplorer, and then select a project within that PackageExplorer is as follows: PackageExplorer packageExplorer = new PackageExplorer(); packageExplorer.open(); packageExplorer.getProject("myTestProject").select(); And, the code to retrieve all the projects within that PackageExplorer is simply: packageExplorer.getProjects(); The result are that your tests are easier to write and maintain and you can focus on testing your application’s logic instead of writing brittle code to navigate through the application. Installing Red Deer The only prerequisites to using Red Deer are Eclipse and Java. In this post, we’ll use Eclipse Kepler and OpenJDK 1.7, running on Red Hat Enterprise Linux (RHEL) 6. To install Red Deer 0.4 (this is the latest stable milestone version as of this writing) follow these steps: Open up Eclipse Navigate to: Help->Install New Software Define a new download site using the Red Deer update site URL: http://download.jboss.org/jbosstools/updates/stable/kepler/core/reddeer/0.4.0/ Select Red Deer, click on the Finish button and Red Deer will install Now that you have Red Deer installed, let’s move onto building a new Red Deer test. Building your First Red Deer Test To create a new Red Deer test project, you make use of the Red Deer UI tooling and select New->Project->Other->Red Deer Test: Before we move on, let’s take a look at the WEB-INF/MANIFEST.MF file that is created in the project: Manifest-Version: 1.0 Bundle-ManifestVersion: 2 Bundle-Name: com.example.reddeer.sample Bundle-SymbolicName: com.example.reddeer.sample;singleton:=true Bundle-Version: 1.0.0.qualifier Bundle-ActivationPolicy: lazy Bundle-Vendor: Sample Co Bundle-RequiredExecutionEnvironment: JavaSE-1.6 Require-Bundle: org.junit, org.jboss.reddeer.junit, org.jboss.reddeer.swt, org.jboss.reddeer.eclipse The line we’re interested in is the final line in the file. These are the bundles that are required by Red Deer. After the empty project is created by the wizard, you can define a package and create a test class. Here's the code for a minimal functional test. The test will verify that the eclipse configuration is not empty. package com.example.reddeer.sample; import static org.junit.Assert.assertFalse; import java.util.List; import org.jboss.reddeer.swt.api.TreeItem; import org.jboss.reddeer.swt.impl.button.PushButton; import org.jboss.reddeer.swt.impl.menu.ShellMenu; import org.jboss.reddeer.swt.impl.tree.DefaultTree; import org.junit.Test; import org.junit.runner.RunWith; import org.jboss.reddeer.junit.runner.RedDeerSuite; @RunWith(RedDeerSuite.class) public class SimpleTest { @Test public void TestIt() { new ShellMenu("Help", "About Eclipse Platform").select(); new PushButton("Installation Details").click(); DefaultTree ConfigTree = new DefaultTree(); List ConfigItems = ConfigTree.getAllItems(); assertFalse ("The list is empty!", ConfigItems.isEmpty()); for (TreeItem item : ConfigItems) { System.out.println ("Found: " + item.getText()); } } } After you save the test's source file, you can run the test. To run the test, select the Run As->Red Deer Test option: And - there's the green bar! Simplifying Tests with Requirements Red Deer requirements enable you to define actions that you want happen before a test is executed. The advantage to using requirements is that you define the actions with annotations instead of using a @BeforeClass method. The result is that your test code is easier to read and maintain. The biggest difference between a Red Deer requirement and the the @BeforeClass annotation from the JUnit framework is that if a requirement cannot be fulfilled the test is not executed. Like everything else in Red Deer, you can make use of predefined requirements, or you can extend the feature by adding your own custom requirements. These custom requirements can be made complex and for convenience can be stored in external properties files. (We’ll take a look at defining custom requirements in a later post in this series when we examine how to create and contribute extensions to Red Deer.) The current milestone release of Red Deer provides predefined requirements that enable you to clean out your current workspace and open a perspective. Let’s add these to our example. To do this, we need to add these import statements: import org.jboss.reddeer.eclipse.ui.perspectives.JavaBrowsingPerspective; import org.jboss.reddeer.requirements.cleanworkspace.CleanWorkspaceRequirement.CleanWorkspace; import org.jboss.reddeer.requirements.openperspective.OpenPerspectiveRequirement.OpenPerspective; And these annotations: @CleanWorkspace @OpenPerspective(JavaBrowsingPerspective.class) And, we also have to a reference to org.jboss.reddeer.requirements to the required bundle list in our example’s MANIFEST.MF file: Require-Bundle: org.junit, org.jboss.reddeer.junit, org.jboss.reddeer.swt, org.jboss.reddeer.eclipse, org.jboss.reddeer.requirements When we’re done, our example looks like this: package com.example.reddeer.sample; import static org.junit.Assert.assertFalse; import java.util.List; import org.jboss.reddeer.swt.api.TreeItem; import org.jboss.reddeer.swt.impl.button.PushButton; import org.jboss.reddeer.swt.impl.menu.ShellMenu; import org.jboss.reddeer.swt.impl.tree.DefaultTree; import org.junit.Test; import org.junit.runner.RunWith; import org.jboss.reddeer.junit.runner.RedDeerSuite; import org.jboss.reddeer.eclipse.ui.perspectives.JavaBrowsingPerspective; import org.jboss.reddeer.requirements.cleanworkspace.CleanWorkspaceRequirement.CleanWorkspace; import org.jboss.reddeer.requirements.openperspective.OpenPerspectiveRequirement.OpenPerspective; @RunWith(RedDeerSuite.class) @CleanWorkspace @OpenPerspective(JavaBrowsingPerspective.class) public class SimpleTest { @Test public void TestIt() { new ShellMenu("Help", "About Eclipse Platform").select(); new PushButton("Installation Details").click(); DefaultTree ConfigTree = new DefaultTree(); List ConfigItems = ConfigTree.getAllItems(); assertFalse ("The list is empty!", ConfigItems.isEmpty()); for (TreeItem item : ConfigItems) { System.out.println ("Found: " + item.getText()); } } } Notice how we were able to add those functions to the test code, while only adding a very small amount of actual new code? Yes, it can pay to be a lazy programmer. ;-) What’s Next? What’s next for Red Deer is its continued development as it progresses through its incubation stage until its 1.0 release. What’s next for this series of posts will be discussions about: The Red Deer Recorder - To enable you to capture manual actions and convert them into test programs How you can Extend Red Deer - To provide test coverage for your plugins’ specific functions. And How you can Contribute these extensions to the Red Deer project. How you can Define Complex Requirements - To enable you to perform setup tasks for your tests. Red Deer’s Integration with Selenium - To enable you to test web interfaces provided by your plugins. Running Red Deer tests with Jenkins - To enable you to take advantage of Jenkins’ Continuous Integration (CI) test framework. Author’s Acknowledgements I’d like to thank all the contributors to Red Deer for their vision and contributions. It’s a new project, but it is growing fast! The contributors (in alphabetic order) are: Stefan Bunciak, Radim Hopp, Jaroslav Jankovic, Lucia Jelinkova, Marian Labuda, Martin Malina, Jan Niederman, Vlado Pakan, Jiri Peterka, Andrej Podhradsky, Milos Prchlik, Radoslav Rabara, Petr Suchy, and Rastislav Wagner.

A product roadmap is a high-level, strategic plan, which provides a longer-term outlook on the product. This creates a continuity of purpose, and it helps product managers and owners acquire funding for their product; it sets expectations, aligns stakeholders, and facilitates prioritization; it makes it easier to coordinate the development and launch of different products, and it provides reassurance to the customers (if the product roadmap is made public). Unfortunately, I find that many product managers and product owners struggle with their roadmaps, as they are dominated by features: There are too many features, and the features are often too detailed. This turns a roadmap into a tactical planning tool that competes with the Product Canvas or product backlog. What’s more, the features are sometimes regarded as a commitment by senior management than part of a high-level plan that is likely to change. The GO Product Roadmap Explained Faced with this situation, I have developed a new goal-oriented agile roadmap — the GO product roadmap, or “GO” for short. GO is based on my experience of teaching and coaching product managers and product owners, as well as using product roadmaps in my own business. The following pictures shows what the GO product roadmap looks like. You can download a PDF and Excel template by simply clicking on the picture. The first row of the GO roadmap depicted above contains the date or timeframe for the upcoming releases. You can work with a specific date such as 1st of March, or a period such as the first or second quarter. The second row states the name or version of the releases, for instance, iOS 7 or Windows 8.1. The third row provides the goal of each release, the reason why it is worthwhile to develop and launch it. Sample goals are to acquire or to activate users, to retain users by enhancing the user experience, or to accelerate development by removing technical debt. Working with goals shifts the conversation from debating individual features to agreeing on desired benefits making strategic product decisions. The development team, the stakeholders, and the management sponsor should all buy into the goals. The fourth row provides the features necessary to reach the goal. The features are means to an end, but not an end in themselves: They serve to create value and to reach the goal. Try to limit the number of features for each release to three, but do not state more than five. Refrain from detailing the features, and focus on the product capabilities that are necessary to meet the goal. Your product roadmap should be a high-level plan. The details should be covered in the Product Canvas or product backlog, and commitments should be limited to individual sprints. The last row states the metrics, the measurements or key performance indicators (KPIs) that help determine if the goal has been met, and if the release was successful. Make sure that the metrics you select allow you to measure if and to which extent you have met the goal. A Sample GO Product Roadmap To illustrate how the GO template can be applied, imagine we are about to develop a new dance game for girls aged eight to 12 years. The app should be fun and educational allowing the players to modify the characters, change the music, dance with remote players, and choreograph new dances. Here is what the corresponding GO roadmap could look like: While the roadmap above will have to be updated and refined at a later stage (particularly the metrics), I find it good enough to show how the product may evolve and make an investment decision. When creating your GO roadmap make sure you determine the goal of each release before you identify the features. This ensures that the features do serve the goal. Filling in the roadmap template from top to bottom and from left to right works well for me. Wrap-up The GO product roadmap provides a new, powerful way to do product roadmapping. Rather than focussing on features, GO emphasizes the importance of shared goals. This makes it easier to communicate the roadmap, create alignment, and use it as a strategic planning tool that provides an umbrella for the Product Canvas and the product backlog. The metrics provided by the tool ensure that the goals are measurable rather than lofty and fuzzy ideas. Download the template now, and try it out! You can learn more about creating effective product roadmap and working with the GO product roadmap by attending my Agile Product Planning training course. I would love to hear your questions about the roadmap and your experiences of creating product roadmaps. Please leave a comment below, or contact me.

The second blog post in the Exploring the landscape of large scale agile frameworks - series “Wet agile”, “the agile waterfall” and “the Agile-Waterfall Hybrid” … this controversial, mixed-method baby has as many names as formats. Some have received a lot of dedicated thought, are fit-for-purpose and manage to preserve the main benefits of the more pure methods. Other hybrid models have resulted by accident; either as a consequence of… … a half-way-stopped agile transitioning program that dropped dead somewhere between waterfall and scrum in an unsuccessful change initiative … a compromise between method-promoters of different ideologies (often management being on the more waterfall friendly side vs. developers generally promoting more agility) The result of the latter two cases can be horrible to witness and worse to experience for everyone involved regardless if they sit within the development organization or are on the receiving end waiting for the product. However, I do not think the frequent occurrence of poor agile-waterfall hybrids we can witness in the industry is a reason to consistently argue against mixing the two arts as many agilists do. Rather, these cases highlight the need to bring the successful and proven versions of agile-waterfall hybrids into the light and reserve a place for them among their purer cousins in the landscape for large scale agile frameworks. Founders of more successful hybrid versions often come from product development companies that deal with both software and hardware. The driver behind the method mixing in these cases is, if we simplify to the extreme, the realization that many aspects of hardware development benefit from waterfall processes, whereas software development has much to gain from an agile approach. The Agile-Waterfall Hybrid described by Erick Bergmann and Andy Hamilton from Schneider Electric is a great example of a model that merges Agile and Project Management Process/waterfall in a good way without compromising the methods’ core principles to much. Erick Bergmann and Andy Hamilton presented the Agile-Waterfall Hybrid at Agile 2013 and the key concepts of the model can be well understood from the presenting material; https://submissions.agilealliance.org/system/sessions/attachments/000/000/760/original/Agile-Waterfall_Hybrid_01AUG2013.pdf The model allows software teams to adapt agile practices while hardware development and overall product management is handled through a traditional PMP/waterfall approach The overall PMP process has a well-defined interface to the agile software development which is continuous right from the start of the concept/feasibility study up until validation and production. The interface has the format of close collaboration for all activities ranging from definition of requirements and scoping to continuous software deliveries and feedback between the agile side and the waterfallish PMP side The model accommodates development where the same/similar software is used in several products with separate POs. Eg., the software teams need to deliver features to multiple stakeholders continuously in a similar fashion as component teams typically need to act towards feature teams. The very challenging backlog management situation resulting for teams in environments like this is addressed in a good way by the model. In short, it describes how you can achieve swift product releases by putting effort into the feature releases planning on the software side There is no perfect alternative. Just like any other hybrid model, this Agile-Waterfall Hybrid compromises with some of the principles of the non-hybrid methods. The waterfall side of development is forced to live with the flexibility surrounding the software requirements and the agile teams must commit to time-fixed delivery dates, cost forecasts and risk assessment. Hybrid development models face many challenges. It is not easy to combine the dependency tracking and clarity of waterfall with the flexibility and openness to change of Agile development without diluting the benefits and create complex work processes. However, examples like the one referred to above prove that it is possible and as the industry need for these types of models is evident I hope to see more of these being published in the future.

Because Agile development teams work from a backlog of stories, one way to inject application security into software development is by writing up application security risks and activities as stories, making them explicit and adding them to the backlog so that application security work can be managed, estimated, prioritized and done like everything else that the team has to do. Security Stories SAFECode has tried to do this by writing a set of common, non-functional Security Stories following the well-known “As a [type of user] I want {something} so that {reason}” template. These stories are not customer- or user-focused: not the kind that a Product Owner would understand or care about. Instead, they are meant for the development team (architects, developers and testers). Example: As a(n) architect/developer, I want to ensure AND as QA, I want to verify that sensitive data is kept restricted to actors authorized to access it. There are stories to prevent/check for the common security vulnerabilities in applications: XSS, path traversal, remote execution, CSRF, OS command injection, SQL injection, password brute forcing. Checks for information exposure through error messages, proper use of encryption, authentication and session management, transport layer security, restricted uploads and URL redirection to un-trusted sites; and basic code quality issues: NULL pointer checking, boundary checking, numeric conversion, initialization, thread/process synchronization, exception handling, use of unsafe/restricted functions. SAFECode also includes a list of secure development practices (operational tasks) for the team that includes making sure that you’re using the latest compiler, patching the run-time and libraries, static analysis, vulnerability scanning, code reviews of high-risk code, tracking and fixing security bugs; and more advanced practices that require help from security experts like fuzzing, threat modeling, pen tests, environmental hardening. Altogether this is a good list of problems that need to be watched out for and things that should be done on most projects. But although SAFECode’s stories look like stories, they can’t be used as stories by the team. These Security Stories are non-functional requirements (NFRs) and technical constraints that (like requirements for scalability and maintainability and supportability) need to be considered in the design of the system, and may need to be included as part of the definition of done and conditions of acceptance for every user story that the team works on. Security Stories can’t be pulled from the backlog and delivered like other stories and removed from the backlog when they are done, because they are never “done”. The team has to keep worrying about them throughout the life of the project and of the system. As Rohit Sethi points out, asking developers to juggle long lists of technical constraints like this is not practical: If you start adding in other NFR constraints, such as accessibility, the list of constraints can quickly grow overwhelming to developers. Once the list grows unwieldy, our experience is that developers tend to ignore the list entirely. They instead rely on their own memories to apply NFR constraints. Since the number of NFRs continues to grow in increasingly specialized domains such as application security, the cognitive burden on developers’ memories is substantial. OWASP Evil User Stories – Hacking the Backlog Someone at OWASP has suggested an alternative, much smaller set of non-functional Evil User Stories that can be "hacked" into the backlog: A way for a security guy to get security on the agenda of the development team is by “hacking the backlog”. The way to do this is by crafting Evil User Stories, a few general negative cases that the team needs to consider when they implement other stories. Example #1. "As a hacker, I can send bad data in URLs, so I can access data and functions for which I'm not authorized." Example #2. "As a hacker, I can send bad data in the content of requests, so I can access data and functions for which I'm not authorized." Example #3. "As a hacker, I can send bad data in HTTP headers, so I can access data and functions for which I'm not authorized." Example #4. "As a hacker, I can read and even modify all data that is input and output by your application." Thinking like a Bad Guy – Abuse Cases and Abuser Stories Another way to beef up security in software development is to get the team to carefully look at the system they are building from the bad guy's perspective. In “Misuse and Abuse Cases: Getting Past the Positive”, Dr. Gary McGraw at Cigital talks about the importance of anticipating things going wrong, and thinking about behaviour that the system needs to prevent. Assume that the customer/user is not going to behave, or is actively out to attack the application. Question all of the assumptions in the design (the can’ts and won’ts), especially trust conditions – what if the bad guy can be anywhere along the path of an action (for example, using an attack proxy between the client and the server)? Abuse Cases are created by security experts working with the team as part of a critical review – either of the design or of an existing application. The goal of a review like this is to understand how the system behaves under attack/failure conditions, and document any weaknesses or gaps that need to be addressed. At Agile 2013 Judy Neher presented a hands-on workshop on how to write Abuser Stories, a lighter-weight, Agile practice which makes “thinking like a bad guy” part of the team’s job of defining and refining user requirements. Take a story, and as part of elaborating the story and listing the scenarios, step back and look at the story through a security lens. Don’t just think of what the user wants to do and can do - think about what they don’t want to do and can’t do. Get the same people who are working on the story to “put their black hats on” and think evil for a little while, brainstorm to come up with negative cases. As {some kind of bad guy} I want to {do some bad thing}… The {bad guy} doesn’t have to be a hacker. They could be an insider with a grudge or a selfish customer who is willing to take advantage of other users, or an admin user who needs to be protected from making expensive mistakes, or an external system that may not always function correctly. Ask questions like: How do I know who the user is and that I can trust them? Who is allowed to do what, and where are the authorization checks applied? Look for holes in multi-step workflows – what happens if somebody bypasses a check or tries to skip a step or do something out of sequence? What happens if an action or a check times-out or blocks or fails – what access should be allowed, what kind of information should be shown, what kind shouldn’t be? Are we interacting with children? Are we dealing with money? With dangerous command-and-control/admin functions? With confidential or pirvate data? Look closer at the data. Where is it coming from? Can I trust it? Is the source authenticated? Where is it validated – do I have to check it myself? Where is it stored (does it have to be stored)? If it has to be stored, should it be encrypted or masked (including in log files)? Who should be able to see it? Who shouldn’t be able to see it? Who can change it, and to the changes need to be audited? Do we need to make sure the data hasn't been tampered with (checksum, HMAC, digital signature)? Use this exercise to come up with refutation criteria (user can do this, but can’t do that; they can see this but they can’t see that), instead of, or as part of the conditions of acceptance for the story. Prioritize these cases based on risk, add the cases that you agree need to be taken care of as scenarios to the current story, or as new stories to the backlog if they are big enough. “Thinking like a bad guy” as you are working on a story seems more useful and practical than other story-based approaches. It doesn’t take a lot of time, and it’s not expensive. You don’t need to write Abuser Stories for every user Story and the more Abuser Stories that you do, the easier it will get – you'll get better at it, and you’ll keep running into the same kinds of problems that can be solved with the same patterns. You end up with something concrete and functional and actionable, work that has to be done and can be tested. Concrete, actionable cases like this are easier for the team to understand and appreciate – including the Product Owner, which is critical in Scrum, because the Product Owner decides what is important and what gets done. And because Abuser Stories are done in phase, by the people who are working on the stories already (rather than a separate activity that needs to be setup and scheduled) they are more likely to get done. Simple, quick, informal threat modeling like this isn’t enough to make a system secure – the team won’t be able to find and plug all of the security holes in the system this way, even if the developers are well-trained in secure software development and take their work seriously. Abuser Stories are good for identifying business logic vulnerabilities, reviewing security features (authentication, access control, auditing, password management, licensing) improving error handling and basic validation, and keeping onside of privacy regulations. Effective software security involves a lot more work than this: choosing a good framework and using it properly, watching out for changes to the system's attack surface, carefully reviewing high-risk code for design and coding errors, writing good defensive code as much as possible, using static analysis to catch common coding mistakes, and regular security testing (pen testing and dynamic analysis). But getting developers and testers to think like a bad guy as they build a system should go a long way to improving the security and robustness of your app.

Some months ago I wrote an article on how to transition between Scrum and a Lean Kanban operation. It's an important capability for an organization to have, because when a Scrum project finishes it is likely to enter a "leaner" BAU (Business As Usual) support phase. There are consequences arising from such a move which experienced Scrum hands may find surprising, and perhaps even a little off-putting. In this article we'll look at the shift in mindset that is required to do this. "Whoa! Something screwy has happened to our task board, it looks different" Kanban boards are subtly different to the task boards commonly used in Scrum. At first blush they might look similar. Both have columns showing the progress of user story "tickets" from a backlog through states such as in progress, peer review, in test, and done. In either case there might also be a blocked column, although it is equally acceptable to add a "blocked" sticker, or to simply invert the ticket on the board. As the name suggests, a task board will show the progress of the tasks that are needed to complete user stories. Often these tasks will be kept within horizontal swim lanes - one lane per user story. When all of the tasks are done, the user story will also move into done. Each user story therefore "chases" its tasks across the board. A Kanban board on the other hand - which is meant to deal with smaller and finer-grained pieces of work - will typically track the progress of user stories themselves across the board. The requirements should be well understood and there should be little appreciable depth to the solutioning; there will be few if any explicit tasks associated with the user stories. There is therefore no need for horizontal swim lanes to keep tasks and user stories aligned. You might also notice that Work in Progress limits are given particular emphasis in Lean Kanban. This is because scope is not timeboxed into sprints. The only way to throttle the rate of ticket throughput, and to keep it to manageable levels, is therefore by making sure that WIP limits are rigorously enforced. These are often annotated to the column headers on a Kanban board. For example, if there are 3 developers and 1 tester, the WIP for in progress would be 3, and 1 for in test. "Hey…there's just one backlog" That's right. Since there are no sprints in Lean Kanban, there can be no meaningful separation between a "sprint backlog" and a "product backlog". Instead there's just a single backlog of enqueued work items being brought into progress. This has repercussions for product ownership because you no longer have a clear separation between the prioritization that a team does for itself on a sprint backlog, and the prioritization done by a Product Owner on the product backlog. In effect you've just got a product backlog. In this situation clear product ownership can become more important then ever…or it can become a complete non-issue. "The Product Owner has too much power, he keeps jerking our chain" Since there is only one backlog, the Product Owner (or customer representative) must constantly reprioritize the user stories within it. The Product Owner needs to have more operational control in Lean Kanban than in Scrum. Developers can action tickets from the backlog on a daily or even hourly basis. There is no notion of getting a product backlog in shape before "the next sprint starts". Product Owners are therefore much more closely involved in day-to-day delivery than they would be in Scrum, and their involvement in daily standups becomes much more important. Note that the extent of a Product Owner's decision making should not extend beyond the backlog, and a good Kanban Leader will protect the team and its work in progress just like a good ScrumMaster would. "Now the Product Owner has disappeared altogether" Business as Usual work often boils down to the maintenance of existing systems post-delivery. Depending upon the level of demand, it's quite plausible to have one Lean-Kanban team responsible for the maintenance of multiple systems. In this situation there is no product being delivered as such, and consequently there is no clear product ownership. Instead, work items are raised as change requests and triaged by the team who then manage and prioritize their own backlog. This means that the team needs a strong and shared sense of direction and purpose. "There's no vision for this project" That's because a Lean Kanban operation typically isn't a project at all. A defined end point is likely to be missing… remember that it's covering "Business as Usual work". These are small, repeatable changes that may affect diverse systems and without any sort of narrative to bind them together. There'll certainly be a purpose and a rationale for operating a Lean Kanban… but don't expect a project vision. "We don't even seem to have decent sprint goals any more" Yep, they've gone too. Since there is no project vision and no sprints on a Lean Kanban, we won't have any "sprint goals" either. What we might get is a grouping of work requests that fall within a larger epic of changes…but if we do, it could well be a cause for concern. We must ask: are those related changes really representative of "Business as Usual" work, or are they too high risk? Do they constitute a project? "Lean Kanban work seems very bitty. I can't get a decent chunk to chew on" The diet of a Lean Kanban should consist of small, "digestible" pieces of work that do not require much breaking down in order to action them. By definition they must be well-understood and low-risk. A team must know how to handle them without the need for impact analysis or de-scoping. You're unlikely to get a meaty piece of work; you're more likely to be sucking these things up through a straw. Velocity and lead times are particularly significant metrics in Lean Kanban. Having said that, substantial and time consuming pieces of work can be taken on board if they satisfy the criteria of low risk and clear scope. An example would be the sort of work that conforms to a templated change. Of course, this sort of work might not appeal to an agile developer. So let's be clear: it takes a different temperament to do Lean Kanban BAU work than project work in Scrum. They are different skill sets. Agile developers who are happy doing one can find it unsettling, or even unrewarding, if they are switched to the other. "Why aren't we doing planning poker any more?" Without a sprint backlog there is no budget of story points to be brought into a sprint. This in turn means that estimation exercises such as planning poker lose much of their significance. In a Lean Kanban operation velocity can be measured not in terms of story points - either estimated or actual - but simply as the number of tickets actioned over a set period. This also provides an indication of the lead time before a ticket is handled. If tickets are of too variable a size - for example, if they include small ones as well as larger templated changes - then they can be awarded points for how long, or how much effort, they took. T-Shirt sizes is one approach. Remember that these points should represent the actuals, not estimates, so there's still no need for planning poker. Velocity can be averaged for each size. Alternatively the sizes can be mapped to points (e.g. small = 1, medium = 3, large = 7) and an aggregate velocity calculated. "Some of the BAU work that's been coming through looks like project work to me" You could well be right. It's important that you raise your suspicions with your team lead. There's often politics involved, but here's the lowdown. In many organizations "Business as Usual" work is classed - you could almost say "written off" - as an operational expenditure (OpEx), and is not drawn from the capital expenditure (CapEx) assigned to projects. Internal customers often have an incentive to sneak through initiatives as BAU work so as not to incur capital expense on their departmental budgets. This is indeed a political issue. But be on your guard otherwise your team could be hobbled with project work being slipped in on the sly. Be particularly wary of significant numbers of related changes, large changes, a seemingly high level of risk with any work items, or changes of uncertain scope. These suggest, but do not prove, that a fast one might be being pulled. Your team lead (who is analagous to a ScrumMaster) should try and defend against this, so if you as a team member have your suspicions, it's important to bring them to your lead's attention. Conclusion, and what's next In this post we've looked at the important differences between Lean Kanban and Scrum, and what that means for a team. We've also reviewed how a reasonably informed choice can be made between them. In my next post we'll look at a hybrid approach known as ScrumBan which can potentially address both project and BAU work. ScrumBan is becoming increasingly popular and has significant ramifications for project scalability.

When I’m teaching training courses (as I was this week at Skills Matter) or advising clients on the requirements-side of software development (which I’m doing a lot of just now), I talk about a model I call the “3 Styles of Agile.” Incredibly, I’ve never blogged about this -- although the model is hidden inside a couple of articles over the years. So now the day has come… I don’t claim the “3 Styles Model” is the way it should be, I only claim that it is the way I find the world. While “doing Agile” on the code side of software development always comes back to the same things (stand-up meetings, test/behavior driven development, code review/pair programming, stories, boards, etc.) the requirements side is very very variable. The advice that is given is very variable and the degree to which that advice is compatible with corporate structures and working is very variable. However, I find three reoccurring styles in which the requirements side operates and interfaces to development. I call these styles: Iterative, Incremental and Evolutionary, and I usually draw this diagram: I say style because I’m looking for a neutral word. I think you can use Scrum, XP and Kanban (or any other method) in any of the three styles. That said, I believe Kanban is a better fit for evolutionary while Scrum/XP are a better fit for Iterative and Incremental. I try not to be judgmental, I know a lot of Agile folk will see Evolutionary as superior, they may even consider Evolutionary to be the only True Agile but actually I don’t think that is always the case. There are times when the other styles are “right.” Let me describe the three styles: Iterative In this style the development team is doing lots of good stuff like: stand up meetings, planning meetings, short iterations or Kanban flow, test driven development, code review, refactoring, continuous integration and so on. I say they are doing it but it might be better to say “I hope they are doing” because quite often some bit or other is missing. That’s not important for this model. The key thing is the dev team are doing it! In this model, requirements arrive in a requirements document en mass. In fact, the rest of the organization carries on as if nothing has changed, indeed this may be what the organization wants. In this model you hear people say things like “Agile is a delivery mechanism” and “Agile is for developers." The requirement document may even have been written by a consultant or analyst who is now gone. The document is “thrown over the fence” to another analyst or project manager who is expected to deliver everything (scope, features) within some fixed time frame for some budget. Delivery is most likely one “big bang” at the end of the project (when the team may be dissolved). In order to do this they use a bacon slicer. I’ve written about this before and called it Salami Agile. The requirements document exists and the job of the “Product Owner” is to slice off small pieces for the team to do every iteration. The development team is insulated from the rest of the organization. There is probably still a change review board and any increase scope is seen as a problem. I call this iterative because the team is iterating but that’s about it. This is the natural style of large corporations, companies with annual budgets, senior managers who don’t understand IT and in particular banks. Incremental This style is mostly the same as Iterative, it looks similar to start with. The team are still (hopefully) doing good stuff and iterating. There is still a big requirements document, the organization still expects it all delivered and it is still being salami sliced. However, in this model, the team is delivering the software to customers. At the very least, they are demonstrating the software and listening to feedback. More likely, they are deploying the software and (potential) users can start using it today. As a result, the customer/users give feedback about what they want in the software. Sometimes this is an extra feature and functionality (scope creep!) and sometimes it is about removing things that were requested (scope retreat!). The “project” is still done in the traditional sense that everything in the document is “done,” but now some things are crossed out rather than ticked. Plus some additional stuff might be done over and above the requirements document. I call this incremental because the customers/users/stakeholders are seeing the thing grow in increments -- and hopefully early value is being delivered. I actually believe this is the most common style of software development -- whether that work is called Agile, waterfall or anything else. However, in some environments this is seen as wrong, wrong because the upfront requirements are “wrong” or because multiple deliveries need to be made, or because the team aren’t delivering everything they were originally asked to deliver. Evolutionary Here again the development team are iterating much as before. However, this time there is no requirements document. Work has begun with just an idea. Ideally I would want to see a goal, an objective, an aim, which will guide work and help inform what should be done -- and this goal should be stated in a single sentence, a paragraph at most. But sometimes even this is missing, for better or worse. In this model the requirements guy and developers both start at the beginning. They brainstorm some ideas and select something to do. While Mr. Requirements runs off to talk to customers and stakeholders about what the problem is and what is needed, the tech team (maybe just one person) gets started on the best idea so far. Sometime soon (2 weeks tops) they get back together. Mr. Requirements talks about what he has found and the developers demonstrate what they have built. They talk some more and decide what to do next. With that done, the developers gets on with building and Mr. Requirements gets on his bike again, he shows what has been built and talks to people -- some people again and some new people. As soon as possible the team starts to push software out to users and customers to use. This delivers value and provides feedback. And so it goes. It finishes, if it finishes, when the goal is met to the organization decided to use its resources somewhere else. Evolutionary style is most at home in Palo Alto, Mountain View, and anywhere else that start-ups are the norm. Evolutionary is actually a lot more common than is recognized but it is called maintenance or “bug fixing” and seen as something that shouldn’t exist. Having set out the three styles I’ll leave discussion of how to use the model and why you might use each style to another entry. If you want to know more about each model and how I see Agile as spectrum have a look my 2011 “The Agile Spectrum” from ACCU Overload or the recently revised (expanded but unfinished) version by the same title: “Agile Spectrum” (the 2013 version I suppose, online only).

There are two widely opposed (and often misunderstood) positions on how expensive it can be to change or fix software once it has been designed, coded, tested and implemented. One holds that it is extremely expensive to leave changes until late, that the cost of change rises exponentially. The other position is that changes should be left as late as possible, because the cost of changing software is – or at least can be – essentially flat (that’s why we call it software). Which position is right? Why should we care? And what can we do about it? Exponential Cost of Change Back in the early 1980s, Barry Boehm published some statistics (Software Engineering Economics, 1981) which showed that the cost of making a software change or fix increases significantly over time – you can see the original curve that he published here. Boehm looked at data collected from Waterfall-based projects at TRW and IBM in the 1970s, and found that the cost of making a change increases as you move from the stages of requirements analysis to architecture, design, coding, testing and deployment. A requirements mistake found and corrected while you are still defining the requirements costs almost nothing. But if you wait until after you've finished designing, coding and testing the system and delivering it to the customer, it can cost up to 100 times as much. A few caveats here. First, the cost curve is much higher in large projects (in smaller projects, the cost curve is more like 1:4 instead of 1:100). Those cases when the cost of change rises up to 100 times are rare, what Boehm calls Architecture-Breakers, where the team gets a fundamental architectural assumption wrong (scaling, performance, reliability) and doesn't find out until after customers are already using the system and running into serious operational problems. This analysis was all done on a small data sample from more than 30 years ago, when developing code was much more expensive and time-consuming and paperworky, and the tools sucked. A few other studies have been done since then that mostly back up Boehm's findings – at least the basic idea that the longer it takes for you to find out that you made a mistake, the more expensive it is to correct it. These studies have been widely referenced in books like Steve McConnell’s Code Complete, and used to justify the importance of early reviews and testing: Studies over the last 25 years have proven conclusively that it pays to do things right the first time. Unnecessary changes are expensive. Researchers at Hewlett-Packard, IBM, Hughes Aircraft, TRW, and other organizations have found that purging an error by the beginning of construction allows rework to be done 10 to 100 times less expensively than when it's done in the last part of the process, during system test or after release (Fagan 1976; Humphrey, Snyder, and Willis 1991; Leffingwell 1997; Willis et al. 1998; Grady 1999; Shull et al. 2002; Boehm and Turner 2004). In general, the principle is to find an error as close as possible to the time at which it was introduced. The longer the defect stays in the software food chain, the more damage it causes further down the chain. Since requirements are done first, requirements defects have the potential to be in the system longer and to be more expensive. Defects inserted into the software upstream also tend to have broader effects than those inserted further downstream. That also makes early defects more expensive. There’s some controversy over how accurate and complete this data is, how much we can rely on it, and how relevant it is today when we have much better development tools and many teams have moved from heavyweight sequential Waterfall development to lightweight iterative, incremental development approaches. Flattening the Cost of Changing Code The rules of the game should change with iterative and incremental development – because they have to. Boehm realized back in the 1980s that we could catch more mistakes early (and therefore reduce the cost of development) if we think about risks upfront and design and build software in increments, using what he called the Spiral Model, rather than trying to define, design and build software in a Waterfall sequence. The same ideas are behind more modern, lighter Agile development approaches. In Extreme Programming Explained (the first edition, but not the second) Kent Beck states that minimizing the cost of change is one of the goals of Extreme Programming, and that a flattened change cost curve is “the technical premise of XP”: Under certain circumstances, the exponential rise in the cost of changing software over time can be flattened. If we can flatten the curve, old assumptions about the best way to develop software no longer hold … You would make big decisions as late in the process as possible, to defer the cost of making the decisions and to have the greatest possible chance that they would be right. You would only implement what you had to, in hopes that the needs you anticipate for tomorrow wouldn't come true. You would introduce elements to the design only as they simplified existing code or made writing the next bit of code simpler. It’s important to understand that Beck doesn't say that with XP the change curve is flat. He says that these costs can be flattened if teams work toward this, leveraging key practices and principles in XP, such as: Simple Design, doing the simplest thing that works, and deferring design decisions as late as possible (YAGNI), so that the design is easy to understand and easy to change Continuous, disciplined refactoring to keep the code easy to understand and easy to change Test-First Development – writing automated tests upfront to catch coding mistakes immediately, and to build up a testing safety net to catch mistakes in the future Developers collaborating closely and constantly with the customer to confirm their understanding of what they need to build and working together in pairs to design solutions and solve problems, and catch mistakes and misunderstandings early Relying on working software over documentation to minimize the amount of paperwork that needs to be done with each change (write code, not specs) The team’s experience working incrementally and iteratively – the more that people work and think this way, the better they will get at it. All of this makes sense and sounds right, although there are no studies that back up these assertions, which is why Beck dropped this change curve discussion from the second edition of his XP book. But, by then, the idea that change could be flat with Agile development had already become accepted by many people. The Importance of Feedback Scott Amber agrees that the cost curve can be flattened in Agile development, not because of Simple Design, but because of the feedback loops that are fundamental to iterative, incremental development. Agile methods optimize feedback within the team, developers working closely together with each other and with the customer and relying on continuous face-to-face communications. Following technical practices like test-first development, pair programming and continuous integration makes these feedback loops even tighter. But what really matters is getting feedback from the people using the system – it’s only then that you know if you got it right or what you missed. The longer that it takes to design and build something and get feedback from real users, the more time and work that is required to get working software into a real customer’s hands, the higher your cost of change really is. Optimizing and streamlining this feedback loop is what is driving the lean startup approach to development: defining a minimum viable product (something that just barely does the job), getting it out to customers as quickly as you can, and then responding to user feedback through continuous deployment and A/B testing techniques until you find out what customers really want. Even Flat Change Can Still Be Expensive Even if you do everything to optimize these feedback loops and minimize your overheads, this still doesn’t mean that change will come cheap. Being fast isn’t good enough if you make too many mistakes along the way. The Post Agilist uses the example of painting a house: Assume that it costs $1,000 each time you paint the house, whether you paint it blue, red or white. The cost of change is flat. But if you have to paint it blue first, then red, then white before everyone is happy, you’re wasting time and money. “No matter how expensive or cheap the "cost of change" curve may be, the fewer changes that are made, the cheaper and faster the result will be … Planning is not a four letter word.” (However, I would like to point out that “plan” is.) Spending too much time upfront in planning and design is waste. But not spending enough time upfront to find out what you should be building and how you should be building it before you build it, and not taking the care to build it carefully, is also a waste. Change Gets More Expensive Over Time You also have to accept that the incremental cost of change will go up over the life of a system, especially once a system is being used. This is not just a technical debt problem. The more people using the system, the more people who might be impacted by the change if you get it wrong, the more careful you have to be. This means that you need to spend more time on planning and communicating changes, building and testing a roll-back capability, and roll changes out slowly using canary releases and dark launching – which add costs and delays to getting feedback. There are also more operational dependencies that you have to understand and take care of, and more data that you have to change or fix up, making changes even more difficult and expensive. If you do things right, keep a good team together and manage technical debt responsibly, these costs should rise gently over the life of a system – and if you don’t, that exponential change curve will kick in. What is the real cost of change? Is the real cost of change exponential, or is it flat? The truth is somewhere in between. There’s no reason that the cost of making a change to software has to be as high as it was 30 years ago. We can definitely do better today, with better tools and better, cheaper ways of developing software. The keys to minimizing the costs of change seem to be: Get your software into customer hands as quickly as you can. I am not convinced that any organization really needs to push out software changes 10 to 50 to 100 times a day, but you don’t want to wait months or years for feedback, either. Deliver less, but more often. And because you’re going to deliver more often, it makes sense to build a continuous delivery pipeline so that you can push changes out efficiently and with confidence. Use ideas from lean software development and maybe Kanban to identify and eliminate waste and to minimize cycle time. We know that, even with lots of upfront planning and design thinking, we won’t get everything right upfront -- this is the Waterfall fallacy. But it’s also important not to waste time and money iterating when you don’t need to. Spending enough time upfront in understanding requirements and in design to get it at least mostly right the first time can save a lot later on. Whether you’re working incrementally and iteratively, or sequentially, it makes good sense to catch mistakes early when you can, whether you do this through test-first development and pairing, or requirements workshops and code reviews -- whatever works for you.

In a complex system such as a software development team, it's easy for fear to arise.

i went to the local capital kanban meetup yesterday evening. it was a bunch of project managers discussing kanban and waste in it. seemed completely out of my comfort zone and a way to meet new people in tech here in town so i attended. it turned out to be really cool and way more interesting than my expectations were. i wanted to mention some of those here, specifically some of the it wastes that were mentioned i see all the time, the insights i got from the paper airplane factory game, and some after meeting talk that changed my perspective on what i perceive as problems in our industry with good software east of california (hah, trick question, there is no good software done east of california…). it wastes andrea ross did a presentation about waste in it. kanban, the production process used by toyota then turned into a project management and software development, has 7 or 8 forms of what it calls waste. these are primarely in the factory line production process, so you have to draw your own metaphors and similes, and that’s what andrea’s presentation extrapolated on. from a high level, these are: defects / rework overproduction waiting non-standard over processing transportation / logistics intellect motion excess inventory her slides that have bullet point examples for each one are pretty self-explanatory. what was interesting to me was the sheer volume of bullet points i see all the time, together, in the same projects i work on. some can’t be avoided, nature of the business and all that. still, it was pretty eye opening to see that a traditional factory production process has identified these items as the core waste items, and software development has plenty of them with just about the same meanings. i won’t cover them in detail here as her slides do. kanban, bottlenecks, and waste the concept of kanban is to quickly identify bottlenecks in the existing production process, and iterate to improve the process to fix them. notice i said “existing” and “process”. the existing part is where kanban has been easier to market than say six sigma which is bought into wholesale, hence why it’s easier to be a six sigma consultant than a kanban one. kanban you basically overlay on top of what you have and it surfaces the problems your existing process has pretty clearly. meaning, if you see a bunch of cards on a kanban board that are in the “analysis” column, and very few in the rest, it’s pretty clear where the bottle neck is located. now the “process” part is analogous to the production line; in this case all that goes into making software from the traditional waterfall perspective: design, development, deployment. however, the key here is you aren’t fixing the “bottleneck”, but rather the process itself. that is what i learned through our paper airplane factory exercise quite clearly. there are a series of games like this that can be modified, but the point is they help teach the bottleneck vs. process modification process extremely clearly. the key takeaway for me was fixing the bottleneck, like the 5 developers + 1 manager in a war room during a troubling moment during a software project, is actually a form of waste. yes, it’s great teams rise up to tackle these problems in the moment. however, it’s important to note that it’s the project manager’s job to both recognize this as waste and fix the actual process problem. i’ll explain this below. note: if you’re concerned about spoilers, please be aware of 2 things. first, there are more than just the airplane factory game that you can find online. second, if you do read the following section and later participate in the exercise, please either let the teacher/presenter know, or try not to modify the process too much to allow others to learn. airplane factory the game is like so (abridged version, you can find the full instructions here ): divide your people into 4 groups, each sitting adjacent to each other. circle or semi-circular seating arrangements works best to encourage intentional bottleneck adjustment engagement. cut up the airplane folding instructions alone the designated lines. give the first part of the instructions to the first group in the line. give the second part of the instructions to the second group, and repeat on down the line. some people may not necessarily have designated jobs beyond passing papers, etc. this is intentional to illustrate the intellect waste of not using human ip… and also to note how they’ll often become efficient passers, helpers, or even qa. ensure the group/person who’s last in line is aware of how far the plane must fly as a metric of defining a successful plane. setup a 5 minute timer, start it, and yell “go!” after 5 minutes, identify how many successfully flown airplanes were made as well as how much waste (crumpled papers, non-flying planes, etc) were created. that’s the round 1 score. iterate. the iterate step is where you reflect on what just happened and attempt to modify the process. i’ll go over how ours went down so you get an idea. round 1 line setup : we had 8 people in our line. round 1, we had 1 lady do the half fold, the 2nd guy do the additional 2 folds + paper sides cut, andrea and i pass the paper to our left, another lady handle the 1st wing folds, and a gentleman at the end to make the wings and throw it. our last person was a lady who handled qa and scoring. process : very quickly we had a bottleneck with the lady at my stations left. the instructions weren’t very clear and she struggled to learn how to do the first one. both andrea and i quickly went to help; andrea attempting to do it with her, me taking a picture of the instructions with my iphone, and attempting to duplicate at my desk while ensuring i kept passing the planes to my left into an ever growing pile. once i figured it out (i had actually built the exact same plane last week for my daughters birthday present which has an electronic propeller you attach), i told the ladies to ignore the “requirements” as they were crap and i walked them through how to successfully complete their step. i then quickly returned to my desk which had a pile of unmoved inventory. the only person who didn’t struggle with their assembly was the 1st in the line who had to fold paper in half. i believe we ended up with 2 planes and 1 waste. takeaway : our teacher quickly pointed how we went “downstream” to fix the production line process. this is a reactionary, and completely normal mode, to fix production line problems. it’s also wrong. you’re supposed to identify the upstream problem causing it and fix that. additionally, we didn’t stop the line to ensure we fixed this problem first before continuing, also wrong. car companies like toyota do this via a chain that’s pulled to stop the line so they can ensure the problem is resolved. sometimes they even take part of their line off the main line to ensure things keep moving. as a side note, apparently gm used to keep going. door doesn’t fit right? keep going; jam the mofo on there. they’d end up with a lot full of cars pretty quickly… even if they were low quality. ford was similar, but they’d ensure the cars were actually sold first before they sold them, thus not resulting in lots full of inventory they couldn’t sell like gm… even if the quality of pre-sold cars was still low. we also noted various other problems such as no training in each plane’s building instructions, no one stopping if the station after them go overwhelmed with their ever growing stack, and sometimes idle resources (people with not much to do). round 2 improvements : first, the teacher actually implemented designated stack areas with a piece of paper on each station, and then wrote a number on the paper; this was the max amount of completed planes you could place for the next station to build so you didn’t exacerbate a bottleneck. second, i become a designated passer to my left while my partner moved to the left station to have a 2 women team doing the complicated folding. process : my job was pretty easy; pass, and ensure i don’t bottleneck it. every single station was faster since they had practiced their portion. the guy to my right had actually done his first 3 paper cuts wrong in round 1 which caused confusion to my left station, but had it down pat in round 2. the last station still experimented with various angles of folding to see how far the plane could actually fly. we actually had the last women in the line, qa, send a messed up plane back through the line as an unfolded piece of paper because it didn’t fly right; w00t, less waste! a bottleneck, again, formed to my left, but the girls found a way to divide up the 2 step process between them to be more efficient. as our 5 minutes progressed, they got faster and eventually started making progress on the backlog. we made 5 planes, 1 waste. takeaway : we built 4 planes with 1 waste. the first person, as usual, was too fast. the guy to my right had an inefficient process because he’d have to fold, pick up the scissors, cut, then put ‘em down again. we had all abandoned our airplane instructions by this point. round 3 improvements : everything else was fine, so we decided to give me the cutting job and the guy to my right would just fold. the girls to my left on-the-fly modification was good and we kept it. process : my first 3 were slow, but once i practiced, i was uber fast and we were humming. the girls to my left were killing it. i managed to keep my right stack always below or at 2 in the pile. very quickly it became apparent the 1st person was too fast; she was constantly folding and then waiting before she was allowed to make another. we made 8 airplanes, no waste. takeaway : those of with spouses were already getting texted like mad to leave, but we all wanted to see round 3 succeed better than 2, and see if we made the process perfect. we didn’t; it went the complete opposite direction to the front of the line needing minor modifications. overall, though, our output increased a lot, our waste went down, and it was very clear that the adjustments we made + the teachers maximum stack amounts were working well. my overall takeaways i went to this meeting to both meet new people in town to network with as well as to get out of my comfort zone. i find when i do the latter, i learn a lot and sometimes get a new perspective. it gave me a new appreciation for project managers who have not just 1, but 5 projects they have to manage to make an attempt to do this on. this also assumes they get enough time to really learn about each teams issues, where those bottlenecks are, and what the best ways are to address them. not by just fixing the bottlenecks, but by fixing the process itself, ensuring stop guards are in place not as many items/cards in a column, etc. it also made me intimately aware of how i, as a consultant, immediately want to fix the bottleneck, and have learned ways (such as the war room) to solve them… when in reality, it’s a pm issue for a greater process problem. the other thing that makes it more complex is the whole “all things being equal”. for example, a kanban board a pm would use on the whole process vs. just the kanban board my software team would use. if my team fails to do tdd and ends up with a variety of bugs in the system because we’re forced to develop quickly and produce bad work, this show up on hers as us being the bottleneck. without time to talk to us and really empower us to change our process, nothing will change. i see this time and time again. the excuses, which are sometimes valid, range from “the software’s good enough even with the bugs”, or “tdd is too much work for not enough value” or “we can’t write a test suite for a huge mess that isn’t even testable”. …and that’s just a small portion of what i’ve seen gone wrong. if you’ve ever worked for a design agency, or even a large firm that has a huge new client, it’s very apparent many teams have a hard time getting sign off from clients which causes a bottleneck in the analysis column on the kanban board because the items either pile up, or priority constantly shifts… yet they never actually make it out of their column. a pm there who works with the government offered his strategies for dealing with the strange qa cycles government agencies will have where it goes into a black hole for 6 weeks thus really screwing up his kanban metrics. overall, it was neat to be in a room with people who were geeking out on improving process. you see a lot of software developers get bored with programming or frustrated with how their lack of process is going, so they read up on xp and agile. when you look at what these pm’s deal with, it makes you feel like just a small part in a larger overall process. more importantly, my preconceptions about leadership being the problem 99% of my problem projects really had a wrench thrown in. i was bitching about it to one of the pm’s, and quickly explained, in great detail, why some big companies which don’t have a hard line metric such as money to predict performance will often use lean methodologies since “ensuring customer satisfaction” is hard to measure depending on your business, and requires a more exploratory way of doing business. that said, it was great to hear that the common problems i experience in software dev with solutions were the same, just 1 of many that pm’s have to deal with. i highly encourage software developers to partake in one of these exercises, even if you do scrum vs. kanban. really eye opening stuff.

We’re all works in progress, honey. And believe me when I tell you that I’ve had to work harder than most. ― Susan Elizabeth Phillips, "Ain't She Sweet" It's pretty well understood that limiting Work In Progress - or WIP as it is often abbreviated - is a good thing. Ideally, WIP should be limited to one item in progress at a time. Having multiple pieces of inventory on-hand is a form of waste, since each incurs a handling cost, and any work done on one of them will depreciate while another is being worked on. In theory at least, restricting WIP to one item at a time will reduce this waste and get value out of the door as quickly as possible. This principle of Single Piece Flow (SPF) is central to Lean-Kanban ways of working, especially in a manufacturing context. In a software context the accepted WIP limits tend to be rather higher. It is often limited to one item per developer, such as by allowing each developer only one avatar to place on an item, and it can be reduced further if pair-programming is in use. As such, software teams might not often achieve SPF but the value of limiting WIP as far as possible is still understood. There are however problems in interpreting limited WIP in Scrum. This is because a Scrum board will often take the form of a task board ... not a Kanban board. In other words, the work being limited by Scrum teams is not always a user story or similar requirement. It is often a task. Task-limited WIP allows developers to progress tasks from any user story in any order. They could potentially limit themselves to one or two tasks from a story, complete them, then move on to a task from a different story and maybe a task from a third. In effect multiple stories - perhaps even the entire Sprint Backlog of stories - can be in progress before so much as one story gets completed. None of this breaks Scrum rules. There's nothing to stop a team, in Sprint Planning, from organizing the Sprint Backlog into any number of tasks which can be progressed in any order they choose, and from delivering all of the user stories in one go at the end of the Sprint. The Sprint Goal can of course be met by this approach, and there should still be a nice task burn-down to show the associated technical risks being managed. The problem is that it defers approval of each user story to the end of the Sprint (i.e. the Sprint Review), when it is best-practice to get continual sign-off by a Product Owner throughout the iteration. On-going inspection allows the business risks of delivery to be managed well, and not just the technical risks. This is an issue that all Scrum teams must consider when they formulate a Sprint Plan. Is it important to limit WIP in terms of user stories rather than tasks, and thereby facilitate early approval of those stories by a Product Owner? Or would this compromise the team's principle of incremental delivery ... and amount to Lean-Kanban by the back door?

Remembrance and reflection, how allied; What thin partitions sense from thought divide. - Alexander Pope Retrospectives, and why you need them A couple of months ago we looked at how to conduct a Sprint Review. We saw that a Review considers what work was done, and distinguished it from a Sprint Retrospective which reflects upon how work is being done. The distinction between the two can appear to be rather academic, and slurring a Review and a Retrospective together is a mistake that is often made by immature teams. After all, both take a reflective view of a Sprint that has just finished, and both can be said to fulfill a remit of historical inquiry. Yet while the separation of concerns might seem to be a narrow one, it is nonetheless quite precise, and there is great value to be had in maintaining the appropriate focus. A Review looks candidly at what has been achieved, and soberly at what remains to be achieved, with regard to product completion. A Retrospective on the other hand is an opportunity for the Scrum Team to inspect and adapt their actual implementation of the Scrum process. The rationale behind this inspection is methodological but it is in no sense abstract. It is grounded firmly in the desire to achieve worthwhile and practical reform. Perhaps there are certain working practices which the team can make more efficient, or which can otherwise be improved upon. If so, a Retrospective presents the ideal opportunity for those improvements to be discussed and brought into action. Failing to inspect and adapt in this manner will condemn a team to perpetual infancy and the repetition of past mistakes. Sprint Retrospectives help keep a Scrum team on the road to continual improvement. When these sessions are done well, team members will not be afraid to challenge the status quo, and will do so in a constructive and informed manner. The result will be an improved delivery of value – in fact, the sort of productivity gain that might well be identified in the Sprint Review we considered earlier. In this article we’ll switch our attention fully to Retrospectives, and examine the matter of how they should be conducted. Setting up a Retrospective As any event manager will tell you, the key to a successful gig lies in the preparation. Okay…I’ll concede that a holding a Retrospective isn’t as mammoth an undertaking as hosting the Thinking Digital conference, nor can it be said to demand the organizational skills of Bruce Springsteen’s road manager. Nevertheless it’s still important to get a few ducks in a row. Let’s start by lining them up and giving them some admittedly rather unimaginative names: Why, Who, Where, When, and What. We’ve just covered the issue of why a Retrospective needs to be held…that duck’s down. Let’s pop the rest. Who should attend a Sprint Retrospective? The invitation list for a Sprint Retrospective should be simple and uncontroversial. According to the Scrum Guide all Scrum Team members are expected to attend. That’s the Developers, the Scrum Master (who may facilitate the session), and the Product Owner. No others are expected. In fact, it would be quite irregular to extend the invitation to other people, even if they consider themselves to be important players or stakeholders. That’s because it is the Scrum Team who are responsible for the way they have implemented the Scrum Framework. Only they are in a position to inspect and adapt their very own ways of working. For this reason, all members have a duty to be present, to contribute, and to help make each Retrospective a success. Some teams exclude the Product Owner from this activity, arguing that if he or she was present, the team would not be able to have an open and frank discussion. This is a common issue and we’ll return to it later. For now though, just take it as read that a good Retrospective must include all Scrum Team members, and will give each a voice in molding the processes and working environment that they collectively own. Where should a Retrospective be held? Let’s answer this one with another question. If all of the Scrum Team members are co-located, and if they have the necessary equipment to hand (such as their Scrum board, plus a whiteboard for notes), why not hold the retrospective in situ? In other words, why not just hold the session at the team’s desks? Well, although this might sound like a capital idea, there can be problems. Perhaps it would create too much of a disturbance and annoy other teams within earshot? Then again, perhaps the physical layout of the working area is simply not conducive to holding a meeting. Perhaps the team is not entirely co-located in the first place. Any one of these things can tip the balance in favour of booking an actual meeting room, and getting everyone to decamp there for a Sprint Retrospective. If so, remember to book such a room in advance…if possible as a recurring appointment for the anticipated duration of the project. Make sure it has sufficient capacity and the resources needed. When should a Retrospective happen? The glib answer is to say that a Retrospective should happen “at the end of each Sprint”. A more useful answer would say whether or not it should precede or follow the Sprint Review. In my experience it is generally better to do the Review first, because that helps to establish a context within which the Retrospective can happen. The next thing to consider is how long to allow for the session. As with all Scrum events, a Sprint Retrospective is time-boxed. This means that it isn’t allowed to exceed a set length. The rules of Scrum are exact: for a one month Sprint the limit for a Retrospective is 3 hours, which is reduced to one-and-a-half hours for a two week Sprint. You should adjust this value by the same ratio if needed. Note that if a Retrospective finishes before the time-box expires, that’s fine and dandy. You aren’t obliged to use all of the available time. The rule is simply that the time-box must never be exceeded. Scrum is not a philosophy in which matters are allowed to drag on. What topics should the Retrospective cover? This is the biggest duck in the row, and it’ll take a few pings to knock it down. What we have to do is to establish a suitable agenda for a Sprint Retrospective. We have to formulate it in such a way that the inspection of the team’s Scrum implementation does indeed happen. We also have to make sure that any recommendations for its adaptation are elicited, agreed, and turned into achievable action items. The Scrum Guide provides us with something of a starting point. It isn’t much, but I reckon that if you look at it through a beer glass with your head sideways and one eye closed, you can just about discern a notional agenda for holding a Sprint Retrospective. A notional agenda The Scrum Guide is sparing in the advice it gives on how to conduct a Retrospective. We are told that a Scrum Team must: Inspect how the last Sprint went with regards to people, relationships, process, and tools; Identify and order the major items that went well and potential improvements; and, Create a plan for implementing improvements to the way the Scrum Team does its work…[including]…ways to increase product quality by adapting the Definition of “Done” as appropriate. Yes, I know that’s not much to go on, but each of these items is clearly significant. They seem to address the very rubric of agile practice; we can recognize in them a succinct appeal to the three legs of Transparency, Inspection, and Adaptation. In them, we can see not only a notional agenda, but also how critical a Sprint Retrospective is to the Scrum process. A Retrospective is arguably the most important time-boxed event that any agile process can have. If we want to turn these points into a more formal agenda for the session, we’ll have to make sure that each of them is addressed carefully. Towards a canonical format Scrum has been around for well over a decade now, and a fairly standard agenda for conducting a Sprint Retrospective has emerged. Here’s what it looks like. Set the scene. Ways to do this can include any or all of the following: Sketching out a timeline of significant events that occurred in the Sprint, so its historical context can be established Holding the Sprint Review shortly beforehand, so the project context is fresh in attendees’ minds Declaring the Prime Directive in order to define a professional context of mutual respect and openness Assess prior action items. Unless this is the first sprint, there will have been an earlier retrospective in which some improvements will have been proposed. Look back over each of them. Have they been followed through? In short, has the process actually been adapted following that earlier inspection? If any action items remain undone, make a note of them. They’ll have to be considered when determining actions for the future. Set up a Retrospective Board. This can be a whiteboard, or even a large sheet of paper stuck to a wall. Divide it into four quadrants and label each in the following manner. The precise terminology does tend to vary a bit. There can be subtle and not-so-subtle differences in meaning (consider the difference between “good points” and “things to continue doing”). Be aware of these differences, as they will shape the responses and ultimately the results. “What went well” (or “good points”, or “things to continue doing”) “What didn’t go well” (or “bad points”, or “things to stop doing”) “Ideas for improvement” (or things to “start doing”) “Shout-outs” (i.e. recognition of noteworthy individual contributions) Storm the Board. There are several ways in which this can be done. Here are some of the more common ones: Sticky notes. This method is fairly democratic in that each attendee gets a clear say by putting sticky notes on a board. Assertive individuals are therefore less able to dominate others. However, it can be disjointed as attention shifts from one person’s topics to another person’s. As such, it can be hard to develop a line of thought for group discussion. Here’s the process: Blocks of notes are distributed to the attendees. They are given a small time-box (5 or 10 minutes) to jot down their ideas…good points, bad points, improvements, and shouts. Each attendee should write one point per sticky note. There is no limit to the number of points they can make. After the time is up, attendees take it in turn to put their notes on the board and in the relevant quadrants As an attendee puts their sticky note on the board, they briefly state what the point is to the rest of the team Once the last attendee has finished, duplicate points will be identified by the group and removed. Facilitator-as-arbitrator. In this approach a facilitator will act as a scribe for the group, and write their ideas on the board. Group discussion of ideas is encouraged, and the facilitator can arbitrate in the event of disagreement. The downside is that it can favor the more assertive type of individual who ends up doing most of the talking. Here’s how it’s done: The facilitator stands in front of the board with a marker pen Any attendee who has a suggestion to make will make it – a good point, bad point, idea, or shout-out The facilitator writes each suggestion into the appropriate quadrant, disallowing any duplicates. The group discuss the merits of each suggestion The facilitator will erase, keep, or revise each suggestion according to group opinion Hybrid. This uses a mix of techniques, such as a facilitated session for identifying good points and bad points, and a sticky-note approach in order to elicit ideas for improvement. Changing the techniques used in a Retrospective every now and then can help keep the sessions fresh, and is certainly a good idea if you reckon they are getting a bit stale. Propose actions. I have five rules that I apply when “storming the board” with a team: For every bad point there must be an idea for improvement. In other words, for everything that people are being asked to stop doing an alternative and better course of action must be proposed. This rule helps to keep attendees focused on the need to adapt the process constructively, and not to use the session for mere complaint. If you have been storming for “good points” rather than for things to “start doing”, make sure that each of those points is matched with an idea for further improvement. It isn’t enough to look back appreciatively whenever something positive has happened. Your challenge is to translate that observation into an even bigger future win. Re-assess undone action items from the previous Retrospective. If any remain undone, ask if they are worth bringing forward. Ask why they weren’t implemented, with a view to finding out what really needs to happen to expedite them. If these outstanding actions are impractical, or are no longer relevant, jettison them and concentrate on those improvements which are valuable and achievable. Ask the “Five Whys”. For each action item you produce, you need to be sure that you have understood the root cause and that the action will be appropriate. A shallow retrospective is no retrospective at all. It has to be deep and probing. Improve the Definition of Done. The Scrum Guide doesn’t provide much advice about holding Retrospectives, but it is quite clear about the need to revisit the Definition of Done. This is something that many teams, including some quite experienced ones, forget or otherwise fail to do. So be careful to identify any room for improvement in the team’s understanding of what “done” means, and what it should take for work to be considered potentially releasable. Vote. It’s quite possible that the list of proposed actions will be extensive. In aggregate they could amount to too much change if all were to be implemented in the forthcoming Sprint. You can resolve this by getting team members to vote on action items, so that only the most important ones are taken forward. For example, if the team can take forward five items, allow each attendee to vote for five of them. The most popular can then be actioned. Other observations Here are some other things to consider when conducting a Sprint Retrospective. Decide whether or not to precede it with the Scrum “Prime Directive”. This is an assertion which is meant to be said, in earnest, before each and every Retrospective. It isn’t mentioned in the Scrum Guide, but it is widely recognized and some teams do choose to recite it. “Regardless of what we discover, we understand and truly believe that everyone did the best job they could, given what they knew at the time, their skills and abilities, the resources available, and the situation at hand” We considered the significance of this assertion in an earlier article on Agile Teamwork in Practice, so I’m not going to say much more about it here. However, Martin Fowler has expressed his thoughts on the Prime Directive, and I suggest you read his opinion piece in full. All I’ll add is that I am in agreement with his observations and that I share his sense of revulsion. Determine what to do about Product Owner representation. According to the Scrum Primer the Product Owner may attend a Sprint Retrospective. Only “Development Team” members are actually required to be there. Yet according to the Scrum Guide, all “Scrum Team” members must attend. The Scrum Team is a wider group than the Development team and includes the Product Owner. The reason for this discrepancy probably lies in the interpretation of process ownership. If we see the Development Team as owning the process through which iterative and incremental value will be delivered to a Product Owner, then the PO would not indeed have a say in the adaptation of that process. He or she would merely be a consumer of its outputs, and would therefore be a stakeholder in a Sprint Review but not in a Sprint Retrospective. However, if we view the process as a more collaborative one, in which the Development Team works with the Product Owner to deliver potentially releasable increments of value every Sprint, then the PO would indeed be a stakeholder in how that process is managed, and must therefore attend. It’s therefore important to determine what relationship the Development Team has, or should have, with the Product Owner. It’s unquestionably best if a Product Owner is on-side as a team player, and can handle root cause analysis and the exposure of potentially uncomfortable truths. Whether or not that is the case though is only something that the team can decide. Remember they’re human. Bring snacks and drinks to keep attendees refreshed, and allow enough time for breaks – at least 10 minutes every hour. Consider wrapping up the session with a “touchy feely graph” of some sort, which captures the mood and confidence of the team. Allow everyone to mark a dot or cross on a chart to show how positive or negative they feel about things, and then see how the mood changes…hopefully for the better…from one Sprint to the next. Conclusion A Sprint Retrospective is arguably the most important event that a team can hold. It provides the means to inspect and adapt the team’s actual implementation of the Scrum framework. In this article we’ve looked at how to create an agenda for the session and how to facilitate it, and at the issues of when and where it should be held, and who should attend. Those who cannot remember the past are condemned to repeat it. - George Santayana

Story points are a common name for sizing stories in agile projects. Combined with XpVelocity they provide a technique to aid planning by providing a forecast of when stories can be completed. When estimating work, a common approach is to estimate in terms staff-hours, such as a programmer saying "this will take me two days to do". Many people in the early days of agile, especially those in the ExtremeProgramming community, found that teams struggled to come up with useful estimates using this approach, even when they applied an approach of IdealTime. We found the most effective way to estimate was to size stories relative to each other, and then use past experience to determine how much could be done in an iteration. [1] To determine the points for a story, we compare rough relative sizes. If we are estimating the "fibble the foobar" story, we look for a story of similar size that we've already estimated. We sense it's about the same size as "flipping the synergy bit". Then we look at the story point score for "flipping the synergy bit" and score the "fibble the foobar" the same amount. A team using story points uses a small range of story points to work with. Common examples might be 1,2,4,8 or 1,2,3,5,8 [2]. Often the top number in the series represents "too big" and should be broken down further. [3] Allocating story points should be rapid activity. Discussion should only break out when people have contrasting views on the estimate, in which case its useful to have a discussion as it usually means that something about the story isn't clear. Using a ThrownEstimate is a good technique to move things along quickly. To form a plan with time, you use XpVelocity. Some teams don't like using story points, preferring instead to use StoryCounting. I don't have a preference between the two - both seem to work equally well so it's up to the team to try out and go with whichever suits them best. Further Reading The ThoughtWorks ebook on estimation provides includes a good Q&A on story points. Kent and I discussed them in more depth in the tasteful green book. Most books that talk about planning and estimation in an agile context discuss story points in more detail. Notes 1: "Story Points" is the most common name that I hear these days, but various terms have been used over the years, often with whimsical names that emphasized their arbitrary nature. I particularly like Joseph Pelrine's gummi bears and Josh Kerievsky's: NUTs (Nebulous Units of Time). 2: This is a Fibonacci sequence 3: Using the top number as too big is saying that a story sized at '8' really means '8 or more'. If you do this beware of using this top number when making forecasts of things like completion time, since '8' can turn into all sorts of numbers when it finally gets broken down. It's usually better to explicitly say its too big to be estimated rather than use a false marker number.

"A whole leisure day before you, a good novel in hand, and the backlog only just beginning to kindle..." - Backlog Studies, by Charles Dudley Warner A Recap on Backlogs A few weeks ago we took a critical look at Product Backlogs. We considered their purpose, how they are meant to be used, and why the aspirations they represent can so easily fall into a state of "Lost Remembrance". We also saw that a Product Backlog is an ordered list of requirements that are in scope, and if a project is to deliver value, then certain portions of that scope must be delivered in a timely manner. The Product Backlog is an instrument for managing this process. In short it is a queue, and one that is constantly tended and revised by a Product Owner. It is an artifact of diligent curation in which some requirements are determined to be more important than others, and which therefore ought to be delivered first. On the other hand some requirements will be observed to depend upon others, and must therefore be delivered afterwards. Introducing the Sprint Backlog In a very simple agile process - such as an elementary Kanban implementation - there will only be one backlog. Team members will action each item from the backlog in turn. They will be careful to draw only from the top of the queue, in order of priority. More sophisticated methods can include refinements such as “fast track” lanes in which the Quality of Service will be varied. We've already seen how this approach works in the context of managing critical incidents, and also in the context of hybrid agile methods such as Scrumban. Yet when we consider Scrum itself, we see that the Product Backlog is complemented by another of these queues...the Sprint Backlog. The idea is that if the team deliver something of value at regular intervals then the risks of the project can be better managed, and metrics can be generated that show progress towards its completion. Those regular intervals are known as Sprints. The chunk of requirements that the team agrees to work on during Sprint Planning is the Sprint Backlog. All of this is well known to agile developers, and the chances are that most of you reading this will have been working along these lines for years. So now let's challenge some common assumptions that are made about Sprint Backlogs and how a Scrum team is meant to handle them. Have any of these assumptions been made by your team? Assumption: The Sprint Backlog is a subset of the Product Backlog During Sprint Planning, a team will agree with the Product Owner which requirements from the Product Backlog will be worked on and met by the end of the forthcoming iteration. This has lead to the widespread practice of placing corresponding index cards into the Sprint Backlog on the Scrum board. In effect, it's a subset of the Product Backlog. What many teams fail to realize is that although the identification of an appropriate subset of Product Backlog requirements may be fine as a statement of intent, it can hardly be said to represent an actual plan for delivery. Admittedly a suitable plan doesn't have to be documented; it can live entirely in the developers' heads. A Scrum board's Sprint Backlog may indeed only show that subset of Product Backlog requirements which have been chosen for the Sprint. In fact the whole thing may look very like a Kanban board, even to the point that a casual observer might not be able to tell whether Scrum or Kanban rules are in force just by looking at it. The important thing is that a Sprint plan is agreed upon, shared, and understood by the team. Alternatively a task board may be used. Each selected requirement will be planned into tasks, and these will in turn be transcribed onto index cards or sticky notes. The tasks will move across the board in horizontal swim lanes that align each one to its parent requirement. In this model the Sprint Backlog is not represented by a subset of the Product Backlog, but rather by the corresponding tasks that have been planned for delivery. Assumption: A Sprint Backlog consists of tasks If we can see that each User Story has been broken down into tasks, it implies that some attempt has been made at Sprint Planning. It doesn't prove it of course. For all we know, each one of those tasks could have been identified by one person in the back of the pub last night. In other words, the tasks themselves do not amount to a plan. They merely infer by their presence that a team planning session is likely to have occurred, and that a team understanding regarding the delivery of the Sprint Goal has been reached. This means that a Sprint Backlog doesn't have to consist of tasks. It could be that “clean subset” of the Product Backlog we mentioned earlier, and therefore it might consist of User Stories. What matters is whether or not the team have a plan. While tasks imply that such a plan may have been formulated, they are not conclusive evidence of this, and they are certainly not the only way to compose a Sprint Backlog. Assumption: The Sprint Backlog is the Sprint Goal Identifying a meaningful Sprint Goal is usually the hardest part of Sprint Planning. Deciding how many User Stories can be accommodated, and what they should be, is comparatively straightforward. After all the team should know their budget. Time and again, Sprint Planning will boil down to horse-trading with the Product Owner over how many story points can be absorbed. “We've got 13 points left”, is a common refrain in Planning Poker. “We can't do that 20 pointer”. “OK”, says the Product Owner. “I'll bring forward a 5 and an 8 from the next Sprint”. While this satisfies the brutal arithmetic of planning, it does little to help create an increment of value. When the Sprint Backlog consists of disjointed requirements that don't play together as part of a cohesive potential release, the business value you might expect from such a release can hardly be delivered. Product Owners who expect otherwise are doing themselves and the product a disservice, and team members should not be party to such shenanigans. So, can each one of your team members articulate the goal for their current Sprint? Or is the “goal” just to deliver everything that's on the Sprint Backlog? A Sprint Goal is more than the sum of stories to be delivered or the tasks to be performed. It's about releasing business value incrementally and continually. Without that, the Product Owner probably has no idea when the project will reach completion. The common question “When will the project be done” is most often heard when incremental delivery is weak and the corresponding Sprint Goals are shoddy. Assumption: The Product Owner puts the Sprint Backlog in order This assumption is commonly held, but in Scrum terms it's plain wrong. The Development Team wholly own their Sprint Backlog, and it's up to them how they choose to order it. All the Product Owner should care about is whether or not the Sprint Goal is likely to be met by the end of the Sprint. This assumption is commonly held because Scrum is sometimes conflated with Kanban practice. In Kanban, there will normally be just one backlog and a Product Owner might well put it in order, and thereby exercise fine control over what gets delivered and when. Scrum is a different agile method and a different deal. In Scrum, value will be released at the end of the Sprint, not at discrete or arbitrary points within it. Granted, the Development Team should engage with the Product Owner throughout the Sprint, including on such matters as review and signoff, but the schedule for this is up to them. They decide, by creating their Sprint Plan, how the Sprint Backlog will be structured and how the corresponding work will be actioned. Assumption: Developers shouldn't cherry-pick from the Sprint Backlog This is a very good rule, but it is also one that is subject to misunderstanding. The underlying principle is a sound one. Agile teams should be fully cross-trained, and they should action work from a backlog as a team. Kanban team members, for example, should always take the next highest priority item from the backlog, assuming that there is no other work in progress or which is impeded and needs their attention. No team member should ever try and “pre-book” an item on the backlog, regardless of whether they simply want it or because they think they are best qualified to handle it. Each team member should go to where the work is, whatever that work is, and exactly when it needs doing. Scrum fully supports this principle but there is a further consideration that has to be borne in mind...a Scrum Development Team will have a Sprint Plan. When formulating this plan, they will self-organize to meet the Sprint Goal. That means it's quite possible for the team to decide up front, during Sprint Planning and subsequently during each daily Stand-Up, who will do what. It's important to be able to distinguish this behavior from cherry picking. It's also important for a Scrum Master to be able to smell a rat, and to sense when teams genuinely have a good plan or have started to cherry pick or to form undesirable skill silos. Assumption: A team commits to deliver everything in the Sprint Backlog This is a tricky assumption to deal with because until recently it was seen as being quite valid. For a long time, commitment-based planning was pivotal to a Scrum based way of working. Then, in 2011, the Scrum Guide was revised and the Sprint Backlog was positioned as a forecast of what a team could reasonably be expected to deliver. Clearly, a “forecast” is a weaker use of language than “commitment”. The rationale underlying this change is sensible. There are many things that can change during a Sprint, including requirements understanding or the perception of business value. Moreover, estimates are precisely that – estimates. There's something else we have to remember. The Development Team wholly own their Sprint Backlog. Regardless of whether they forecast their deliverables or commit to them, the content of this backlog is up to them and they can revise it at any time. It's the Sprint Goal, and the concomitant potential release of functionality, that is either committed to or forecast for delivery. Assumption: The Sprint Backlog cannot be changed once the Sprint has started This assumption is incorrect, although it is true that the Product Owner can't change the Sprint Backlog unilaterally. Only the Development Team can make such a change, because they wholly own it. If a Product Owner wishes to change something on the Sprint Backlog then that must be negotiated with the team. Now, let's also bear in mind that Scrum does not prescribe how the requirements within a Sprint Backlog are enumerated. User Stories, or the tasks to realize such stories, are the most common form of expression. Since User Stories do not document requirements exhaustively, but are placeholders for a future conversation, it follows that a change in understanding does not necessarily mean a change in the Sprint Backlog itself. Conclusion Sprint Backlogs mean different things to different teams. Some may populate them with tasks, while others may simply transfer over agreed User Stories from the Product Backlog. Either approach is acceptable given that the Development Team wholly own the Sprint Backlog. The important thing is that the team should have a plan for meeting a well defined Sprint Goal that has been agreed with the Product Owner, and they should form their Sprint Backlog in accordance with that plan. The backlog itself should never be mistaken for, or used in lieu of, a coherent goal for delivering a potentially releasable increment of value.

The following are some highlights from Dan North‘s excellent, inspiring, and insightful talk Patterns of Effective Delivery at RootConf 2011. North has a unique take on what agile development is, going beyond the established (and rather rigid) views. I really recommend this talk to learn more about effective teams, about North’s “shocking,” beyond-agile experience, and for great ideas on improving your team. The talk challenges the dogma of some widely accepted principles of “right” software development such as TDD, naming, and the evilness of copy/paste. However the challenge is in a positive way: it makes us think in which contexts these principles really help (in many) and when it might be more effective to (temporarily) postpone them. The result is a much more balanced view giving you a better understanding of their value. A lot of it is inspired by the theory (and practice) of Real Options. What are Patterns of Effective Delivery? Patterns – Strategies that work in a particular context – and not in another (too often we forget the context and to consider the context where a strategy doesn’t work / is counter-productive). Beware: a part of the context is the experience of the developer. For inexperienced devs it might be better to just stick to a process and apply TDD all the time instead of trying to guess when they it is appropriate and when it is not. Effective – Optimize for something: Volume of software produced? Time to market? Learning/discovery? Certanity? User experience? Any of these will work. Delivery – Get stuff that is useful out of the door. Software is not important, the utility it provides is. Know why you write the software. Some of the patterns take years to master and require significant investment before you start seeing the benefits. You might need to fail a few times before getting them right. Disclaimer: These are notes that make sense to me. They will likely make only limited or no sense to people that haven’t heard the talk. It would be best to go and listen to it instead. Selected patterns Spike and Stabilize (or throw away): traditionally we decide whether we are writing production-grade code (with high rigor such as TDD) or just a throw-away spike before we start coding – i.e. at the moment when we know the least about it. We should not decide this up front but “exercise the option of investing in the quality” later, based on experience. Start as a spike and if the code proves valuable, stabilize it, refactor, test etc. Evolve the code based on experience (good naming, quality). Defer the commitment to the quality of the code and optimize for learning An example of spike-and-stabilize regarding test naming: take a test originally named blah – you don’t know what it should do yet but you're experimenting. When the code evolves into something meaningful, name it properly, according to that. Ginger Cake – Copy and paste code, rip irrelevant things out until only the important things are left, then write tests around it. You may end up with code that is similar, but not in the ways you expected. If you started with abstracting, it would be the wrong abstraction. The pattern says: “We know and respect DRY but are not slaves to it.” Short Software Half-Life: 1) We don’t care about the software but the utility it gives us. If writing it gives us better ideas, we can delete it and do the better thing. 2) How would you write the code if 1/2 of it – but you don’t know which half – would be gone in a few weeks? The answer is, start simple (see Spike & St.), extract commonalities, improve quality, etc. For code that has already been around for a while and has proven itself useful; Some architecture styles lend themselves better to such quick evolution – such as small, focused services, popularly known as micro services (see slides, esp. p.42+). “Look at the code as it evolves and decide what to invest in.” (The investment includes thinking about the design.) All code is not equal. Create Urgency – To change a paradigm, the way of thinking, people must be desperate and have no more options along with the knowledge of what to do. Apply this pattern when learning something new. Do it on something real, under self-inflicted pressure. For example, you could commit to do an app with a crazy deadline using some new tech. This would give you a sense of urgency, with no more options. It forces you to learn only the parts you really need. Socratic Testing (coaching style) – Don’t tell the team what’s wrong with their code, which is threatening and thus hard to accept. Pair with them on writing a test and to support the test, make “helper” classes etc. that you’d like to see in the production code. If they really are useful, they will spot it and decide to pull them into the production code. Make them the hero. Respond to their questions with another question. Fits In My Head – we need code that we can understand and reason out (big classes, methods, complex models, etc.). Keep the code simple, optimize for understandability, readability, and obviousness. Build Shared Idioms in the team so that the team members would, given the same context, arrive at the same decisions/design. Something should only differ from the usual way of doing it when there is a good reason for it. Thus a difference provides a hint, difference is data. For example, putting all communication over ZeroMQ at only one place through shared memory. This indicates there is some (most likely performance) reason for it. Communication strategies shouldn’t be picked randomly or ad-hoc. TDD – A pattern that, in a particular context, may make you much more effective. Most of you reading this know what TDD is. Bonus: Micro Services Rough notes from James Lewis’s talk Micro Services: Java, the Unix Way (2013) – especially slides 42+: Use web, do not bypass it – REST, JSON; standardized application protocols and message semantics Small with a single responsibilities (does one thing, fits into one’s head, small enough to rewrite and throw away rather than maintain) Containerless and installed as well-behaved Unix services (executable jar with embedded Jetty + rc.d start scripts and config files) Avoid unnecessary coupling; Domains in different bounded contexts should be distinct; It's ok to have duplication with physical separation to enforce it; There will be common code, but it should be library and infrastructure code; Leverage Conway’s Law to support decoupling Provisioned automatically: “The way to manage the complexity of many small applications is declarative provisioning” (including instance count, scaling, load balancing) Status aware and auto-scaling; In-app status pages; monitoring to autoscaling Each service is entirely decoupled from its clients, scalable, testable and deployable individually Decomposition: This technique goes from product to a set of capabilities (e.g. monitoring, reporting, fulfillment, user) and then to each capability being implemented by a set of small apps/services exposing a uniform interface of Atom Collections. The capabilities form the project by interacting via a uniform interface – HTTP (reverse proxies etc.), HATEOS (link relations drive state changes; its an anti-corruption layer that allows the capability to evolve independently of its clients), and standard media types (usable by many types of clients). Explicit tips from the talk: Divide and conquer – Start on the outside and model business capabilities. Use Conway’s Law to structure teams (and enforce decoupling). The Last Responsible Moment – Don’t decide everything at the point when you know the least. Be of the web, not behind the web. If something is important, make it an explicit part of your design (reify) – an exampoe would be an instance of services creating users by posting to /user. They post a user creation request and get response immediately. The user is then created eventually (reminds me of futures). Favor service choreography over orchestration. Use hypermedia controls to decouple services. Some tools used: SimpleWeb/Jetty, Abdera for Atom, Smoothie charts (JS charts for streaming data), Coda Hale’s metrics, Graphite. Ops: Fabric with boto, AWS, Puppet, … . Remember there are NO SILVER BULLETS. This stuff is hard. Versioning, Integration, Testing, Deployment and eventual consistency are hard for people to wrap their heads around. Note: Comoyo.com, powered by a number of ex-googlers and other smart people, does the same thing. So does Netflix, I believe. Related If you liked this, you might also like Dan North's presentations Accelerating Agile: hyper-performing teams without the hype and Patterns of Effective Teams at NDC Oslo 2013.

How should a team respond to change? The simple answer is “they should respond by being agile”. If there’s one concept about agility that sceptical managers have caught onto it’s this one. When change happens, they expect that a truly agile team will be able to turn on a dime. You can hardly blame them, it sounds like a great idea. It suggests that perhaps managers don’t need to stabilise the working environment. They just need to pass change on. The teams will be able to deal with the impact…if they’re any good. After all, aren’t they meant to be agile? Of course, team members will have a rather different interpretation of this. They’ll tell you that agility isn’t about being reactive – it’s about responding to change in a controlled manner. With seemingly limitless demands on the team, and clearly finite resources, prioritisation becomes essential. Agile teams will work from an ordered backlog, and they’ll plan to deliver value by drawing work requests out of that queue. In other words they plan to follow an agile process…and that means things like “Sprint Planning” can still happen. So let’s ask the question again – how should a team respond to change? A better answer is “they should respond by following agile rules”. It isn’t about turning on a dime, it’s about following rules, and it’s important to understand how those rules differ between agile methods. Nowhere is this made more clear than in the way agile teams respond to a “P1 Incident”. It’s common in service management to rank incidents by priority. A P1 (Priority 1) is considered to be the highest – the business itself is threatened. When a P1 happens the expectation is that all hands will man the pumps. So what does that mean for an agile team that plays by the rules of backlog management? Well, in the case of a Lean-Kanban team, the response model is fairly straightforward. Priority incidents can be moved to the top of the backlog so they are actioned as soon as the next team member becomes available. Alternatively the quality of service can be varied. As soon as a P1 is raised a ticket (card) will be placed in a fast-track lane on the Kanban board. The team will stop what they are working on, mark their tickets as impeded, and swarm over the P1. Once a response has been planned those members who won’t be involved can return to their original work. A Scrum team has a different agile response. They’re rigged for the incremental de-risking of project scope, and plan to meet a Sprint Goal each iteration. If they have to drop everything for a P1, then that goal may no longer be achievable and the iteration could have to be terminated. Clearly they aren’t geared to be as immediate in their response as a Lean Kanban, but not all managers will understand or appreciate that point. Interestingly, some companies have dedicated “incident rooms” to which key personnel are expected to decamp should a P1 occur. These are clearly modelled on the incident rooms of the emergency services, the idea being that if responders are co-located then the crisis should be handled more efficiently. In an agile context however, they are something of an anti-pattern. In a genuinely agile organisation the responders will already be co-located along with the resources needed, and information radiators will be in place to keep stakeholders updated. As long as the agile models in use are understood a P1 incident can be handled without recourse to special measures.