The business world is an increasingly complex one. In the past few IBM CEO surveys, they have highlighted the growing importance of both being able to manage this complexity, and to do so in a collaborative way. This shifting zeitgeist was reflected in a series of seminars hosted by Xincus, and prompted the launch of a two phase study into how Chambers of Commerce can evolve within this new landscape. The study, consisting of in depth one on one interviews and a nationwide online survey, aimed to better understand both how Chambers can adapt, and what changes would be required to do so. The findings from this research are now available in a new paper called Chamber 2.0: Digital – Connected – Global. The paper outlines both the main challenges currently facing Chambers, and the steps they can take to thrive in such an environment. Amongst the main challenges identified by the research was a fundamental desire to change and modernize, with a strategic positioning and business model that would allow Chambers to flourish. There was also a strong desire to work more effectively with partners, both inside and outside of the Chamber network, sharing both resources and insights. The report then concluded with a road map derived by molding these findings from within the network with best practice from the wider business world. The road-map consists of five broad stages, with each one containing more detailed steps Chambers can take to prepare for the modern world. Become a one stop shop for members, including positioning the Chamber brand for the modern world as centers for Business, Innovation, and Economic Development with a new and modernized approach to business that sees an adaptive and responsive leadership style essential to a revitalize business model. Offer new value, with a new emphasis on virtual services to reflect modern ways of working. Chambers will become a solution hub that connects and match makes members, with co-working spaces connecting the physical and virtual worlds. Collaborate beyond borders, by building an extensive Chamber alliance network, allowing Chambers to become specialized regional hubs, whilst tapping into the collective wisdom of the entire network as well as offering “health-club” type e-memberships to professionals, academics, entrepreneurs and “free agent” millennials alike. Nurture new economic development, by facilitating entrepreneurial collaboration between members and stakeholders, connecting the right people with the right resources, helping to forge an innovation economy and a thriving business community and jobs. Foster global innovation ecosystems, by tying all of these communities together to form a hyperconnected ecosystem, with Chambers at its heart, thus empowering the next wave of new economic development around the world. The report makes clear that whilst change is desired, the network remains positive that the right developments will occur. With Chambers striving to maintain their position at the heart of the business community, this report will go some way towards helping them achieve that goal. You can get your copy of the report here. Original post

Due to the technological development with the Internet and social media, markets are no longer created and controlled with broadcast marketing. People can now find and connect with people like themselves all over the world – and no longer limited to the people in their close proximity and to existing ties such as family members, friends, colleagues or neighbors. They can connect with anyone, and they all influence each other, immediately and with multiplier effects. The power is shifting from companies to consumers. It is a radical shift, but it was predicted already in the mid 90’ies by marketing guru Philip Kotler as a consequence of the Internet. So we shouldn’t be too surprised. Yet a lot of companies are. And they haven’t prepared at all for this. Companies and organizations are waking up to a new reality, and the wake-up call can sometimes be harsh. A number of things are changing, and I will mention four of these here. 1. Change and uncertainty is the new normal To start with, today’s business environment is anything but static. It’s changing faster and faster, and in new ways. It’s becoming more and more unpredictable. This means that companies and organizations can’t do long-term planning like they used to. Instead they have to be prepared for change, to quickly adapt to new conditions and situations, such as changing consumer behaviors, new competition, new innovations, and so forth. 2. Diminishing return on optimization efforts The second big change is that the return on optimization efforts is diminishing. The companies that lead the development in their industries, and get all the profit are those that are able to create new value. They don’t do that with optimization. They do it by innovating new product and services, by creating and developing relationships with consumers and others, by collaborating internally and externally, and by constantly learning how change theirs strategies 3. Growth and efficiency is not enough Thirdly, being able to grow in terms of production volumes, market presence and market share is not enough to be successful, neither is it to produce and market products or services as efficiently as possible. Instead, continuous innovation and high responsiveness to change and customer demands is becoming more and more critical. This obviously can’t be addressed solely by streamlining and optimizing transactional processes, as we have done for the last few decades with the help of information technology. Innovation and responsiveness requires empowered people that can collaborate efficiently and effectively. That is why collaboration is the new productivity frontier. 4. Non-routine knowledge work is increasing in importance Finally, we can see that work is shifting from manual work to knowledge work, but most importantly from routine work to non-routine work. Computers and software are taking over repetitive and routine-based knowledge work, just as robots have replaced workers doing repetitive and routine manual work in the factories. The work that is remaining and increasing is the non-routine knowledge work that is often highly interdependent such as problem solving, product development, sales and so forth. Knowledge work is something completely different than most of the work that organizations have tried to improve and optimize during the 20th century. It’s fluid, dynamic, unpredictable, and non-repeatable. Knowledge workers need to look beyond the standard ways of doing things, to question information, rules, and ways of working. This is something completely different to how it is to work at a production line in a factory, where workers follow predefined and highly repeatable processes and procedure. Most organizations have been designed for efficiency and economies of scale, not for empowering people enabling collaboration, innovation and responsiveness. Too often, knowledge workers feel like they are just cogs in a big machinery. And unfortunately, in most cases, their feelings are motivated. We see all over that employees are increasingly dissatisfied with their jobs. Employee engagement is falling. It is especially bad in large and distributed organizations. The consequences are many and severe. Innovation is stifling. Productivity is, if not falling, not improving. People are leaving, or they want to leave, their jobs. It is hard to sustain and improve quality. And it’s not possible to recruit and retain talent by the quality and numbers that are needed. So what is causing this? I have grouped a number of causes into three overall themes; complexity, inflexibility, and disconnectedness. 1. Complexity As knowledge workers we often find ourselves stuck between a rock and a hard place. Workload and complexity at work is increasing, while we at the same time are expected to produce more, faster and faster. And adapt to new conditions. Not only that, we are expected to be creative and innovative as well. Still, if we look at an average day in the life of a knowledge worker, we struggle a lot with finding answers to basic questions, such as what is happening in our work environment, who is doing what, where I can fiend a piece of information, when it is my turn to contribute, and so on. This means that we spend a lot of time on things that are not creating value, just getting ready to create value. For example, Intel estimated that their employees spent one day per week on trying to find information and locating the expertise they needed to do their job. Although the tasks of knowledge workers come in all shapes and sizes, many of them rely on a number of basic capabilities, such as finding information or locating expertise. These capabilities are vital to knowledge worker productivity, but also to innovation, and it is evident that poor capabilities generate a lot of waste. Of course, we constantly get new tools that aim to help us. But when new tools and features are introduced to knowledge worker, there often is no guidance, and little customizing it to fit our needs. The problem is that we already have this huge pile of complex products to deal with, and we need to fit these. This technology-centric approach adds complexity instead of reducing it, instead of making things simpler for us. A study by Oracle found that productivity of enterprise application users had fallen almost 1/5 over a period of only three years. It’s like giving everybody Friday off. How can that be? I would argue that it’s the increasing complexity that is hampering productivity. 2. Inflexibility The second theme is inflexibility. By this I mean that our organizations and the systems that are there to help us get our work done are designed in a way that makes change, creativity and improvisation hard. Instead of empowering knowledge workers, our organizations often constrain and prevent us from being productive and innovative. First of all, there is a mismatch between what science knows and what organizations do when it comes to how they try to motivate knowledge workers to perform better. In most organizations, existing performance models are built on extrinsic motivators, or carrots and sticks if you like. These models worked pretty fine for routine, left-brain, rule-based work of 20th century, but they are not working very well for right-brained, creative, and self-propelled people performing non-routine and highly collaborative conceptual tasks. For example, bonuses and commissions don’t work for this kind of work. As a matter of fact, science shows they have the opposite effect than intended; the higher the extrinsic rewards, the worse the performance gets. Organizations are apparently making important decisions about their future based on the wrong assumptions. The left circle in this venn diagram represents things that have been considered important for trying to maximize the productivity of manual routine work. The right circle represents things that are important for motivating knowledge workers doing non-repetitive work. There is still little understanding and experience of how to do the things the right circle, so organizations and managers tend to stick with the things they know how to do. Those are the things in the left circle. Furthermore, knowledge workers need to have flexible working conditions. When it comes to knowledge work, work is not a place, it is something you do. Most knowledge worker tasks can be performed from any location, even those that require close collaboration with others. Organizations need to support this, not only to increase performance, but also to make people more engaged at work. Research shows that what employees of all age groups want is the flexibility to determine for themselves where, when, and how they work, and that increasing workplace flexibility has a positive effect on employee engagement and thereby also on employee productivity. A Virgin Media Business study found that 40% of the surveyed organizations often overhear employees complain about being tied to their desks and 7 in 10 organizations believe flexible working would make their employees both happier and more productive, boosting employee engagement. 3. Inconnectedness Finally, we have a theme that I call disconnectedness. It is about people and information being disconnected from each other, and thereby unable to share, cooperate and collaborate as is required to be productive and deal with the challenges organizations face. Collaborating isn’t as easy as it sometimes might sound, especially not in large and distributed organizations; there are too many barriers to collaborate naturally across an organization and across locations. In a complex and constantly changing work environment, it becomes even harder to find time and energy to overcome these barriers. It is only natural that we tend to share, cooperate and collaborate with people in our close proximity and that we already know and trust, failing to help and collaborate with others or share information that they might have use for. People work in silos. Silo thinking is a typical phenomenon in large organizations. Teams tend to focus on the parts they are responsible for and specialize in. They sub-optimize and focus on their own goals. They become organizational barriers that limit communication and impede sharing, collaboration, and innovation within the enterprise. Organizations have also created digital work environments to optimize personal productivity and teamwork, but doing so they have neglected the fact that knowledge work is increasingly relying on collaboration in networks across locations and organizations and stretching far beyond teams. It might seem as a paradox, but the modern and increasingly digital work environments have in fact made people more isolated and unaware of what is happening at work. This disconnectedness means that people become less engaged. And in a rapidly changing and complex work environment, this has serious implications, such as lost productivity and innovations. Or worse – talent is wasted and people leave. What do to about it? So what should organizations do to avoid the negative consequences of complexity, inflexibility and disconnectedness? The simple answer is that they should start working towards increased simplicity, flexibility and connectedness. What they should do and how, I will return to in my next post.

Programmers seem to be fairly productive people. You always see them typing at their desks; they chafe for meetings to finish so that they can go back to their desks and code. When asked, they will say that there is not enough time to produce the code, and the sooner they can start coding, the sooner they will be done. So writing code must be the most important thing, correct? If the average programmer writes about 50 lines of production code a day. A 50,000 line program would take 1,000 man days to produce. The 50,000 line listing can be entered by a programmer at about 1,000 lines a day or about 50 man days. So what the heck are the developers doing for the other 950 days? Before addressing that issue, lets make a simple observation. Capers Jones has compared many methodologies (RUP, XP, Agile, Waterfall, etc) and programming languages over thousands of projects and determined that programmers write between 325 and 750 lines of code (LOC) per month, which is less than the 1,000 LOC per month suggested above 1. Even if programmers do not average 50 lines of code per day, the following is clear 2. Methodology does not explain the apparent productivity gap No language accounts for the apparent productivity gap The reality is that only a fraction of a developer's time is actually spent writing production code. If a developer is typing in code all the time then they are really trying different combinations of code until they finally find the combination of code that works. Or more correctly, the combination that seems to match the requirements until either QA or the business analyst comes back and lets them know there is a problem. That is why developers that plan their code before using the keyboard tend to outperform other developers. Not only do only a few developers really plan out their code before coding but also years of experience do not teach developers to learn to plan. In fact studies over 40 years show that developer productivity does not change with years of experience. (see No Experience Required!) Years of experience do not lead to higher productivity Interestingly enough, there are methodologies that have been around for a long time that emphasize planning code. Watts Humphrey is the creator of the Personal Software Process (PSP) 3. Using PSP has been measured to: PSP can raise productivity by 21.2% and quality by 31.2% If you are interested there are many other proven methods of raising code quality that are not commonly used (see Not Planning is for Losers). If your developers at their keyboard and not planning at a white board then odds are that your productivity is not as high as it could be. Bibliography 1 The The Mythical Man Month is even more pessimistic suggesting that programmers produce 10 production lines of code per day 2 Jones, Capers and Bonsignour, Olivier. The Economics of Software Quality. Addison Wesley. 2011 3 Watts, Humphrey. Introduction to the Personal Software Process, Addison Wesley Longman. 1997

If your employees aren't continuing to grow, your company will become stagnant. Here we examine goal-setting case study with the senior dev and the designer.

This is the second in a series of posts on the new “Red Deer” (https://github.com/jboss-reddeer/reddeer) open source testing framework for Eclipse. In the previous post in this series, we introduced Red Deer, learned how to install it into Eclipse, examined some of its cool features, and built and ran a sample test program from scratch. One of the challenges in creating effective automated tests is in making the tests self-sufficient enough to be able to set up their required operation environment, and robust enough to be able to determine whether that operating environment has been set up correctly. In the first post in this series, we took a quick look at Red Deer’s implementation of Requirements classes. In this post, we’ll take a more detailed look at Requirements, including how Red Deer supports your creating custom Requirements. The Case for Automated Test Requirements Let’s start by setting the context for why test programs need requirements. It’s often the case that a set of automated tests runs unattended and all the tests fail, not due to a bug in the software under test, but due to a broken or incomplete test environment. When we refer to a Red Deer “requirement,” we’re talking about actions that must be performed, or objects that must be created, before a test can be run. Examples of these requirements are having a user account defined or a connection to a database created and verified. What makes using Red Deer requirements different from your creating a less formal set of requirements with the @BeforeClass annotation provided by JUnit, is that if requirements are not met, then the test in question is not run. This can save you a lot of test execution time and test failure debugging time. Red Deer requirements are implemented in the RedDeerSuite. A test that makes use of requirements is must be run with a RedDeerSuite suite and annotated with @RunWith(RedDeerSuite.class) OOTB Red Deer Requirements As we saw in the first post in this series, Red Deer currently provides OOTB (out of the box) predefined requirements that enable you to clean out your current workspace and open a perspective. Using these requirements is simple. All you have to do is to add these import statements to your Red Deer test programs: import org.jboss.reddeer.eclipse.ui.perspectives.JavaBrowsingPerspective; import org.jboss.reddeer.requirements.cleanworkspace.CleanWorkspaceRequirement.CleanWorkspace; import org.jboss.reddeer.requirements.openperspective.OpenPerspectiveRequirement.OpenPerspective; And, we also have to add a reference to org.jboss.reddeer.requirements to the required bundle list in our example’s MANIFEST.MF file. And finally, add these annotations to the test program: @CleanWorkspace @OpenPerspective(JavaBrowsingPerspective.class) What if you want to define your own custom requirements? Let’s move on and examone how Red Deer supports that too. Different Ways to Implement New Red Deer Requirements Red Deer supports (4) ways to implement new requirements. We’ll look at them in order of their relative complexity: Simple Requirements Requirements with Parameters Requirements with Property Based Configuration Requirements with a Custom Schema In order to examine how Red Deer supports implementing new requirements, we’ll actually create some new requirements in Red Deer source code. In order to do this, we’ll have to download a copy of Red Deer’s source code. To perform this download, navigate to your desired directory and enter this command: git clone https://github.com/jboss-reddeer/reddeer.git And then, import Red Deer into eclipse as a set of existing Maven projects: If you navigate to the top level of the directory into which you downloaded the Red Deer source code, you’ll see this: What you want to do is to select all of the Red Deer projects. After you press the “Next>” key, Eclipse will import all the Red Deer packages as maven projects. (This may take a few minutes.) OK, now we can move on to creating some new requirements. We’ll start with the simplest of the (4) types, a simple requirement. Implementing a Simple Requirement A simple requirement consists of (2) parts: a “fulfilling” class that provides the code executed when the requirement is invoked, and an annotation that references that fulfilling class. As an illustration, let’s look at the skeleton “AdminUserRequirement” provided with your Red Deer download. This requirement is intended to serve as an example for implementing a full requirement to ensure that an admin-level user is defined before an attempt is made to run a test. The source file you want to look for is: /plugins/org.jboss.reddeer.examples/src/org/jboss/reddeer/junit/annotation/simple/AdminUserRequirement.java While it’s a small file, it’s a full example. It’s worthwhile examining it line-by-line: package org.jboss.reddeer.junit.annotation.simple; import java.lang.annotation.ElementType; import java.lang.annotation.Retention; import java.lang.annotation.RetentionPolicy; import java.lang.annotation.Target; import org.jboss.reddeer.junit.requirement.Requirement; import org.jboss.reddeer.junit.annotation.simple.AdminUserRequirement.AdminUser; /** * Admin user test requirement * @author lucia jelinkova * */ public class AdminUserRequirement implements Requirement { @Retention(RetentionPolicy.RUNTIME) @Target(ElementType.TYPE) public @interface AdminUser { } public boolean canFulfill() { // return true if you can connect to the database return true; } public void fulfill() { // create an admin user in the database if it does not exist yet } public void setDeclaration(AdminUser declaration) { // no need to access the annotation } } The important elements in this file are: Line 17 - @Retention - Specifies how the marked annotation is stored—Whether in code only, compiled into the class, or available at runtime through reflection. Line 18 - @Target - Marks another annotation to restrict the types of Java elements to which the the annotation can be applied Line 20 - AdminUser interface - This defines the object type used by the defined requirement. Line 23 - canFulfill method - In a fully written requirement this method will include the code to determine if the requirement can be met (or “fulfilled”). This method is set to always return a value of true. Line 32 - fulfill method - And here is the code that will be executed if the canFulfill method returns a value of true. For an example of the corresponding annotation in action, let’s look at the test program that is included with the fulfilling class. The test program is here: /plugins/org.jboss.reddeer.examples/src/org/jboss/reddeer/junit/annotation/simple/AdminUserTest.java This test program is also very short as it is a skeleton. The outline is there, but the specific logic that to implement the AdminUser requirement is left as an “exercise for the reader.” package org.jboss.reddeer.junit.annotation.simple; import org.jboss.reddeer.junit.runner.RedDeerSuite; import org.jboss.reddeer.junit.annotation.simple.AdminUserRequirement.AdminUser; import org.junit.Test; import org.junit.runner.RunWith; @RunWith(RedDeerSuite.class) @AdminUser /** * Test with AdminUser requirement * @author lucia jelinova * */ public class AdminUserTest { @Test public void test(){ // put test logic here } } The @AdminUser annotation on line NN tells the whole story. When this annotation is executed, the fulfilling class is invoked and if the “canFulfill()” method returns true, the test is executed. If the method returns false, then the test is not executed. Let’s run this test and see what happens. First, locate the AdminUserTest.java file in the eclipse Navigator view: Then, right-click and specify that it be executed as a JUnit test: And, not surprisingly, here’s the successful output from the test: Before we move on, let’s modify the canFulFill() method to return a false value, and rerun the test. The results look like this: 22:11:04.923 INFO [main][RequirementsRunnerBuilder] Found test class org.jboss.reddeer.junit.annotation.simple.AdminUserTest 22:11:04.924 INFO [main][RequirementsBuilder] Creating requirements for test class org.jboss.reddeer.junit.annotation.simple.AdminUserTest 22:11:04.925 DEBUG [main][RequirementsBuilder] Found requirement class org.jboss.reddeer.junit.annotation.simple.AdminUserRequirement for annotation interface org.jboss.reddeer.junit.annotation.simple.AdminUserRequirement$AdminUser 22:11:04.927 INFO [main][Requirements] Requirement class org.jboss.reddeer.junit.annotation.simple.AdminUserRequirement can be fulfilled: false 22:11:04.927 INFO [main][RequirementsRunnerBuilder] All requirements cannot be fulfilled, the test will NOT run So, this time, the requirement was not met and the test was not run. Note that the requirement did the work for us. We did not have to write a lot of new code to determine if the requirement had been met to decide whether or not to run the test. That’s all well and good for a simple requirement. But what about if we want to make the requirement a bit more flexible by enabling us to pass it a parameter? Let’s look at that next. Implementing a Requirement with Parameters In order to implement a requirement that accepts one or more parameters, we have to make two additions to the simple requirement that we just examined. First, we have to use a different requirement definition. The code that we want to look at this time is here: /plugins/org.jboss.reddeer.examples/src/org/jboss/reddeer/junit/annotation/advanced/UserRequirement.java The file looks like this: package org.jboss.reddeer.junit.annotation.advanced; import java.lang.annotation.ElementType; import java.lang.annotation.Retention; import java.lang.annotation.RetentionPolicy; import java.lang.annotation.Target; import org.jboss.reddeer.junit.requirement.Requirement; import org.jboss.reddeer.junit.annotation.advanced.UserRequirement.User; /** * Parameterized requirement with parameter name * @author vpakan * */ public class UserRequirement implements Requirement { @Retention(RetentionPolicy.RUNTIME) @Target(ElementType.TYPE) public @interface User { String name(); } private User user; public boolean canFulfill() { // return true if you can connect to the database return true; } public void fulfill() { System.out.println("Fulfilling reuirement User with name: " + user.name()); // create an admin user in the database if it does not exist yet } public void setDeclaration(User user) { this.user = user; } } The important difference between this class and the original AdminUserRequirement that we examined a moment ago is: Line 20 - The interface “User” now defines a String parameter “name” and on line NNN here the User object is defined. Second, we have to change the declaration of the requirement in the test program. The test program that we’ll look at this time is here: /plugins/org.jboss.reddeer.examples/src/org/jboss/reddeer/junit/annotation/advanced/UserTest.java Finally, our test program for this requirement looks like this: package org.jboss.reddeer.junit.annotation.advanced; import org.jboss.reddeer.junit.runner.RedDeerSuite; import org.jboss.reddeer.junit.annotation.advanced.UserRequirement.User; import org.junit.Test; import org.junit.runner.RunWith; @RunWith(RedDeerSuite.class) @User(name="admin") /** * Test with parameterized requirement User * @author lucia jelinkova * */ public class UserTest { @Test public void test(){ // put test logic here } } The interesting line in this test is: Line 8 - @User(name="admin") - Where we set the value of the “name” parameter. When we run the UserTest as a JUnit test, we see this output: 20:46:03.554 INFO [main][RequirementsRunnerBuilder] Found test class org.jboss.reddeer.junit.annotation.advanced.UserTest 20:46:03.555 INFO [main][RequirementsBuilder] Creating requirements for test class org.jboss.reddeer.junit.annotation.advanced.UserTest 20:46:03.556 DEBUG [main][RequirementsBuilder] Found requirement class org.jboss.reddeer.junit.annotation.advanced.UserRequirement for annotation interface org.jboss.reddeer.junit.annotation.advanced.UserRequirement$User 20:46:03.558 INFO [main][Requirements] Requirement class org.jboss.reddeer.junit.annotation.advanced.UserRequirement can be fulfilled: true 20:46:03.558 INFO [main][RequirementsRunnerBuilder] All requirements can be fulfilled, the test will run 20:46:03.575 INFO [main][RedDeerSuite] RedDeer suite created 20:46:03.584 INFO [main][Requirements] Fulfilling requirement of class org.jboss.reddeer.junit.annotation.advanced.UserRequirement Fulfilling requirement User with name: admin 20:46:03.585 DEBUG [main][RequirementsRunner] Injecting fulfilled requirements into test instance 20:46:03.587 INFO [main][RequirementsRunner] Started test: test(org.jboss.reddeer.junit.annotation.advanced.UserTest)20:46:03.588 INFO [main][RequirementsRunner] Finished test: test(org.jboss.reddeer.junit.annotation.advanced.UserTest) While it makes requirements more flexible when you are able to add parameters to their definition, it is still limited as a solution as you have to handle the individual parameters one by one. Fortunately, Red Deer also supports defining test configurations in your own custom XML schemas. Defining Complex Configurations - Two Approaches Red Deer supports two different approaches to defining complex configurations. You can either: Define the configuration as a set of (key=value) properties. If you choose this approach, you will have to also define setter methods for each property in your requirement’s fulfilling class. Create a custom XML schema. If you choose this approach, you will have to create a configuration object in your test code and then inject that object into your requirement. Regardless of which approach you choose, you store the configuration data in either a single XML file, or directory of XML files and then pass those files to your test program by defining this JVM argument when you run your test programs: -Dreddeer.config=/home/path/to/file/or/directory Let’s examine each of these approaches in detail. We’ll start with the properties based approach. Requirements with a Property Based Configuration The first thing we have to do to use a property based configuration is to define the properties. We’ll do this in an an XML file that complies with the Red Deer requirements XSD schema file. You can view the XSD here: http://cloud.github.com/downloads/jboss-reddeer/reddeer/RedDeerSchema.xsd The code for this example is here: /jboss/local/reddeer_fork/reddeer/plugins/org.jboss.reddeer.examples/src/org/jboss/reddeer/junit/configuration/simple And - here’s our properties file. Note that the requirement defined in this file contains two properties: name and ip (IP address). Let’s now expand on the “UserRequirement” example that we defined a few minutes ago. What we want to be able to do is to remove hardcoded requirements data from the source code and instead define that data in set of properties. To use this requirements.xml file, we have to make some changes to the UserRequirement.java class. package org.jboss.reddeer.junit.annotation.simple; import java.lang.annotation.ElementType; import java.lang.annotation.Retention; import java.lang.annotation.RetentionPolicy; import java.lang.annotation.Target; import org.jboss.reddeer.junit.requirement.Requirement; import org.jboss.reddeer.junit.annotation.simple.UserRequirement.User; import org.jboss.reddeer.junit.requirement.PropertyConfiguration; /** * Admin user test requirement * @author lucia jelinkova */ public class UserRequirement implements Requirement , PropertyConfiguration { @Retention(RetentionPolicy.RUNTIME) @Target(ElementType.TYPE) public @interface User { } private String name; private String ip; public boolean canFulfill() { // return true if you can connect to the database return true; } public void fulfill() { System.out.println("Fulfilling User requirement with\nName: " + name + "\nIP: " + ip); // create an admin user in the database if it does not exist yet } @Override public void setDeclaration(User user) { // annotation has no parameters no need to store reference to it } public void setName(String name) { this.name = name; } public void setIp(String ip) { this.ip = ip; } public String getName() { return name; } public String getIp() { return ip; } } The important changes are the addition of this import statement at line 8: import org.jboss.reddeer.junit.requirement.PropertyConfiguration And the addition of the implement clauses for the Requirement (with a type of User), and the PropertyConfiguration (so that the properties can be read) at line 15: public class UserRequirement implements Requirement , PropertyConfiguration And addition of the setter methods for the name and ip properties. Finally, here is the updated test program: package org.jboss.reddeer.junit.annotation.simple; import org.jboss.reddeer.junit.runner.RedDeerSuite; import org.jboss.reddeer.junit.annotation.simple.UserRequirement.User; import org.junit.Test; import org.junit.runner.RunWith; import org.jboss.reddeer.junit.requirement.inject.InjectRequirement; @RunWith(RedDeerSuite.class) @User /** * Test with AdminUser requirement * @author lucia jelinova * */ public class UserTest { @InjectRequirement private UserRequirement userRequirement; @Test public void test(){ System.out.println("The test is running"); System.out.println(userRequirement.getName()); // put test logic here } } What’s new in the test program is the addition of the import statement for the requirement injection: import org.jboss.reddeer.junit.requirement.inject.InjectRequirement; And the code to define and inject the UserRequirement: @InjectRequirement private UserRequirement userRequirement; When we run the test, we have to reference the configuration file via a Java VM argument . This means that we must define a new “run configuration” that is based on the JUnit run configuration provided in Eclipse and provide the VM argument that references the configuration file: In our example, the -Dreddeer.config VM argument is defined as: -Dreddeer.config=/jboss/local/reddeer_fork/reddeer/plugins/org.jboss.reddeer.examples/src/org/jboss/reddeer/junit/annotation/simple/reddeer.xml To execute the test, right-click on the UserTest class, and select the run configuration we just created: And, the test generates this test output in the console: 22:40:50.988 INFO [main][RedDeerSuite] Creating RedDeer suite... 22:40:50.990 INFO [main][SuiteConfiguration] Looking up configuration files defined via property reddeer.config=/jboss/local/reddeer_fork/reddeer/plugins/org.jboss.reddeer.examples/src/org/jboss/reddeer/junit/annotation/simple/reddeer.xml 22:40:50.991 INFO [main][SuiteConfiguration] Found configuration file /jboss/local/reddeer_fork/reddeer/plugins/org.jboss.reddeer.examples/src/org/jboss/reddeer/junit/annotation/simple/reddeer.xml 22:40:50.992 INFO [main][RedDeerSuite] Adding suite with name reddeer.xml to RedDeer suite 22:40:51.012 INFO [main][RequirementsRunnerBuilder] Found test class org.jboss.reddeer.junit.annotation.simple.UserTest 22:40:51.025 INFO [main][RequirementsBuilder] Creating requirements for test class org.jboss.reddeer.junit.annotation.simple.UserTest 22:40:51.027 DEBUG [main][RequirementsBuilder] Found requirement class org.jboss.reddeer.junit.annotation.simple.UserRequirement for annotation interface org.jboss.reddeer.junit.annotation.simple.UserRequirement$User 22:40:51.027 DEBUG [main][PropertyBasedConfigurator] Setting property based configuration to requirement class org.jboss.reddeer.junit.annotation.simple.UserRequirement 22:40:51.031 DEBUG [main][XMLReader] Reading configuration for class org.jboss.reddeer.junit.internal.configuration.entity.PropertyBasedConfiguration 22:40:51.827 DEBUG [main][PropertyBasedConfigurator] Configuration successfully set 22:40:51.828 INFO [main][Requirements] Requirement class org.jboss.reddeer.junit.annotation.simple.UserRequirement can be fulfilled: true 22:40:51.828 INFO [main][RequirementsRunnerBuilder] All requirements can be fulfilled, the test will run 22:40:51.865 INFO [main][RedDeerSuite] RedDeer suite created 22:40:51.874 INFO [main][Requirements] Fulfilling requirement of class org.jboss.reddeer.junit.annotation.simple.UserRequirement Fulfilling User requirement with Name: USERS_ADMINISTRATION IP: 127.0.0.1 22:40:51.875 DEBUG [main][RequirementsRunner] Injecting fulfilled requirements into test instance 22:40:51.876 INFO [main][RequirementsRunner] Started test: test reddeer.xml(org.jboss.reddeer.junit.annotation.simple.UserTest) 22:40:51.876 INFO [main][RequirementsRunner] Started test: test reddeer.xml(org.jboss.reddeer.junit.annotation.simple.UserTest) The test is running USERS_ADMINISTRATION 22:40:51.878 INFO [main][RequirementsRunner] Finished test: test reddeer.xml(org.jboss.reddeer.junit.annotation.simple.UserTest) 22:40:51.878 INFO [main][RequirementsRunner] Finished test: test reddeer.xml(org.jboss.reddeer.junit.annotation.simple.UserTest) Requirements with a Custom Schema The fourth and final approach to defining new requirements is to create a custom XML schema. This is the most complex approach, but it also provides you with the most flexibility as you can more easily share requirements in multiple configuration files. Also, this approach can protect you against forgetting to define properties in the configuration files by designating specific properties as required XML elements. To use this approach, you create a custom XML schema, then you create a configuration object in the test programs, and inject that object into your requirement. The configuration details are defined in an XML file and accessed through JAXB annotations. Let’s take a look at an example. The code for this example is available in Red Deer here: /plugins/org.jboss.reddeer.examples/src/org/jboss/reddeer/junit/configuration/advanced In order to use a custom XML schema, you need a custom schema. In this example, the schema is defined in a local file: /plugins/org.jboss.reddeer.examples/src/org/jboss/reddeer/junit/configuration/advanced/RedDeerRequirements.xsd This example schema is fairly simple, but it provides the flexibility needed for the example to define a test configuration of key=value pairs in the context of testruns and requirements. Also, the schema enforces the “required” setting for the requirement name. The configuration for requirement is defined in an XML requirement configuration file, the format of which complies with the custom schema: USERS_ADMINISTRATION 127.0.0.1 1111 In order to make use of this configuration, the Requirement class must instantiate a “UserConfiguration” object for the requirement. The UserRequirement class implements the org.jboss.reddeer.junit.requirement.CustomConfiguration interface with and specifies a type of UserConfiguration to enable the use of custom configurations: package org.jboss.reddeer.junit.configuration.advanced; import java.lang.annotation.ElementType; import java.lang.annotation.Retention; import java.lang.annotation.RetentionPolicy; import java.lang.annotation.Target; import org.jboss.reddeer.junit.requirement.CustomConfiguration; import org.jboss.reddeer.junit.requirement.Requirement; import org.jboss.reddeer.junit.configuration.advanced.UserRequirement.User; /** * User requirement using configuration from custom xml file * @author lucia jelinkova * */ public class UserRequirement implements Requirement, CustomConfiguration { @Retention(RetentionPolicy.RUNTIME) @Target(ElementType.TYPE) public @interface User { String name(); } private User user; private UserConfiguration userConfiguration; public boolean canFulfill() { // return true if you can connect to the database return true; } public void fulfill() { System.out.println("fulfiling requirement User with\nName: " + user.name() + "\nDB name: " + userConfiguration.getDbName() + "\nPort: " + userConfiguration.getPort() + "\nIP: " + userConfiguration.getIp()); // create an admin user in the database if it does not exist yet } public void setDeclaration(User user) { this.user = user; } public Class getConfigurationClass() { return UserConfiguration.class; } public void setConfiguration(UserConfiguration config) { this.userConfiguration = config; } } The UserConfiguration object (see line 25) is used by the org.jboss.reddeer.junit.requirement.CustomConfiguration class to provide the values for the requirement. The UserConfiguration definition (see below) maps the requirement as defined in the elements defined in the requirement XML file. package org.jboss.reddeer.junit.configuration.advanced; import javax.xml.bind.annotation.XmlElement; import javax.xml.bind.annotation.XmlRootElement; /** * Stores user requirement configuration loaded from custom xml file * @author lucia jelinkova * */ @XmlRootElement(name="user-requirement", namespace="http://www.jboss.org/NS/user-schema") public class UserConfiguration { private String dbName; private String ip; private String port; public String getIp() { return ip; } @XmlElement(namespace="http://www.jboss.org/NS/user-schema") public void setIp(String ip) { this.ip = ip; } public String getPort() { return port; } @XmlElement(namespace="http://www.jboss.org/NS/user-schema") public void setPort(String port) { this.port = port; } public String getDbName() { return dbName; } @XmlElement(name="db-name", namespace="http://www.jboss.org/NS/user-schema") public void setDbName(String dbName) { this.dbName = dbName; } } Note the getter and setter methods in the class definition. These methods make use of JAXB annotations to access the configuration element values. The test program looks largely the same as the test programs that we’ve used in the earlier examples. (It’s a nice characteristic of Red Deer tests in that since the “heavy lifting” is performed by the Red Deer harness, the tests can be kept simple, and therefore kept easy to maintain.) package org.jboss.reddeer.junit.configuration.advanced; import org.jboss.reddeer.junit.runner.RedDeerSuite; import org.jboss.reddeer.junit.configuration.advanced.UserRequirement.User; import org.junit.Test; import org.junit.runner.RunWith; /** * User test using configuration from custom xml file * Set VM parameter -Dreddeer.config to point to directory with requirements.xml file * -Dreddeer.config=${project_loc}/src/org/jboss/reddeer/junit/configuration/advanced * @author lucia jelinkova */ @RunWith(RedDeerSuite.class) @User(name="admin") public class UserTest { @Test public void test(){ // put your test logic here } } When the program is run, the console shows that the requirement was successfully met: 21:26:25.075 INFO [main][RedDeerSuite] Creating RedDeer suite... 21:26:25.077 INFO [main][SuiteConfiguration] Looking up configuration files defined via property reddeer.config=/jboss/local/reddeer_fork/reddeer/plugins/org.jboss.reddeer.examples/src/org/jboss/reddeer/junit/configuration/advanced/requirements.xml 21:26:25.077 INFO [main][SuiteConfiguration] Found configuration file /jboss/local/reddeer_fork/reddeer/plugins/org.jboss.reddeer.examples/src/org/jboss/reddeer/junit/configuration/advanced/requirements.xml 21:26:25.078 INFO [main][RedDeerSuite] Adding suite with name requirements.xml to RedDeer suite 21:26:25.084 INFO [main][RequirementsRunnerBuilder] Found test class org.jboss.reddeer.junit.configuration.advanced.UserTest 21:26:25.087 INFO [main][RequirementsBuilder] Creating requirements for test class org.jboss.reddeer.junit.configuration.advanced.UserTest 21:26:25.089 DEBUG [main][RequirementsBuilder] Found requirement class org.jboss.reddeer.junit.configuration.advanced.UserRequirement for annotation interface org.jboss.reddeer.junit.configuration.advanced.UserRequirement$User 21:26:25.089 DEBUG [main][CustomConfigurator] Setting custom configuration to requirement class org.jboss.reddeer.junit.configuration.advanced.UserRequirement 21:26:25.090 DEBUG [main][CustomConfigurator] Configuration object associated with requirement class org.jboss.reddeer.junit.configuration.advanced.UserRequirement is class org.jboss.reddeer.junit.configuration.advanced.UserConfiguration 21:26:25.090 DEBUG [main][XMLReader] Reading configuration for class org.jboss.reddeer.junit.configuration.advanced.UserConfiguration 21:26:25.782 DEBUG [main][CustomConfigurator] Configuration successfully set 21:26:25.832 INFO [main][Requirements] Requirement class org.jboss.reddeer.junit.configuration.advanced.UserRequirement can be fulfilled: true 21:26:25.832 INFO [main][RequirementsRunnerBuilder] All requirements can be fulfilled, the test will run 21:26:25.911 INFO [main][RedDeerSuite] RedDeer suite created 21:26:25.921 INFO [main][Requirements] Fulfilling requirement of class org.jboss.reddeer.junit.configuration.advanced.UserRequirement fulfiling requirement User with Name: admin DB name: USERS_ADMINISTRATION Port: 1111 IP: 127.0.0.1 21:26:25.922 DEBUG [main][RequirementsRunner] Injecting fulfilled requirements into test instance 21:26:25.923 INFO [main][RequirementsRunner] Started test: test requirements.xml(org.jboss.reddeer.junit.configuration.advanced.UserTest) 21:26:25.924 INFO [main][RequirementsRunner] Started test: test requirements.xml(org.jboss.reddeer.junit.configuration.advanced.UserTest) 21:26:25.925 INFO [main][RequirementsRunner] Finished test: test requirements.xml(org.jboss.reddeer.junit.configuration.advanced.UserTest) 21:26:25.925 INFO [main][RequirementsRunner] Finished test: test requirements.xml(org.jboss.reddeer.junit.configuration.advanced.UserTest) Before we move on, let’s try introducing an error in the XML test configuration and then see how Red Deer can trap that error. I don’t know about you, but avoiding typos is sometimes difficult for me. Let’s “inadvertently” remove the (required) name for the requirement. And rerun the test. This time, the console output shows: ERROR [main][XMLReader] cvc-complex-type.4: Attribute 'name' must appear on element 'user:user-requirement'. And the Junit output shows: org.jboss.reddeer.junit.configuration.RedDeerConfigurationException: Xml configuration is not valid. Recap In this post, we examined the (4) ways in which Red Deer supports creating your own custom test configurations. These methods range from simple requirements that optionally include parameters, to more complex requirements that can be defined in external XML files, either as key=value pairs or in a custom schema, that can be be shared between multiple test cases. It’s often the case that automated test runs can fail not because of bugs in software under test, but because the environment required by the test was properly initialized. Red Deer, by providing multiple approaches to create custom requirements helps you to ensure that your test failures can be more easily debugged and configuration errors are detected. What’s Next? In the next post in this series, we’ll take a look at how Red Deer makes creating new tests from scratch easier through its keystroke recorder feature. References https://github.com/jboss-reddeer/reddeer/wiki/Requirements http://www.oracle.com/technetwork/articles/javase/index-140168.html (JAXB)

I find it helpful to consider three questions when choosing a sprint goal: Why do we carry out the sprint? How do we reach its goal? And how do we know that the goal has been met? My sprint goal template therefore consists of three main parts: the actual goal, the method employed to reach the goal, and the metrics to determine if the goal has been met. It additionally provides a header section that allows you to state to which product and sprint the goal belongs, as the picture below shows. You can download the template as a PDF from romanpichler.com/tools/sprint-goal-template/ or by clicking on the image below. The template above has grown out of my experience of working with Scrum for more than ten years, and it is inspired by the scientific method and Lean Startup. Let’s have a look at the template sections in more detail. The Goal Section The goal section states why it is worthwhile to undertake the sprint. Examples are: Test an assumption about the user interaction and learn what works best for the user, for instance: “Will users be willing to register before using the product features?” Address a technical risk such as: “Does the architecture enable the desired performance?” Release a feature, for instance: “Get the reporting feature for general release.” The sprint goal hence differs from listing the user stories that should be implemented. It communicates the reason for carrying out the work, and it provides a motivation for running the sprint. The sprint goal should be shared: The product owner and the development team should believe that working towards the goal is the right thing to do. To choose the right sprint goal I find it helpful to consider the amount of uncertainty present. In the early sprints, addressing risks and testing assumptions allows me to learn about what the product should look like and do and how it is built. Once the key risks and critical assumptions have been dealt with, I like to focus on completing and optimising features, as the following picture shows: The Method Section This section addresses the question of how the goal is met. The default Scrum answer is simple: Create a (potentially shippable) product increment using the high-priority product backlog items, and demo it to the stakeholders in the sprint review meeting. But writing software and employing a product demo are not always the best methods to achieve the goal! A paper prototype can be good enough to test a visual design idea or an assumption about the user interaction, for instance. What’s more, other methods such as carrying out a usability test or releasing software to run an A/B test may well be more effective than a product demo. You should therefore carefully choose the right method and state it in this section. But don’t stop there. Determine the test group, the people who should provide feedback and data. Who these individuals are depends on the sprint goal: If you are validating an assumption about the visual design, the user interaction or the product functionality, then you probably want to collect feedback and data from the users. But if you are addressing a technical risk, then users may not be able to help you. Consider inviting a senior developer or architect from another team instead. Stating the test group clarifies who “the stakeholders” are, who is required to provide feedback so that the right product is developed. The Metrics Section The metrics section communicates how you determine if the goal has been met. Which metrics you use depends on the method chosen. For a product demo, you may state that at least two thirds of the stakeholders present should respond positively to the new feature, for instance; for a usability test, at least three of the five testers are complete the task successfully in less than a minute; and for the release of a new feature, you might say that at least 80% of the users use the new functionality at least once within five days after launching the feature. Whichever metrics you choose, make sure that they allow you to understand if and to which extent you have met the goal. The Header Section The header section consists of the two subsections “Product” and “Sprint”. They simply allow you to state which product and which sprint the goal belongs to. Customise this section according to your needs. If you work for an agencies or an IT solution provider, you could replace “Product” with “Project”, for instance. User Stories and the Sprint Goal You may be wondering how the template relates to the user stories. Let me first reiterate that your sprint goal should differ from your user stories. The goal explains the why it is a good idea to carry out the sprint an implement the stories. The user stories enable you to reach the goal. It’s a common mistake to confuse the two. To connect the template and the stories you have two options: You can state the relevant user stories in the template’s method section, or you can list them separately on the sprint backlog, as the following picture illustrates. In the picture above, the sprint goal is stated on the left to the sprint backlog, which lists the user stories and the tasks required to meet the goal in form of a task board. Learn more You can learn more about choosing effective sprint gaols and applying the sprint goal template by attending my Certified Scrum Product Owner training course. I have written in more detail about sprint planning in my book “Agile Product Management with Scrum”. Please contact me for onsite and virtual product owner training.

"And let Reform her columns roll. With thunder peal, and lightning flash..." - Ignis, "The Genius of Liberty" Vol III No. 2 Introduction In the past couple of articles we've seen how a Kanban board is able to help in the attainment of transparency and the stabilization of an agile team. Today we'll see if we can resolve one of the most common queries that result from this usage: how does a team decide which columns should appear on the board for tracking the progress of work items? The simplest case...and why it may not be enough When we set up a Kanban board in the last article, there were only three columns - or to use the correct term, "stations". These were a "Backlog" station (essentially a "To Do" list of work that has not yet been started), a station for showing which work is "In Progress", and a finally a station for representing the work that has been completed. You can't get much simpler than that, and it begs the question as to why you would want to make it more complicated. In practice however, there are at least two situations in which this minimalist approach will be found wanting: A team isn't cross-trained, and its members effectively work in skill silos. Consequently we can expect dependencies between team members, some of whom may become blocked while waiting for others to complete their part of the work. The incurral of this wasted time will not be apparent if it is all considered to be "work in progress". For example, the team may be split into developers and testers, and bottlenecks may arise as work passes between them. We may need to break Work In Progress down into further stations in order to expose this waste more fully. Bottlenecks arise due to constraints in the workflow, which is a different problem. In this situation a team might be fully cross-trained, and none of its members become blocked waiting for another. Rather, waste arises because the work itself is inefficiently staged. This often happens with activities like development, review, and test. For example if two people are required for a review, but only one is needed for development and test, then a bottleneck may well occur. Work will build-up awaiting review due to contention for these resources and the value of the investment in effort will start to depreciate. Again the incurral of this waste will not be apparent if it is all considered to be "work in progress". More stations are needed to expose it. Adding further stations These then are the two key things to consider when choosing additional stations. We're out to expose waste caused by work silos, or by the inappropriate staging of activities. Either of these can introduce constraints and become the source of bottlenecks in a value stream. Sometimes blockages can occur due to a dependency on something that must be done outside of the team. When this happens, it implies that the team are not fully in control of their own process, and consequently are unable to meet their own Definition of Done. They don't have all of the skills or resources needed. This is a problem and a contra-indication to agile practice. If it happens it's essential to make the dependency clear so that it can be challenged and removed. We may therefore choose to have an "externally blocked" column on the board to expose problems of this nature. It isn't really a station, because it doesn't represent a state in which value is added. Rather, it shows that an item has stalled within the value stream and that the team are not in a position to provide remedy. Another option is to place a red, day-glo sticky note on the ticket highlighting the seriousness of the problem. This is a clear signal that an impediment has occurred...that is to say, in Lean-Kanban terminology, it is an andon flag. In this case the flag shows that a major blockage has arisen and needs resolving. Challenging the boundaries Now we need to turn our attention to the boundaries of the board. There are two principal areas we should look at. Firstly, on the leftmost side of the board, we can see the work that a team inducts into its "Backlog" prior to actioning it. Secondly, on the rightmost side, we can see the work that the team considers to be "Done". These two boundaries are very often a source of waste. To understand why, just consider how backlogs are often allowed to grow without effective limit, and at how completed work may be permitted to accumulate in a "Done" column. These stations may not represent work in progress as far as the team is concerned, but it would be foolish to deny that they are batches too. After all, they are still part of the value stream. They represent inventory that is depreciating in value, or relevance, until something useful is done with it all. It behooves us to query the waste that is incurred, and to ask how the size of these batches may be constrained. Specifically: How can work be inducted into a backlog with minimal accumulation and delay? How can value be delivered to consumers as soon as work is completed? In short, what can be done to "lean" these process boundaries, so that inventory in the team's part of the value stream enters and exits in a "just-in-time" fashion? We can answer these questions by improving transparency still further. This can mean the refinement of the "Backlog" and "Done" columns into other, more finely-grained stations. For example, work might be building up in a Product Backlog because it is not being triaged appropriately, or perhaps because acceptance criteria are insufficiently well defined. We might be able to expose these problems by replacing a backlog with "Triaged", "Accepted", and "Ready" stations. At the other side of the board, completed work may be building up in the "Done" column because a release cannot yet be made. Additional stations such as "System Integrated", "In User Acceptance", and "Awaiting Release" could add clarity here. Removing stations The simple, 3 column board we started has now exploded into a behemoth of perhaps ten columns or more. This may seem like an excessively complex structure for a workflow and a casual observer may criticize it for being fundamentally unagile. After all, inventory should either be work in progress by an agile team, or it will be awaiting their attention or have already been completed. The criticism is a valid one but we need to bear one thing in mind: these stations are there to expose problems. Only once transparency has been attained can we hope to provide remedy. The bottlenecks, along with the diagnostic stations we added to reveal them, can then be removed. Conclusion Knowing how many "columns" to include on an agile board, and what they should be, is something of a black art to many agile teams. In this article we've looked at the issues involved in making this decision. The board of a fully cross-trained team should be elegant in its simplicity, but when problems arise we must be prepared to do some digging in order to root out their causes.

Background SPNego is RFC 4178 used for negotiation either NTLM or Kerberos based SSO. A typical use case is for web applications to reuse the authentication used by Desktops such as Windows or Linux. In this article, we will explore approaches for SPNego authentication with JBoss Enterprise Application Platform. JBoss Negotiation is the library that provides the SPNego authentication support in JBoss. This library has been integrated in JBoss EAP and WildFly Application Server. Checklist Obtain JBoss EAP from jboss.org. Enable your JavaEE Web Application for SPNego Authentication. Configure JBoss EAP for SPNego. Configure your Browsers for SPNego. Start JBoss EAP. Test your web application. Obtain JBoss EAP from jboss.org Download JBoss EAP 6.2 or newer from http://www.jboss.org/products/eap You can also use WildFly Application Server from http://www.wildfly.org. Your configuration may vary slightly. Enable your JavaEE Web Application for SPNego Authentication It is easier to use a demo web application as a starting point. You can then modify your web application for SPNego authentication. The demo web application we use for this article is called spnego-demo, by my colleague, Josef Cazek. The demo web application is available at https://github.com/kwart/spnego-demo . You can also download the spnego-demo.war from here . Fully configured web application spnego-demo.war can be obtained from this location . Copy the spnego-demo.war in your jboss-eap-6.2/standalone/deployments directory. Configure JBoss EAP for SPNego Authentication You will need to configure a couple of security domains and system properties in JBoss EAP6. There are two ways by which you can configure: either manual editing or using CLI tool. Manual Editing of configuration file standalone.xml in jboss-eap-6.2/standalone/configuration Add system properties to this file. Remember to put this block right after the extensions block (around line 25 of the configuration file). Add security domains to this file. Remember to put these blocks in the block. Using Command Line Interface to update JBoss EAP Go to the bin directory of JBoss EAP 6.2 and run the following. $ cat << EOT > $SPNEGO_TEST_DIR/cli-commands.txt /subsystem=security/security-domain=host:add(cache-type=default) /subsystem=security/security-domain=host/authentication=classic:add(login-modules=[{"code"=>"Kerberos", "flag"=>"required", "module-options"=>[ ("debug"=>"true"),("storeKey"=>"true"),("refreshKrb5Config"=>"true"),("useKeyTab"=>"true"),("doNotPrompt"=>"true"),("keyTab"=>"$SPNEGO_TEST_DIR/http.keytab"),("principal"=>"HTTP/[email protected]")]}]) {allow-resource-service-restart=true} /subsystem=security/security-domain=SPNEGO:add(cache-type=default) /subsystem=security/security-domain=SPNEGO/authentication=classic:add(login-modules=[{"code"=>"SPNEGO", "flag"=>"required", "module-options"=>[("serverSecurityDomain"=>"host")]}]) {allow-resource-service-restart=true} /subsystem=security/security-domain=SPNEGO/mapping=classic:add(mapping-modules=[{"code"=>"SimpleRoles", "type"=>"role", "module-options"=>[("[email protected]"=>"Admin"),("[email protected]"=>"User")]}]) {allow-resource-service-restart=true} /system-property=java.security.krb5.conf:add(value="$SPNEGO_TEST_DIR/krb5.conf") /system-property=java.security.krb5.debug:add(value=true) /system-property=jboss.security.disable.secdomain.option:add(value=true) :reload() EOT $ ./jboss-cli.sh -c --file=$SPNEGO_TEST_DIR/cli-commands.txt This is explained in https://github.com/kwart/spnego-demo/blob/master/README.md We will need a keytab file. In this example, we will use the Kerberos Server using ApacheDS (as explained in Appendix A). $ java -classpath kerberos-using-apacheds.jar org.jboss.test.kerberos.CreateKeytab HTTP/[email protected] httppwd http.keytab Note that the http.keytab has been configured in the security domain called "host" in standalone.conf. So place the keytab file appropriately while correcting the path defined in security domain. More information is available at https://github.com/kwart/kerberos-using-apacheds/blob/master/README.md JBoss EAP will need a keytab file. In this example we use a keytab called as http.keytab Different tools such as ktutil exist to create keytab files. Keytab files contain Kerberos Principals and encrypted keys. It is important to safeguard keytab files. It is very important that JBoss EAP configuration for keytab in the security domain "host" refers to the actual path of the keytab file. Configure your Browsers for SPNego The browsers such as Microsoft IE, Mozilla Firefox, Google Chrome, Apple Safari have different settings for enabling SPNego or Integrated Authentication. Start JBoss EAP Go to the bin directory of JBoss EAP 6.2 and either use standalone.sh (Unix/Linux) or standalone.bat to start your instance. Test your Web Application Assuming you have followed Appendix A steps to start the kerberos server and done kinit, you are ready to test the web application. In this article we have used spnego-demo, we can test that by going to http://localhost:8080/spnego-demo/ You can click on the "User Page" link and you should be able to see the principal name as "[email protected]" Appendix A Local Kerberos Server Download the zip file https://github.com/kwart/kerberos-using-apacheds/archive/master.zip Unzip the zip file into a directory. Build the package using maven. $ mvn clean package Start the Kerberos Server as $ java -jar target/kerberos-using-apacheds.jar test.ldif A krb5.conf file has been created. Login now using [email protected] $ kinit [email protected] Password for [email protected]: secret Launch Firefox via command line from where the kinit was run On MacOSX $open -a firefox http://localhost:8080/spnego-demo/ Appendix B Kerberos Command Line Utilities klist can be used to see the current kerberos tickets. $ klist Credentials cache: API:501:10 Principal: [email protected] Issued Expires Principal Feb 9 21:19:30 2014 Feb 10 07:19:27 2014 krbtgt/[email protected] kdestroy can be used to clear the current kerberos tickets. References SPNego Demo Web Application: https://github.com/kwart/spnego-demo Kerberos Server using ApacheDS: https://github.com/kwart/kerberos-using-apacheds JBoss EAP 6 http://www.jboss.org/products/eap PicketLink Open Source Project: http://www.picketlink.org Troubleshooting https://docs.jboss.org/author/display/PLINK/SPNego+Support+Questions Remember krb5.conf is important for client side kerberos interactions. You can use a environment variable on Unix/Linux/Mac systems called KRB5_CONFIG to point to your krb5.conf Acknowledgement Darran Lofthouse for the wonderful JBoss Negotiation Project and Josef Czacek for the SPNego-demo and Kerberos_using_Apache DS projects.

This is the first in a series of posts on the new “Red Deer” (https://github.com/jboss-reddeer/reddeer) open source testing framework for Eclipse. In this post, we’ll introduce Red Deer, and take a look at the some of the advantages that it offers by building a sample test program from scratch. Some of the features that Red Deer automated offers are: An easy to use, high-level API for testing standard Eclipse components Support for creating custom extensions for your own applications A requirements validation mechanism to assist you in configuring complex tests Eclipse Tooling to Assist in Creating new Projects A record and playback tool to enable you to quickly create automated tests An integration with Selenium for testing web based applications Support for running tests in a Jenkins CI environment Note that as of this writing, Red Deer is in an incubation stage. The current release is at level 0.5. The target date for the 1.0 release of Red Deer is late 2014. But, as a community-based, open source project, now is a great time to try Red Deer and make suggestions or even contribute code! A Look at Red Deer’s Architecture The Red Deer project itself is comprised of utilities and the API that supports the development and execution of automated tests. The API (the parts of the above diagram that are enclosed in dashed line boxes) can be thought of as having three layers: The top layer consists of extensions to Red Deer’s abstract classes or implementations for Eclipse components such as Views, Editors, Wizards, or Shells. For example, if you are writing tests for a feature that uses a custom Eclipse View, you can extend Red Deer’s View class by adding support for the specific functions of the feature. The advantage that this API layer gives you is that your test programs do not have to focus on manipulating the individual UI elements directly to perform operations. Your programs can instead instantiate an instance of an Eclipse component such as a View, and then use that instance’s methods to perform operations on the View. This layer of abstraction makes your test programs easier to write, understand, and maintain. The middle layer consists of the Red Deer implementations for SWT UI elements such as: Button, Combo, Label, Menu, Shell, TabItem, Table, ToolBar, Tree. This API layer supports the API’s higher level by providing the building blocks for the API’s Views, Editors, Shells, and WIzards. This middle layer of the API also provides Red Deer packages that enable your tests to enforce requirements, so that necessary setup tasks are performed before a test is run. The bottom layer consists of Red Deer packages that support the execution of tests such as: Conditions, Matchers, Widgets, Workbench, and Red Deer extensions to JUnit. What Makes Red Deer different from other Tools? A Layer of Abstraction The top-most layer of the API enables you to instantiate Eclipse UI elements as objects, and then manipulate them through their methods. The resulting code is easier to read and maintain, instead of being brittle and subject to failures when the UI changes. For example, for a test that has to open a view and press a button, without Red Deer, the test would have to navigate the top level menu, find the view menu, then the view type in that menu, then find the view open dialog, then locate the “OK” button, etc. Your test would have to spend a lot of time navigating through the UI elements before it could even begin to perform the test’s steps. With Red Deer, the code to open a view (in this case, the servers view) is simply: ServersView view = new ServersView(); view.open(); Furthermore, within that ServersView, your test program can perform operations on the View through methods which are defined in the view (and are incidentally also well debugged by the Red Deer team), instead of having to explicitly locate and manipulate the UI elements directly. For example, to obtain a list of all the servers, instead of locating the UI tree that contains the server list, and extracting that list of servers into an array, your Red Deer program can simply call the “getServers()” method. Likewise, the code to open a PackageExplorer, and then select a project within that PackageExplorer is as follows: PackageExplorer packageExplorer = new PackageExplorer(); packageExplorer.open(); packageExplorer.getProject("myTestProject").select(); And, the code to retrieve all the projects within that PackageExplorer is simply: packageExplorer.getProjects(); The result are that your tests are easier to write and maintain and you can focus on testing your application’s logic instead of writing brittle code to navigate through the application. Installing Red Deer The only prerequisites to using Red Deer are Eclipse and Java. In this post, we’ll use Eclipse Kepler and OpenJDK 1.7, running on Red Hat Enterprise Linux (RHEL) 6. To install Red Deer 0.4 (this is the latest stable milestone version as of this writing) follow these steps: Open up Eclipse Navigate to: Help->Install New Software Define a new download site using the Red Deer update site URL: http://download.jboss.org/jbosstools/updates/stable/kepler/core/reddeer/0.4.0/ Select Red Deer, click on the Finish button and Red Deer will install Now that you have Red Deer installed, let’s move onto building a new Red Deer test. Building your First Red Deer Test To create a new Red Deer test project, you make use of the Red Deer UI tooling and select New->Project->Other->Red Deer Test: Before we move on, let’s take a look at the WEB-INF/MANIFEST.MF file that is created in the project: Manifest-Version: 1.0 Bundle-ManifestVersion: 2 Bundle-Name: com.example.reddeer.sample Bundle-SymbolicName: com.example.reddeer.sample;singleton:=true Bundle-Version: 1.0.0.qualifier Bundle-ActivationPolicy: lazy Bundle-Vendor: Sample Co Bundle-RequiredExecutionEnvironment: JavaSE-1.6 Require-Bundle: org.junit, org.jboss.reddeer.junit, org.jboss.reddeer.swt, org.jboss.reddeer.eclipse The line we’re interested in is the final line in the file. These are the bundles that are required by Red Deer. After the empty project is created by the wizard, you can define a package and create a test class. Here's the code for a minimal functional test. The test will verify that the eclipse configuration is not empty. package com.example.reddeer.sample; import static org.junit.Assert.assertFalse; import java.util.List; import org.jboss.reddeer.swt.api.TreeItem; import org.jboss.reddeer.swt.impl.button.PushButton; import org.jboss.reddeer.swt.impl.menu.ShellMenu; import org.jboss.reddeer.swt.impl.tree.DefaultTree; import org.junit.Test; import org.junit.runner.RunWith; import org.jboss.reddeer.junit.runner.RedDeerSuite; @RunWith(RedDeerSuite.class) public class SimpleTest { @Test public void TestIt() { new ShellMenu("Help", "About Eclipse Platform").select(); new PushButton("Installation Details").click(); DefaultTree ConfigTree = new DefaultTree(); List ConfigItems = ConfigTree.getAllItems(); assertFalse ("The list is empty!", ConfigItems.isEmpty()); for (TreeItem item : ConfigItems) { System.out.println ("Found: " + item.getText()); } } } After you save the test's source file, you can run the test. To run the test, select the Run As->Red Deer Test option: And - there's the green bar! Simplifying Tests with Requirements Red Deer requirements enable you to define actions that you want happen before a test is executed. The advantage to using requirements is that you define the actions with annotations instead of using a @BeforeClass method. The result is that your test code is easier to read and maintain. The biggest difference between a Red Deer requirement and the the @BeforeClass annotation from the JUnit framework is that if a requirement cannot be fulfilled the test is not executed. Like everything else in Red Deer, you can make use of predefined requirements, or you can extend the feature by adding your own custom requirements. These custom requirements can be made complex and for convenience can be stored in external properties files. (We’ll take a look at defining custom requirements in a later post in this series when we examine how to create and contribute extensions to Red Deer.) The current milestone release of Red Deer provides predefined requirements that enable you to clean out your current workspace and open a perspective. Let’s add these to our example. To do this, we need to add these import statements: import org.jboss.reddeer.eclipse.ui.perspectives.JavaBrowsingPerspective; import org.jboss.reddeer.requirements.cleanworkspace.CleanWorkspaceRequirement.CleanWorkspace; import org.jboss.reddeer.requirements.openperspective.OpenPerspectiveRequirement.OpenPerspective; And these annotations: @CleanWorkspace @OpenPerspective(JavaBrowsingPerspective.class) And, we also have to a reference to org.jboss.reddeer.requirements to the required bundle list in our example’s MANIFEST.MF file: Require-Bundle: org.junit, org.jboss.reddeer.junit, org.jboss.reddeer.swt, org.jboss.reddeer.eclipse, org.jboss.reddeer.requirements When we’re done, our example looks like this: package com.example.reddeer.sample; import static org.junit.Assert.assertFalse; import java.util.List; import org.jboss.reddeer.swt.api.TreeItem; import org.jboss.reddeer.swt.impl.button.PushButton; import org.jboss.reddeer.swt.impl.menu.ShellMenu; import org.jboss.reddeer.swt.impl.tree.DefaultTree; import org.junit.Test; import org.junit.runner.RunWith; import org.jboss.reddeer.junit.runner.RedDeerSuite; import org.jboss.reddeer.eclipse.ui.perspectives.JavaBrowsingPerspective; import org.jboss.reddeer.requirements.cleanworkspace.CleanWorkspaceRequirement.CleanWorkspace; import org.jboss.reddeer.requirements.openperspective.OpenPerspectiveRequirement.OpenPerspective; @RunWith(RedDeerSuite.class) @CleanWorkspace @OpenPerspective(JavaBrowsingPerspective.class) public class SimpleTest { @Test public void TestIt() { new ShellMenu("Help", "About Eclipse Platform").select(); new PushButton("Installation Details").click(); DefaultTree ConfigTree = new DefaultTree(); List ConfigItems = ConfigTree.getAllItems(); assertFalse ("The list is empty!", ConfigItems.isEmpty()); for (TreeItem item : ConfigItems) { System.out.println ("Found: " + item.getText()); } } } Notice how we were able to add those functions to the test code, while only adding a very small amount of actual new code? Yes, it can pay to be a lazy programmer. ;-) What’s Next? What’s next for Red Deer is its continued development as it progresses through its incubation stage until its 1.0 release. What’s next for this series of posts will be discussions about: The Red Deer Recorder - To enable you to capture manual actions and convert them into test programs How you can Extend Red Deer - To provide test coverage for your plugins’ specific functions. And How you can Contribute these extensions to the Red Deer project. How you can Define Complex Requirements - To enable you to perform setup tasks for your tests. Red Deer’s Integration with Selenium - To enable you to test web interfaces provided by your plugins. Running Red Deer tests with Jenkins - To enable you to take advantage of Jenkins’ Continuous Integration (CI) test framework. Author’s Acknowledgements I’d like to thank all the contributors to Red Deer for their vision and contributions. It’s a new project, but it is growing fast! The contributors (in alphabetic order) are: Stefan Bunciak, Radim Hopp, Jaroslav Jankovic, Lucia Jelinkova, Marian Labuda, Martin Malina, Jan Niederman, Vlado Pakan, Jiri Peterka, Andrej Podhradsky, Milos Prchlik, Radoslav Rabara, Petr Suchy, and Rastislav Wagner.

Sometimes, you need your jobs to be persisted to a database. Existing solutions such as Gearman only used relational or file-based persistence, so they were a no-go for us and we went with MongoDB. Fast-forward a few months, and we have some problems with the database load. However, it's not that workers are pestering it too much: the problem was related to locks. MongoDB locking model As of 2.4, MongoDB holds write locks on an entire database for each write operation. Since atomicity is guaranteed only on a single document, this isn't usually a problem because even if you are inserting thousands of documents you are doing so in thousands of different operations that can be interleaved with queries and other inserts with a fair policy. This sometimes results in count() queries being inconsistent as documents are moved and indexes are asynchronously updated. However, write corruption is inexistent as documents are a very cohesive entity. However, atomic operations over a single document still lock the whole database, as in the case of findAndModify(), which looks for a document matching a certain query and updates it with a $set operation before returning it; all in a single shot and with the guarantee no other process will be able to perform the same operation of reading and writing at the same time. You can see this operation is ideal for implementing workers based on a pull model, each asking the database for a new job to do and locking it with '$set: {locked: true}'. However, after the number of workers increases a little bit, locks become a problem. Lock duration We cleaned up the working space collection of our MongoDB database by keeping in it only the unfinished jobs, and moving all the rest (completed or failed) to a different collection for archival. As the load increases due to new contracts, we saw the locking time increase as well: the application and the workers were insisting on the same database. The first of the problems was that after reducing the specs of our primary server, we started seeing timeouts of unrelated code even if the CPU and IO usage were low. The locks taken by workers to pick jobs were starting to take seconds or tens of seconds. Moreover, the MongoDB server started filling the logs with: Fri Dec 6 00:01:07 [conn280998] warning: ClientCursor::yield can't unlock b/c of recursive lock... I'm a user, not MongoDB guru but that seems not very good, especially given hundreds of these messages were written every day (although the queues continued to work correctly.) We did not find any explanation for these messages in the documentation, but I suppose they mean some operations are taking so long that they have to yield to make room for others, but in the case of atomic operations they can't to preserve consistency. An easy solution Since MongoDB does not have collection-wide locks yet, we decided to move the job pool and the completed job collections to a different database. In this way, we had a main database with the usual collections and one containing just these two, named with a '_queue' suffix. Note that we're still writing to the same database server: there is still the same number of connections being created by each process. This solution preallocates more space given two databases are involved, but as you know space is cheap nowadays. Both insertion of jobs and worker reads must take place on the same database. Here is where we discovered cohesion pays: if you have this information in a single place it is very easy to change configuration. If you have a singleton database, because "we should only have one database in this application, it will never change" this feature would cost you a lot. Fortunately, in our case it was about 10 lines of code, including the refactoring on the Factory Methods that created MongoDB database objects. Long term This solution is not for the long term, as we know the numbers of machines and their workers pool will increase in the future; a sufficiently high number of workers will saturate the connections available on the MongoDB server and lock the common collection until a pick of a job takes dozens of seconds. The design towards which we are moving includes one "foreman" to each machine, and many workers under his control; only the foreman polls the database and may lock the common collection. Distributing the job pool is not what we want for ease of retrieval of a job in case something goes bad (ever done a query on multiple databases?). Also, we don't want a push solution as it will involve the registration of workers or foremen to a central point of failure that assignes them their jobs. Since most of our servers are shutdown and rebooted according to the user load, we prefer a dynamic solution where a server can start picking jobs whenever it wants and stop without notifying remote machines.

A product roadmap is a high-level, strategic plan, which provides a longer-term outlook on the product. This creates a continuity of purpose, and it helps product managers and owners acquire funding for their product; it sets expectations, aligns stakeholders, and facilitates prioritization; it makes it easier to coordinate the development and launch of different products, and it provides reassurance to the customers (if the product roadmap is made public). Unfortunately, I find that many product managers and product owners struggle with their roadmaps, as they are dominated by features: There are too many features, and the features are often too detailed. This turns a roadmap into a tactical planning tool that competes with the Product Canvas or product backlog. What’s more, the features are sometimes regarded as a commitment by senior management than part of a high-level plan that is likely to change. The GO Product Roadmap Explained Faced with this situation, I have developed a new goal-oriented agile roadmap — the GO product roadmap, or “GO” for short. GO is based on my experience of teaching and coaching product managers and product owners, as well as using product roadmaps in my own business. The following pictures shows what the GO product roadmap looks like. You can download a PDF and Excel template by simply clicking on the picture. The first row of the GO roadmap depicted above contains the date or timeframe for the upcoming releases. You can work with a specific date such as 1st of March, or a period such as the first or second quarter. The second row states the name or version of the releases, for instance, iOS 7 or Windows 8.1. The third row provides the goal of each release, the reason why it is worthwhile to develop and launch it. Sample goals are to acquire or to activate users, to retain users by enhancing the user experience, or to accelerate development by removing technical debt. Working with goals shifts the conversation from debating individual features to agreeing on desired benefits making strategic product decisions. The development team, the stakeholders, and the management sponsor should all buy into the goals. The fourth row provides the features necessary to reach the goal. The features are means to an end, but not an end in themselves: They serve to create value and to reach the goal. Try to limit the number of features for each release to three, but do not state more than five. Refrain from detailing the features, and focus on the product capabilities that are necessary to meet the goal. Your product roadmap should be a high-level plan. The details should be covered in the Product Canvas or product backlog, and commitments should be limited to individual sprints. The last row states the metrics, the measurements or key performance indicators (KPIs) that help determine if the goal has been met, and if the release was successful. Make sure that the metrics you select allow you to measure if and to which extent you have met the goal. A Sample GO Product Roadmap To illustrate how the GO template can be applied, imagine we are about to develop a new dance game for girls aged eight to 12 years. The app should be fun and educational allowing the players to modify the characters, change the music, dance with remote players, and choreograph new dances. Here is what the corresponding GO roadmap could look like: While the roadmap above will have to be updated and refined at a later stage (particularly the metrics), I find it good enough to show how the product may evolve and make an investment decision. When creating your GO roadmap make sure you determine the goal of each release before you identify the features. This ensures that the features do serve the goal. Filling in the roadmap template from top to bottom and from left to right works well for me. Wrap-up The GO product roadmap provides a new, powerful way to do product roadmapping. Rather than focussing on features, GO emphasizes the importance of shared goals. This makes it easier to communicate the roadmap, create alignment, and use it as a strategic planning tool that provides an umbrella for the Product Canvas and the product backlog. The metrics provided by the tool ensure that the goals are measurable rather than lofty and fuzzy ideas. Download the template now, and try it out! You can learn more about creating effective product roadmap and working with the GO product roadmap by attending my Agile Product Planning training course. I would love to hear your questions about the roadmap and your experiences of creating product roadmaps. Please leave a comment below, or contact me.

The second blog post in the Exploring the landscape of large scale agile frameworks - series “Wet agile”, “the agile waterfall” and “the Agile-Waterfall Hybrid” … this controversial, mixed-method baby has as many names as formats. Some have received a lot of dedicated thought, are fit-for-purpose and manage to preserve the main benefits of the more pure methods. Other hybrid models have resulted by accident; either as a consequence of… … a half-way-stopped agile transitioning program that dropped dead somewhere between waterfall and scrum in an unsuccessful change initiative … a compromise between method-promoters of different ideologies (often management being on the more waterfall friendly side vs. developers generally promoting more agility) The result of the latter two cases can be horrible to witness and worse to experience for everyone involved regardless if they sit within the development organization or are on the receiving end waiting for the product. However, I do not think the frequent occurrence of poor agile-waterfall hybrids we can witness in the industry is a reason to consistently argue against mixing the two arts as many agilists do. Rather, these cases highlight the need to bring the successful and proven versions of agile-waterfall hybrids into the light and reserve a place for them among their purer cousins in the landscape for large scale agile frameworks. Founders of more successful hybrid versions often come from product development companies that deal with both software and hardware. The driver behind the method mixing in these cases is, if we simplify to the extreme, the realization that many aspects of hardware development benefit from waterfall processes, whereas software development has much to gain from an agile approach. The Agile-Waterfall Hybrid described by Erick Bergmann and Andy Hamilton from Schneider Electric is a great example of a model that merges Agile and Project Management Process/waterfall in a good way without compromising the methods’ core principles to much. Erick Bergmann and Andy Hamilton presented the Agile-Waterfall Hybrid at Agile 2013 and the key concepts of the model can be well understood from the presenting material; https://submissions.agilealliance.org/system/sessions/attachments/000/000/760/original/Agile-Waterfall_Hybrid_01AUG2013.pdf The model allows software teams to adapt agile practices while hardware development and overall product management is handled through a traditional PMP/waterfall approach The overall PMP process has a well-defined interface to the agile software development which is continuous right from the start of the concept/feasibility study up until validation and production. The interface has the format of close collaboration for all activities ranging from definition of requirements and scoping to continuous software deliveries and feedback between the agile side and the waterfallish PMP side The model accommodates development where the same/similar software is used in several products with separate POs. Eg., the software teams need to deliver features to multiple stakeholders continuously in a similar fashion as component teams typically need to act towards feature teams. The very challenging backlog management situation resulting for teams in environments like this is addressed in a good way by the model. In short, it describes how you can achieve swift product releases by putting effort into the feature releases planning on the software side There is no perfect alternative. Just like any other hybrid model, this Agile-Waterfall Hybrid compromises with some of the principles of the non-hybrid methods. The waterfall side of development is forced to live with the flexibility surrounding the software requirements and the agile teams must commit to time-fixed delivery dates, cost forecasts and risk assessment. Hybrid development models face many challenges. It is not easy to combine the dependency tracking and clarity of waterfall with the flexibility and openness to change of Agile development without diluting the benefits and create complex work processes. However, examples like the one referred to above prove that it is possible and as the industry need for these types of models is evident I hope to see more of these being published in the future.

One piece to Docker that is interesting AMAZING is the Remote API that can be used to programatically interact with docker. I recently had a situation where I wanted to run many containers on a host with a single container managing the other containers through the API. But the problem I soon discovered is that at the moment when you turn networking on it is an all or nothing type of thing… you can’t turn networking off selectively on a container by container basis. You can disable IPv4 forwarding, but you can still reach the docker remote API on the machine if you can guess the IP address of it. One solution I came up with for this is to use nginx to expose the unix socket for docker over HTTPS and utilize client-side ssl certificates to only allow trusted containers to have access. I liked this setup a lot so I thought I would share how it’s done. Disclaimer: assumes some knowledge of docker! Generate The SSL Certificates We’ll use openssl to generate and self-sign the certs. Since this is for an internal service we’ll just sign it ourselves. We also remove the password from the keys so that we aren’t prompted for it each time we start nginx. # Create the CA Key and Certificate for signing Client Certs openssl genrsa -des3 -out ca.key 4096 openssl rsa -in ca.key -out ca.key # remove password! openssl req -new -x509 -days 365 -key ca.key -out ca.crt # Create the Server Key, CSR, and Certificate openssl genrsa -des3 -out server.key 1024 openssl rsa -in server.key -out server.key # remove password! openssl req -new -key server.key -out server.csr # We're self signing our own server cert here. This is a no-no in production. openssl x509 -req -days 365 -in server.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out server.crt # Create the Client Key and CSR openssl genrsa -des3 -out client.key 1024 openssl rsa -in client.key -out client.key # no password! openssl req -new -key client.key -out client.csr # Sign the client certificate with our CA cert. Unlike signing our own server cert, this is what we want to do. openssl x509 -req -days 365 -in client.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out client.crt Another option may be to leave the passphrase in and provide it as an environment variable when running a docker container or through some other means as an extra layer of security. We’ll move ca.crt, server.key and server.crt to /etc/nginx/certs. Setup Nginx The nginx setup for this is pretty straightforward. We just listen for traffic on localhost on port 4242. We require client-side ssl certificate validation and reference the certificates we generated in the previous step. And most important of all, set up an upstream proxy to the docker unix socket. I simply overwrote what was already in /etc/nginx/sites-enabled/default. upstream docker { server unix:/var/run/docker.sock fail_timeout=0; } server { listen 4242; server localhost; ssl on; ssl_certificate /etc/nginx/certs/server.crt; ssl_certificate_key /etc/nginx/certs/server.key; ssl_client_certificate /etc/nginx/certs/ca.crt; ssl_verify_client on; access_log on; error_log /dev/null; location / { proxy_pass http://docker; proxy_redirect off; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; client_max_body_size 10m; client_body_buffer_size 128k; proxy_connect_timeout 90; proxy_send_timeout 120; proxy_read_timeout 120; proxy_buffer_size 4k; proxy_buffers 4 32k; proxy_busy_buffers_size 64k; proxy_temp_file_write_size 64k; } } One important piece to make this work is you should add the user nginx runs as to the docker group so that it can read from the socket. This could be www-data, nginx, or something else! Hack It Up! With this setup and nginx restarted, let’s first run a curl command to make sure that this setup correctly. First we’ll make a call without the client cert to double check that we get denied access then a proper one. # Is normal http traffic denied? curl -v http://localhost:4242/info # How about https, sans client cert and key? curl -v -s -k https://localhost:4242/info # And the final good request! curl -v -s -k --key client.key --cert client.crt https://localhost:4242/info For the first two we should get some run of the mill 400 http response codes before we get a proper JSON response from the final command! Woot! But wait there’s more… let’s build a container that can call the service to launch other containers! For this example we’ll simply build two containers: one that has the client certificate and key and one that doesn’t. The code for these examples are pretty straightforward and to save space I’ll leave the untrusted container out. You can view the untrusted container on github (although it is nothing exciting). First, the node.js application that will connect and display information: https = require 'https' fs = require 'fs' options = host: 172.42.1.62 port: 4242 method: 'GET' path: '/containers/json' key: fs.readFileSync('ssl/client.key') cert: fs.readFileSync('ssl/client.crt') headers: { 'Accept': 'application/json'} # not required, but being semantic here! req = https.request options, (res) -> console.log res req.end() And the Dockerfile used to build the container. Notice we add the client.crt and client.key as part of building it! FROM shykes/nodejs MAINTAINER James R. Carr ADD ssl/client* /srv/app/ssl ADD package.json /srv/app/package.json ADD app.coffee /srv/app/app.coffee RUN cd /srv/app && npm install . CMD cd /srv/app && npm start That’s about it. Run docker build . and docker run -n >IMAGE ID< and we should see a json dump to the console of the actively running containers. Doing the same in the untrusted directory should present us with some 400 error about not providing a client ssl certificate. I’ve shared a project with all this code plus a vagrant file on github for your own prusual. Enjoy!

Because Agile development teams work from a backlog of stories, one way to inject application security into software development is by writing up application security risks and activities as stories, making them explicit and adding them to the backlog so that application security work can be managed, estimated, prioritized and done like everything else that the team has to do. Security Stories SAFECode has tried to do this by writing a set of common, non-functional Security Stories following the well-known “As a [type of user] I want {something} so that {reason}” template. These stories are not customer- or user-focused: not the kind that a Product Owner would understand or care about. Instead, they are meant for the development team (architects, developers and testers). Example: As a(n) architect/developer, I want to ensure AND as QA, I want to verify that sensitive data is kept restricted to actors authorized to access it. There are stories to prevent/check for the common security vulnerabilities in applications: XSS, path traversal, remote execution, CSRF, OS command injection, SQL injection, password brute forcing. Checks for information exposure through error messages, proper use of encryption, authentication and session management, transport layer security, restricted uploads and URL redirection to un-trusted sites; and basic code quality issues: NULL pointer checking, boundary checking, numeric conversion, initialization, thread/process synchronization, exception handling, use of unsafe/restricted functions. SAFECode also includes a list of secure development practices (operational tasks) for the team that includes making sure that you’re using the latest compiler, patching the run-time and libraries, static analysis, vulnerability scanning, code reviews of high-risk code, tracking and fixing security bugs; and more advanced practices that require help from security experts like fuzzing, threat modeling, pen tests, environmental hardening. Altogether this is a good list of problems that need to be watched out for and things that should be done on most projects. But although SAFECode’s stories look like stories, they can’t be used as stories by the team. These Security Stories are non-functional requirements (NFRs) and technical constraints that (like requirements for scalability and maintainability and supportability) need to be considered in the design of the system, and may need to be included as part of the definition of done and conditions of acceptance for every user story that the team works on. Security Stories can’t be pulled from the backlog and delivered like other stories and removed from the backlog when they are done, because they are never “done”. The team has to keep worrying about them throughout the life of the project and of the system. As Rohit Sethi points out, asking developers to juggle long lists of technical constraints like this is not practical: If you start adding in other NFR constraints, such as accessibility, the list of constraints can quickly grow overwhelming to developers. Once the list grows unwieldy, our experience is that developers tend to ignore the list entirely. They instead rely on their own memories to apply NFR constraints. Since the number of NFRs continues to grow in increasingly specialized domains such as application security, the cognitive burden on developers’ memories is substantial. OWASP Evil User Stories – Hacking the Backlog Someone at OWASP has suggested an alternative, much smaller set of non-functional Evil User Stories that can be "hacked" into the backlog: A way for a security guy to get security on the agenda of the development team is by “hacking the backlog”. The way to do this is by crafting Evil User Stories, a few general negative cases that the team needs to consider when they implement other stories. Example #1. "As a hacker, I can send bad data in URLs, so I can access data and functions for which I'm not authorized." Example #2. "As a hacker, I can send bad data in the content of requests, so I can access data and functions for which I'm not authorized." Example #3. "As a hacker, I can send bad data in HTTP headers, so I can access data and functions for which I'm not authorized." Example #4. "As a hacker, I can read and even modify all data that is input and output by your application." Thinking like a Bad Guy – Abuse Cases and Abuser Stories Another way to beef up security in software development is to get the team to carefully look at the system they are building from the bad guy's perspective. In “Misuse and Abuse Cases: Getting Past the Positive”, Dr. Gary McGraw at Cigital talks about the importance of anticipating things going wrong, and thinking about behaviour that the system needs to prevent. Assume that the customer/user is not going to behave, or is actively out to attack the application. Question all of the assumptions in the design (the can’ts and won’ts), especially trust conditions – what if the bad guy can be anywhere along the path of an action (for example, using an attack proxy between the client and the server)? Abuse Cases are created by security experts working with the team as part of a critical review – either of the design or of an existing application. The goal of a review like this is to understand how the system behaves under attack/failure conditions, and document any weaknesses or gaps that need to be addressed. At Agile 2013 Judy Neher presented a hands-on workshop on how to write Abuser Stories, a lighter-weight, Agile practice which makes “thinking like a bad guy” part of the team’s job of defining and refining user requirements. Take a story, and as part of elaborating the story and listing the scenarios, step back and look at the story through a security lens. Don’t just think of what the user wants to do and can do - think about what they don’t want to do and can’t do. Get the same people who are working on the story to “put their black hats on” and think evil for a little while, brainstorm to come up with negative cases. As {some kind of bad guy} I want to {do some bad thing}… The {bad guy} doesn’t have to be a hacker. They could be an insider with a grudge or a selfish customer who is willing to take advantage of other users, or an admin user who needs to be protected from making expensive mistakes, or an external system that may not always function correctly. Ask questions like: How do I know who the user is and that I can trust them? Who is allowed to do what, and where are the authorization checks applied? Look for holes in multi-step workflows – what happens if somebody bypasses a check or tries to skip a step or do something out of sequence? What happens if an action or a check times-out or blocks or fails – what access should be allowed, what kind of information should be shown, what kind shouldn’t be? Are we interacting with children? Are we dealing with money? With dangerous command-and-control/admin functions? With confidential or pirvate data? Look closer at the data. Where is it coming from? Can I trust it? Is the source authenticated? Where is it validated – do I have to check it myself? Where is it stored (does it have to be stored)? If it has to be stored, should it be encrypted or masked (including in log files)? Who should be able to see it? Who shouldn’t be able to see it? Who can change it, and to the changes need to be audited? Do we need to make sure the data hasn't been tampered with (checksum, HMAC, digital signature)? Use this exercise to come up with refutation criteria (user can do this, but can’t do that; they can see this but they can’t see that), instead of, or as part of the conditions of acceptance for the story. Prioritize these cases based on risk, add the cases that you agree need to be taken care of as scenarios to the current story, or as new stories to the backlog if they are big enough. “Thinking like a bad guy” as you are working on a story seems more useful and practical than other story-based approaches. It doesn’t take a lot of time, and it’s not expensive. You don’t need to write Abuser Stories for every user Story and the more Abuser Stories that you do, the easier it will get – you'll get better at it, and you’ll keep running into the same kinds of problems that can be solved with the same patterns. You end up with something concrete and functional and actionable, work that has to be done and can be tested. Concrete, actionable cases like this are easier for the team to understand and appreciate – including the Product Owner, which is critical in Scrum, because the Product Owner decides what is important and what gets done. And because Abuser Stories are done in phase, by the people who are working on the stories already (rather than a separate activity that needs to be setup and scheduled) they are more likely to get done. Simple, quick, informal threat modeling like this isn’t enough to make a system secure – the team won’t be able to find and plug all of the security holes in the system this way, even if the developers are well-trained in secure software development and take their work seriously. Abuser Stories are good for identifying business logic vulnerabilities, reviewing security features (authentication, access control, auditing, password management, licensing) improving error handling and basic validation, and keeping onside of privacy regulations. Effective software security involves a lot more work than this: choosing a good framework and using it properly, watching out for changes to the system's attack surface, carefully reviewing high-risk code for design and coding errors, writing good defensive code as much as possible, using static analysis to catch common coding mistakes, and regular security testing (pen testing and dynamic analysis). But getting developers and testers to think like a bad guy as they build a system should go a long way to improving the security and robustness of your app.

Over the years I have been a panelist in many of the interviews for Java Developers. I have previously written a post titled Top 7 tips for succeeding in a technical interview for software engineers which covers few of the general guidelines. In this post I will share a mind map containing general topics covered in a Java developer interview. I have prepared this as a general reference for myself to remember the pointers and to keep a common standard across the multiple interviews. XMind gives a nice listing of the map. You can find the map here. Here is Image which you can download and use. Finally here is a old fashioned tabbed content list which is easier to copy paste. Java-Topics OOPs Encapsulation Abstraction Inheritance Interface - Abstract Class Casting IS-A vs HAS-A Relationships Aggregation vs Composition Plymorphism Method overloading vs Method Overloading Compile time vs Runtime Threads Creating threads Multitasking Synchronization Thread Transitions Marker Interface Serialization Clonnable Shallow copy vs Deep Copy Collections Map, List and Set Equals - Hashcode Legacy - Synchronized Classes JVM Stack vs Heap Memory Garbage Collection JRE, JVM, JDK Class loaders Exception Checked Vs Unchecked Exceptions Exception handling best practices try, catch, finally, throw, throws APIs Files String - StringBuffer - String Builder Java IO XML SAX Based & DOM Based JAXB - Java API for XML Binding Access specifier Access modifier public protected deafult private final static synchronized abstract transient volatile Inner/Nested Classes JavaEE Basics Packaging the Applications WAR EAR Basics MVC Servlets Listeners Lifecycle JSPs APIs JPA JAX-WS SOAP, WSDL Webservices basics Contract first vs JAX-RS RESTful and its advantages JSF This is a work in progress and I hope to refine it further. Let me know if you have any comments. - See more at: http://jyops.blogspot.ie/2013/10/a-mindmap-for-java-developer-interviews.html#sthash.K0A5wDAz.dpuf

Some months ago I wrote an article on how to transition between Scrum and a Lean Kanban operation. It's an important capability for an organization to have, because when a Scrum project finishes it is likely to enter a "leaner" BAU (Business As Usual) support phase. There are consequences arising from such a move which experienced Scrum hands may find surprising, and perhaps even a little off-putting. In this article we'll look at the shift in mindset that is required to do this. "Whoa! Something screwy has happened to our task board, it looks different" Kanban boards are subtly different to the task boards commonly used in Scrum. At first blush they might look similar. Both have columns showing the progress of user story "tickets" from a backlog through states such as in progress, peer review, in test, and done. In either case there might also be a blocked column, although it is equally acceptable to add a "blocked" sticker, or to simply invert the ticket on the board. As the name suggests, a task board will show the progress of the tasks that are needed to complete user stories. Often these tasks will be kept within horizontal swim lanes - one lane per user story. When all of the tasks are done, the user story will also move into done. Each user story therefore "chases" its tasks across the board. A Kanban board on the other hand - which is meant to deal with smaller and finer-grained pieces of work - will typically track the progress of user stories themselves across the board. The requirements should be well understood and there should be little appreciable depth to the solutioning; there will be few if any explicit tasks associated with the user stories. There is therefore no need for horizontal swim lanes to keep tasks and user stories aligned. You might also notice that Work in Progress limits are given particular emphasis in Lean Kanban. This is because scope is not timeboxed into sprints. The only way to throttle the rate of ticket throughput, and to keep it to manageable levels, is therefore by making sure that WIP limits are rigorously enforced. These are often annotated to the column headers on a Kanban board. For example, if there are 3 developers and 1 tester, the WIP for in progress would be 3, and 1 for in test. "Hey…there's just one backlog" That's right. Since there are no sprints in Lean Kanban, there can be no meaningful separation between a "sprint backlog" and a "product backlog". Instead there's just a single backlog of enqueued work items being brought into progress. This has repercussions for product ownership because you no longer have a clear separation between the prioritization that a team does for itself on a sprint backlog, and the prioritization done by a Product Owner on the product backlog. In effect you've just got a product backlog. In this situation clear product ownership can become more important then ever…or it can become a complete non-issue. "The Product Owner has too much power, he keeps jerking our chain" Since there is only one backlog, the Product Owner (or customer representative) must constantly reprioritize the user stories within it. The Product Owner needs to have more operational control in Lean Kanban than in Scrum. Developers can action tickets from the backlog on a daily or even hourly basis. There is no notion of getting a product backlog in shape before "the next sprint starts". Product Owners are therefore much more closely involved in day-to-day delivery than they would be in Scrum, and their involvement in daily standups becomes much more important. Note that the extent of a Product Owner's decision making should not extend beyond the backlog, and a good Kanban Leader will protect the team and its work in progress just like a good ScrumMaster would. "Now the Product Owner has disappeared altogether" Business as Usual work often boils down to the maintenance of existing systems post-delivery. Depending upon the level of demand, it's quite plausible to have one Lean-Kanban team responsible for the maintenance of multiple systems. In this situation there is no product being delivered as such, and consequently there is no clear product ownership. Instead, work items are raised as change requests and triaged by the team who then manage and prioritize their own backlog. This means that the team needs a strong and shared sense of direction and purpose. "There's no vision for this project" That's because a Lean Kanban operation typically isn't a project at all. A defined end point is likely to be missing… remember that it's covering "Business as Usual work". These are small, repeatable changes that may affect diverse systems and without any sort of narrative to bind them together. There'll certainly be a purpose and a rationale for operating a Lean Kanban… but don't expect a project vision. "We don't even seem to have decent sprint goals any more" Yep, they've gone too. Since there is no project vision and no sprints on a Lean Kanban, we won't have any "sprint goals" either. What we might get is a grouping of work requests that fall within a larger epic of changes…but if we do, it could well be a cause for concern. We must ask: are those related changes really representative of "Business as Usual" work, or are they too high risk? Do they constitute a project? "Lean Kanban work seems very bitty. I can't get a decent chunk to chew on" The diet of a Lean Kanban should consist of small, "digestible" pieces of work that do not require much breaking down in order to action them. By definition they must be well-understood and low-risk. A team must know how to handle them without the need for impact analysis or de-scoping. You're unlikely to get a meaty piece of work; you're more likely to be sucking these things up through a straw. Velocity and lead times are particularly significant metrics in Lean Kanban. Having said that, substantial and time consuming pieces of work can be taken on board if they satisfy the criteria of low risk and clear scope. An example would be the sort of work that conforms to a templated change. Of course, this sort of work might not appeal to an agile developer. So let's be clear: it takes a different temperament to do Lean Kanban BAU work than project work in Scrum. They are different skill sets. Agile developers who are happy doing one can find it unsettling, or even unrewarding, if they are switched to the other. "Why aren't we doing planning poker any more?" Without a sprint backlog there is no budget of story points to be brought into a sprint. This in turn means that estimation exercises such as planning poker lose much of their significance. In a Lean Kanban operation velocity can be measured not in terms of story points - either estimated or actual - but simply as the number of tickets actioned over a set period. This also provides an indication of the lead time before a ticket is handled. If tickets are of too variable a size - for example, if they include small ones as well as larger templated changes - then they can be awarded points for how long, or how much effort, they took. T-Shirt sizes is one approach. Remember that these points should represent the actuals, not estimates, so there's still no need for planning poker. Velocity can be averaged for each size. Alternatively the sizes can be mapped to points (e.g. small = 1, medium = 3, large = 7) and an aggregate velocity calculated. "Some of the BAU work that's been coming through looks like project work to me" You could well be right. It's important that you raise your suspicions with your team lead. There's often politics involved, but here's the lowdown. In many organizations "Business as Usual" work is classed - you could almost say "written off" - as an operational expenditure (OpEx), and is not drawn from the capital expenditure (CapEx) assigned to projects. Internal customers often have an incentive to sneak through initiatives as BAU work so as not to incur capital expense on their departmental budgets. This is indeed a political issue. But be on your guard otherwise your team could be hobbled with project work being slipped in on the sly. Be particularly wary of significant numbers of related changes, large changes, a seemingly high level of risk with any work items, or changes of uncertain scope. These suggest, but do not prove, that a fast one might be being pulled. Your team lead (who is analagous to a ScrumMaster) should try and defend against this, so if you as a team member have your suspicions, it's important to bring them to your lead's attention. Conclusion, and what's next In this post we've looked at the important differences between Lean Kanban and Scrum, and what that means for a team. We've also reviewed how a reasonably informed choice can be made between them. In my next post we'll look at a hybrid approach known as ScrumBan which can potentially address both project and BAU work. ScrumBan is becoming increasingly popular and has significant ramifications for project scalability.

Even among the large IT/development publications, I see inconsistencies in the use of the word front-end. Is it hyphenated or not?

When I’m teaching training courses (as I was this week at Skills Matter) or advising clients on the requirements-side of software development (which I’m doing a lot of just now), I talk about a model I call the “3 Styles of Agile.” Incredibly, I’ve never blogged about this -- although the model is hidden inside a couple of articles over the years. So now the day has come… I don’t claim the “3 Styles Model” is the way it should be, I only claim that it is the way I find the world. While “doing Agile” on the code side of software development always comes back to the same things (stand-up meetings, test/behavior driven development, code review/pair programming, stories, boards, etc.) the requirements side is very very variable. The advice that is given is very variable and the degree to which that advice is compatible with corporate structures and working is very variable. However, I find three reoccurring styles in which the requirements side operates and interfaces to development. I call these styles: Iterative, Incremental and Evolutionary, and I usually draw this diagram: I say style because I’m looking for a neutral word. I think you can use Scrum, XP and Kanban (or any other method) in any of the three styles. That said, I believe Kanban is a better fit for evolutionary while Scrum/XP are a better fit for Iterative and Incremental. I try not to be judgmental, I know a lot of Agile folk will see Evolutionary as superior, they may even consider Evolutionary to be the only True Agile but actually I don’t think that is always the case. There are times when the other styles are “right.” Let me describe the three styles: Iterative In this style the development team is doing lots of good stuff like: stand up meetings, planning meetings, short iterations or Kanban flow, test driven development, code review, refactoring, continuous integration and so on. I say they are doing it but it might be better to say “I hope they are doing” because quite often some bit or other is missing. That’s not important for this model. The key thing is the dev team are doing it! In this model, requirements arrive in a requirements document en mass. In fact, the rest of the organization carries on as if nothing has changed, indeed this may be what the organization wants. In this model you hear people say things like “Agile is a delivery mechanism” and “Agile is for developers." The requirement document may even have been written by a consultant or analyst who is now gone. The document is “thrown over the fence” to another analyst or project manager who is expected to deliver everything (scope, features) within some fixed time frame for some budget. Delivery is most likely one “big bang” at the end of the project (when the team may be dissolved). In order to do this they use a bacon slicer. I’ve written about this before and called it Salami Agile. The requirements document exists and the job of the “Product Owner” is to slice off small pieces for the team to do every iteration. The development team is insulated from the rest of the organization. There is probably still a change review board and any increase scope is seen as a problem. I call this iterative because the team is iterating but that’s about it. This is the natural style of large corporations, companies with annual budgets, senior managers who don’t understand IT and in particular banks. Incremental This style is mostly the same as Iterative, it looks similar to start with. The team are still (hopefully) doing good stuff and iterating. There is still a big requirements document, the organization still expects it all delivered and it is still being salami sliced. However, in this model, the team is delivering the software to customers. At the very least, they are demonstrating the software and listening to feedback. More likely, they are deploying the software and (potential) users can start using it today. As a result, the customer/users give feedback about what they want in the software. Sometimes this is an extra feature and functionality (scope creep!) and sometimes it is about removing things that were requested (scope retreat!). The “project” is still done in the traditional sense that everything in the document is “done,” but now some things are crossed out rather than ticked. Plus some additional stuff might be done over and above the requirements document. I call this incremental because the customers/users/stakeholders are seeing the thing grow in increments -- and hopefully early value is being delivered. I actually believe this is the most common style of software development -- whether that work is called Agile, waterfall or anything else. However, in some environments this is seen as wrong, wrong because the upfront requirements are “wrong” or because multiple deliveries need to be made, or because the team aren’t delivering everything they were originally asked to deliver. Evolutionary Here again the development team are iterating much as before. However, this time there is no requirements document. Work has begun with just an idea. Ideally I would want to see a goal, an objective, an aim, which will guide work and help inform what should be done -- and this goal should be stated in a single sentence, a paragraph at most. But sometimes even this is missing, for better or worse. In this model the requirements guy and developers both start at the beginning. They brainstorm some ideas and select something to do. While Mr. Requirements runs off to talk to customers and stakeholders about what the problem is and what is needed, the tech team (maybe just one person) gets started on the best idea so far. Sometime soon (2 weeks tops) they get back together. Mr. Requirements talks about what he has found and the developers demonstrate what they have built. They talk some more and decide what to do next. With that done, the developers gets on with building and Mr. Requirements gets on his bike again, he shows what has been built and talks to people -- some people again and some new people. As soon as possible the team starts to push software out to users and customers to use. This delivers value and provides feedback. And so it goes. It finishes, if it finishes, when the goal is met to the organization decided to use its resources somewhere else. Evolutionary style is most at home in Palo Alto, Mountain View, and anywhere else that start-ups are the norm. Evolutionary is actually a lot more common than is recognized but it is called maintenance or “bug fixing” and seen as something that shouldn’t exist. Having set out the three styles I’ll leave discussion of how to use the model and why you might use each style to another entry. If you want to know more about each model and how I see Agile as spectrum have a look my 2011 “The Agile Spectrum” from ACCU Overload or the recently revised (expanded but unfinished) version by the same title: “Agile Spectrum” (the 2013 version I suppose, online only).

There are two widely opposed (and often misunderstood) positions on how expensive it can be to change or fix software once it has been designed, coded, tested and implemented. One holds that it is extremely expensive to leave changes until late, that the cost of change rises exponentially. The other position is that changes should be left as late as possible, because the cost of changing software is – or at least can be – essentially flat (that’s why we call it software). Which position is right? Why should we care? And what can we do about it? Exponential Cost of Change Back in the early 1980s, Barry Boehm published some statistics (Software Engineering Economics, 1981) which showed that the cost of making a software change or fix increases significantly over time – you can see the original curve that he published here. Boehm looked at data collected from Waterfall-based projects at TRW and IBM in the 1970s, and found that the cost of making a change increases as you move from the stages of requirements analysis to architecture, design, coding, testing and deployment. A requirements mistake found and corrected while you are still defining the requirements costs almost nothing. But if you wait until after you've finished designing, coding and testing the system and delivering it to the customer, it can cost up to 100 times as much. A few caveats here. First, the cost curve is much higher in large projects (in smaller projects, the cost curve is more like 1:4 instead of 1:100). Those cases when the cost of change rises up to 100 times are rare, what Boehm calls Architecture-Breakers, where the team gets a fundamental architectural assumption wrong (scaling, performance, reliability) and doesn't find out until after customers are already using the system and running into serious operational problems. This analysis was all done on a small data sample from more than 30 years ago, when developing code was much more expensive and time-consuming and paperworky, and the tools sucked. A few other studies have been done since then that mostly back up Boehm's findings – at least the basic idea that the longer it takes for you to find out that you made a mistake, the more expensive it is to correct it. These studies have been widely referenced in books like Steve McConnell’s Code Complete, and used to justify the importance of early reviews and testing: Studies over the last 25 years have proven conclusively that it pays to do things right the first time. Unnecessary changes are expensive. Researchers at Hewlett-Packard, IBM, Hughes Aircraft, TRW, and other organizations have found that purging an error by the beginning of construction allows rework to be done 10 to 100 times less expensively than when it's done in the last part of the process, during system test or after release (Fagan 1976; Humphrey, Snyder, and Willis 1991; Leffingwell 1997; Willis et al. 1998; Grady 1999; Shull et al. 2002; Boehm and Turner 2004). In general, the principle is to find an error as close as possible to the time at which it was introduced. The longer the defect stays in the software food chain, the more damage it causes further down the chain. Since requirements are done first, requirements defects have the potential to be in the system longer and to be more expensive. Defects inserted into the software upstream also tend to have broader effects than those inserted further downstream. That also makes early defects more expensive. There’s some controversy over how accurate and complete this data is, how much we can rely on it, and how relevant it is today when we have much better development tools and many teams have moved from heavyweight sequential Waterfall development to lightweight iterative, incremental development approaches. Flattening the Cost of Changing Code The rules of the game should change with iterative and incremental development – because they have to. Boehm realized back in the 1980s that we could catch more mistakes early (and therefore reduce the cost of development) if we think about risks upfront and design and build software in increments, using what he called the Spiral Model, rather than trying to define, design and build software in a Waterfall sequence. The same ideas are behind more modern, lighter Agile development approaches. In Extreme Programming Explained (the first edition, but not the second) Kent Beck states that minimizing the cost of change is one of the goals of Extreme Programming, and that a flattened change cost curve is “the technical premise of XP”: Under certain circumstances, the exponential rise in the cost of changing software over time can be flattened. If we can flatten the curve, old assumptions about the best way to develop software no longer hold … You would make big decisions as late in the process as possible, to defer the cost of making the decisions and to have the greatest possible chance that they would be right. You would only implement what you had to, in hopes that the needs you anticipate for tomorrow wouldn't come true. You would introduce elements to the design only as they simplified existing code or made writing the next bit of code simpler. It’s important to understand that Beck doesn't say that with XP the change curve is flat. He says that these costs can be flattened if teams work toward this, leveraging key practices and principles in XP, such as: Simple Design, doing the simplest thing that works, and deferring design decisions as late as possible (YAGNI), so that the design is easy to understand and easy to change Continuous, disciplined refactoring to keep the code easy to understand and easy to change Test-First Development – writing automated tests upfront to catch coding mistakes immediately, and to build up a testing safety net to catch mistakes in the future Developers collaborating closely and constantly with the customer to confirm their understanding of what they need to build and working together in pairs to design solutions and solve problems, and catch mistakes and misunderstandings early Relying on working software over documentation to minimize the amount of paperwork that needs to be done with each change (write code, not specs) The team’s experience working incrementally and iteratively – the more that people work and think this way, the better they will get at it. All of this makes sense and sounds right, although there are no studies that back up these assertions, which is why Beck dropped this change curve discussion from the second edition of his XP book. But, by then, the idea that change could be flat with Agile development had already become accepted by many people. The Importance of Feedback Scott Amber agrees that the cost curve can be flattened in Agile development, not because of Simple Design, but because of the feedback loops that are fundamental to iterative, incremental development. Agile methods optimize feedback within the team, developers working closely together with each other and with the customer and relying on continuous face-to-face communications. Following technical practices like test-first development, pair programming and continuous integration makes these feedback loops even tighter. But what really matters is getting feedback from the people using the system – it’s only then that you know if you got it right or what you missed. The longer that it takes to design and build something and get feedback from real users, the more time and work that is required to get working software into a real customer’s hands, the higher your cost of change really is. Optimizing and streamlining this feedback loop is what is driving the lean startup approach to development: defining a minimum viable product (something that just barely does the job), getting it out to customers as quickly as you can, and then responding to user feedback through continuous deployment and A/B testing techniques until you find out what customers really want. Even Flat Change Can Still Be Expensive Even if you do everything to optimize these feedback loops and minimize your overheads, this still doesn’t mean that change will come cheap. Being fast isn’t good enough if you make too many mistakes along the way. The Post Agilist uses the example of painting a house: Assume that it costs $1,000 each time you paint the house, whether you paint it blue, red or white. The cost of change is flat. But if you have to paint it blue first, then red, then white before everyone is happy, you’re wasting time and money. “No matter how expensive or cheap the "cost of change" curve may be, the fewer changes that are made, the cheaper and faster the result will be … Planning is not a four letter word.” (However, I would like to point out that “plan” is.) Spending too much time upfront in planning and design is waste. But not spending enough time upfront to find out what you should be building and how you should be building it before you build it, and not taking the care to build it carefully, is also a waste. Change Gets More Expensive Over Time You also have to accept that the incremental cost of change will go up over the life of a system, especially once a system is being used. This is not just a technical debt problem. The more people using the system, the more people who might be impacted by the change if you get it wrong, the more careful you have to be. This means that you need to spend more time on planning and communicating changes, building and testing a roll-back capability, and roll changes out slowly using canary releases and dark launching – which add costs and delays to getting feedback. There are also more operational dependencies that you have to understand and take care of, and more data that you have to change or fix up, making changes even more difficult and expensive. If you do things right, keep a good team together and manage technical debt responsibly, these costs should rise gently over the life of a system – and if you don’t, that exponential change curve will kick in. What is the real cost of change? Is the real cost of change exponential, or is it flat? The truth is somewhere in between. There’s no reason that the cost of making a change to software has to be as high as it was 30 years ago. We can definitely do better today, with better tools and better, cheaper ways of developing software. The keys to minimizing the costs of change seem to be: Get your software into customer hands as quickly as you can. I am not convinced that any organization really needs to push out software changes 10 to 50 to 100 times a day, but you don’t want to wait months or years for feedback, either. Deliver less, but more often. And because you’re going to deliver more often, it makes sense to build a continuous delivery pipeline so that you can push changes out efficiently and with confidence. Use ideas from lean software development and maybe Kanban to identify and eliminate waste and to minimize cycle time. We know that, even with lots of upfront planning and design thinking, we won’t get everything right upfront -- this is the Waterfall fallacy. But it’s also important not to waste time and money iterating when you don’t need to. Spending enough time upfront in understanding requirements and in design to get it at least mostly right the first time can save a lot later on. Whether you’re working incrementally and iteratively, or sequentially, it makes good sense to catch mistakes early when you can, whether you do this through test-first development and pairing, or requirements workshops and code reviews -- whatever works for you.

A recently published academic paper by Prof. Dror Feitelson at Hebrew University, Eitan Frachtenberg a research scientist at Facebook, and Kent Beck (who is also doing something at Facebook), describes Facebook’s approach to developing and deploying its front-end software. While it would be more interesting to understand how back-end development is done (this is where the real heavy lifting is done scaling up to handle hundreds of millions of users), there are a few things in the paper that are worth knowing about. Continuous Deployment at Facebook is Not Continuous Deployment Rather than planning work out into projects or breaking work into time-boxed Sprints, Facebook developers do most of their work in independent, small changes that are released frequently. This makes sense in Facebook’s online business model, everyone constantly tuning the platform and trying out new options and applications in different user communities, seeing what sticks. It’s a credit to their architecture that so many small, independent changes can actually be done independently and cheaply. Facebook says that it follows Continuous Deployment, but it’s not Continuous Deployment the way that IMVU made popular where every change is pushed out to customers immediately, or even how a company like Etsy does Continuous Deployment. At Facebook, code can be released twice a day, but this is done mostly for bug fixes and internal code. New production code is released once per week: thousands of changes by hundreds of developers are packaged up by their small release team on Sundays, run through automated regression testing, and released on Tuesday if the developers who contributed the changes are present. Release engineers assess the risk of changes based on the size of the change, the amount of discussion done in code reviews (which is recorded through an internal code review tool), and on each developer’s “push karma”: how many problems they have seen from code by this developer before. A tool called “Gatekeeper” controls what features are available to which customers to support dark launching, and all code is released incrementally – to staging, then a subset of users, and so on. Changes can be rolled-back if necessary – individually, or, as a last resort, an entire code release. However, like a lot of Silicon Valley DevOps shops, they mostly follow the “Real Men only Roll Forward” motto. Code Ownership A key to the culture at Facebook is that developers are individually responsible for the code that they wrote, for testing it and supporting it in production. This is reflected in their code ownership model: Developers must also support the operational use of their software — a combination that’s become known as “DevOps.” This further motivates writing good code and testing it thoroughly. Developers’ personal stake in keeping the system running smoothly complements the engineering procedures and lets the system maintain quality at scale. Methodologies and tools aren’t enough by themselves because they can always be misused. Thus, a culture of personal responsibility is critical. Consequently, most source files are modified by only a few engineers. Although at least one other engineer reviews all changes before they’re committed, a third of the source files have only been edited by one engineer, and another quarter by two. Only 10 percent of the files are handled by more than seven engineers. On the other hand, the distribution of engineers per file has a heavy tail, with the most widely shared file handled by no fewer than 870 distinct engineers. These widely shared files are predominantly library files and also include major configuration and top-level PHP files. Testing? We don’t need no stinking testing … Facebook doesn't have an independent test team, because, it says, doesn'tneed one. First, they depend a lot on code reviews to find bugs: At Facebook, code review occupies a central position. Every line of code that’s written is reviewed by a different engineer than the original author. This serves multiple purposes: the original engineer is motivated to ensure that the code is of high quality, the reviewer comes with a fresh mind and might find defects or suggest alternatives, and, in general, knowledge about coding practices and the code itself spreads throughout the company. Developers are also responsible for writing unit tests and their own regression tests – they have “tens of thousands of regression tests” (which doesn't sound like nearly enough for 10+ million lines of mostly PHP code compiled into C++, in both of which languages coding mistakes are easy to make) and automated performance tests. And developers also test the software by using the development version of Facebook for their personal Facebook use. According to the authors, “this is just one aspect of the departure from traditional software development”. But Facebook developers using their own software internally (and passing this off as “testing”) is no different than the early days at Microsoft where employees were supposed to “eat their own dog food”, a practice that did little if anything to improve the quality of Microsoft products. Facebook also depends on customers to test the software for it. Software is released in steps for A/B testing and “live experimentation” on subsets of the user base, whether customers want to participate in this testing or not. Because its customer base is so large, it can get meaningful feedback from testing with even a small percentage of users, which at least minimizes the risk and inconvenience to customers. Security??? While performance is an important consideration for developers at Facebook, there is no mention of security checks or testing anywhere in this description of how Facebook develops and deploys software. No static analysis, dynamic analysis/scanning, pen testing or explanation of how the security team and developers work together, not even for “privacy sensitive code” – although this code is “held to a higher standard” it doesn’t explain what this “higher standard” is. Presumably it relies on the use of libraries and frameworks to handle at least some AppSec problems, and possibly to look for security bugs in its code reviews, but it doesn't say. There isn’t much information available on Facebook’s AppSec program anywhere. The security team at Facebook seems to spend a lot of time educating people on how to use Facebook safely and how to develop Facebook apps safely and running their bug bounty program which pays outsiders to find security bugs for them. A search on security on Facebook mostly comes back with a long list of public security failures, privacy violations and application security vulnerabilities found over the years and continuing up to the present day. Maybe the lack of an effective AppSec program is the reason for this. This is the way Facebook is Developed. Should you care? While it’s interesting to get a look inside a high-profile organization like Facebook and how it approaches development at scale, it’s not clear why this paper was written. There is little about what Facebook is doing (on its front-end development at least) that is unique or innovative, except maybe the way it uses BitTorrent to push code changes out to thousands of servers like Twitter does, something that I already heard about a few years ago at Velocity and that has been written about before. I like the idea of developers being responsible for their work, all the way into production, which is a principle that we also follow. Code reviews are good. Dark launching features is a good practice and has been a common practice in systems for a long time (even before it was called "dark launching"). Not having testers or doing AppSec is not good. Otherwise, I'm not sure what the rest of us can learn from or would want to use from this.