If you need to schedule jobs in Java, it is fairly common in the industry to use Quartz directly or via Spring integration, but you might want to think twice.

Everyone knows from reading The Mythical Man Month that as you add more people to a software development project you will see diminishing marginal returns. When you add a person to a team, there’s a short-term hit as the rest of the team slows down to bring the new team member up to speed and adjusts to working with another person, making sure that they fit in and can contribute. There’s also a long-term cost. More people means more people who need to talk to each other (n x n-1 / 2), which means more opportunities for misunderstandings and mistakes and misdirections and missed handoffs, more chances for disagreements and conflicts, more bottleneck points. As you continue to add people, the team needs to spend more time getting each new person up to speed and more time keeping everyone on the team in synch. Adding more people means that the team speeds up less and less, while people costs and communications costs and overhead costs keep going up. At some point negative returns set in – if you add more people, the team’s performance will decline and you will get less work done, not more. Diminishing Returns from any One Practice But adding too many people to a project isn’t the only case of diminishing returns in software development. If you work on a big enough project, or if you work in maintenance for long enough, you will run into problems of diminishing returns everywhere that you look. Pushing too hard in one direction, depending too much on any tool or practice, will eventually yield diminishing returns. This applies to: - Manual functional and acceptance testing - Test automation - Any single testing technique - Code reviews - Static analysis bug finding tools - Penetration tests and other security reviews Aiming for 100% code coverage on unit tests is a good example. Building a good automated regression safety net is important – as you wire in tests for key areas of the system, programmers get more confidence and can make more changes faster. How many tests are enough? In Continuous Delivery, Jez Humble and David Farley set 80% coverage as a target for each of automated unit testing, functional testing and acceptance testing. You could get by with lower coverage in many areas, higher coverage in core areas. You need enough tests to catch common and important mistakes. But beyond this point, more tests get more difficult to write, and find fewer problems. Unit testing can only find so many problems in the first place. In Code Complete, Steve McConnell explains that unit testing can only find between 15% and 50% (on average 30%) of the defects in your code. Rather than writing more unit tests, people’s time would be better spent on other approaches like exploratory system testing and code reviews or stress testing or fuzzing to find different kinds of errors. Too much of anything is bad, but too much whiskey is enough. Mark Twain, as quoted in Code Complete Refactoring is important for maintaining and improving the structure and readability of code over time. It is intended to be a supporting practice – to help make changes and fixes simpler and clearer and safer. When refactoring becomes an end in itself or turns into Obsessive Refactoring Disorder, it not only adds unnecessary costs as programmers waste time over trivial details and style issues, it can also add unnecessary risks and create conflict in a team. Make sure that refactoring is done in a disciplined way, and focus refactoring on those areas that need it the most: on code that is frequently changed, routines that are too big, too hard to read, too complex and error-prone. Putting most of your attention refactoring (or if necessary rewriting) this code will get you the highest returns. Less and Less over Time Diminishing returns also set in over time. The longer that you spend working the same way and with the same tools, the less benefits you will see. Even core practices that you’ve grown to depend on don’t pay back over time, and at some point may cost more than they are worth. It’s time again for New Year’s resolutions – time to sign up at a gym and start lifting weights. If you stick with the same routine for a couple of months, you will start to see good results. But after a while your body will get used to the work – if you keep doing the same things the same way your performance will plateau and you will stop seeing gains. You will get bored and stop going to the gym, which will leave more room for people like me. If you do keep going, trying to push harder for returns, you will overtrain and injure yourself. The same thing happens to software teams following the same practices, using the same tools. Some of this is due to inertia. Teams, organizations reach an equilibrium point and they want to stay there. Because it is comfortable, and it works – or at least they understand it. And because the better the team is working, the harder it is to get better – all the low-hanging fruit has been picked. People keep doing what worked for them in the past. They stop looking beyond their established routines, stop looking for new ideas. Competence and control lead to complacency and acceptance. Instead of trying to be as good as possible, they settle for being good enough. This is the point of inspect-and-adapt in Scrum and other time boxed methods – asking the team to regularly re-evaluate what they are doing and how they are doing it, what’s going well and what isn’t, what they should do more of or less of, challenging the status quo and finding new ways to move forward. But even the act of assessing and improving is subject to diminishing returns. If you are building software in 2-week time boxes, and you’ve been doing this for 3, 4 or 5 years, then how much meaningful feedback should you really expect from so many superficial reviews? After a while the team finds themselves going over the same issues and problems and coming up with the same results. Reviews become an unnecessary and empty ritual, another waste of time. The same thing happens with tools. When you first start using a static analysis bug checking tool for example, there’s a good chance that you will find some interesting problems that you didn’t know were in the code – maybe even more problems than you can deal with. But once you triage this and fix up the code and use the tool for a while, the tool will find fewer and fewer problems until it gets to the point where you are paying for insurance – it isn’t finding problems any more, but it might someday. In "Has secure software development reached its limits?” William Jackson argues that SDLCs – all of them – eventually reach a point of diminishing returns from a quality and security standpoint, and that Microsoft and Oracle and other big shops are already seeing diminishing returns from their SDLCs. Their software won’t get any better – all they can do is to keep spending time and money to stay where they are. The same thing happens with Agile methods like Scrum or XP – at some point you’ve squeezed everything that you can from this way or working, and the team’s performance will plateau. What can you do about diminishing returns? First, understand and expect returns to diminish over time. Watch for the signs, and factor this into your expectations – that even if you maintain discipline and keep spending on tools, you will get less and less return for your time and money. Watch for the team’s velocity to plateau or decline. Expect this to happen and be prepared to make changes, even force fundamental changes on the team. If the tools that you are using aren’t giving returns any more, then find new ones, or stop using them and see what happens. Keep reviewing how the team is working, but do these reviews differently: review less often, make the reviews more focused on specific problems, involve different people from inside and outside of the team. Use problems or mistakes as an opportunity to shake things up and challenge the status quo. Dig deep using Root Cause Analysis and challenge the team’s way of thinking and working, look for something better. Don’t settle for simple answers or incremental improvements. Remember the 80/20 rule. Most of your problems will happen in the same small number of areas, from a small number of common causes. And most of your gains will come from a few initiatives. Change the team’s driving focus and key metrics, set new bars. Use Lean methods and Lean Thinking to identify and eliminate bottlenecks, delays and inefficiencies. Look at the controls and tests and checks that you have added over time, question whether you still need them, or find steps and checks that can be combined or automated or simplified. Focus on reducing cycle time and eliminating waste until you have squeezed out what you can. Then change your focus to quality and eliminating bugs, or to simplifying the release and deployment pipeline, or some other new focus that will push the team to improve in a meaningful way. And keep doing this and pushing until you see the team slowing down and results declining. Then start again, and push the team to improve again along another dimension. Keep watching, keep changing, keep moving ahead. Source: http://swreflections.blogspot.com/2011/11/diminishing-returns-in-software.html

For most large web applications, uptime is of foremost importants. Any outage can be seen by customers as a frustration, or opportunity to move to a competitor. What's more for a site that also includes e-commerce, it can mean real lost sales. Zero Downtime describes a site without service interruption. To achieve such lofty goals, redundancy becomes a critical requirement at every level of your infrastructure. If you're using cloud hosting, are you redundant to alternate availability zones and regions? Are you using geographically distributed load balancing? Do you have multiple clustered databases on the backend, and multiple webservers load balanced. All of these requirements will increase uptime, but may not bring you close to zero downtime. For that you'll need thorough testing. The solution is to pull the trigger on sections of your infrastructure, and prove that it fails over quickly without noticeable outage. The ultimate test is the outage itself. Sean Hull on Quora: What is zero downtime and why is it important? Source: http://www.iheavy.com/2011/06/23/zero-downtime-what-is-it-and-why-is-it-important/

I’ve been going over a couple of posts by Steve Kilner that question whether Agile methods can be used effectively in software maintenance. It’s a surprising question really. There are a lot of maintenance teams who have had success following Agile methods like Scrum and Extreme Programming (XP) for some time now. We’ve been doing it for almost 5 years, enhancing and maintaining and supporting enterprise systems, and I know that it works. Agile development naturally leads into maintenance – the goal of incremental Agile development is to get working software out to customers as soon as possible, and get customers using it. At some point, when customers are relying on the software to get real business done and need support and help to keep the system running, teams cross from development over to maintenance. But there’s no reason for Agile development teams to fundamentally change the way that they work when this happens. It is harder to introduce Agile practices into a legacy maintenance team – there are a lot of technical requirements and some cultural changes that need to be made. But most maintenance teams have little to lose and lots to gain from borrowing from what Agile development teams are doing. Agile methods are designed to help small teams deal with a lot of change and uncertainty, and to deliver software quickly – all things that are at least as important in maintenance as they are in development. Technical practices in Extreme Programming especially help ensure that the code is always working – which is even more important in maintenance than it is in development, because the code has to work the first time in production. Agile methods have to be adapted to maintenance, but most teams have found it necessary to adapt these methods to fit their situations anyways. Let’s look at what works and what has to be changed to make Agile methods like Scrum and XP work in maintenance. What works well and what doesn’t Planning Game Managing maintenance isn’t the same as managing a development project – even an Agile development project. Although Agile development teams expect to deal with ambiguity and constant change, maintenance teams need to be even more flexible and responsive, to manage conflicts and unpredictable resourcing problems. Work has to be continuously reviewed and prioritized as it comes in – the customer can’t wait for 2 weeks for you to look at a production bug. The team needs a fast path for urgent changes and especially for hot fixes. You have to be prepared for support demands and interruptions. Structure the team so that some people can take care of second-level support, firefighting and emergency bug fixing and the rest of the team can keep moving forward and get something done. Build slack into schedules to allow for last-minute changes and support escalation. You will also have to be more careful in planning out maintenance work, to take into account technical and operational dependencies and constraints and risks. You’re working in the real world now, not the virtual reality of a project. Standups Standups play an important role in Agile projects to help teams come up to speed and bond. But most maintenance teams work fine without standups – since a lot of maintenance work can be done by one person working on their own, team members don’t need to listen to each other each morning talking about what they did yesterday and what they’re going to do – unless the team is working together on major changes. If someone has a question or runs into a problem, they can ask for help without waiting until the next day. Small releases Most changes and fixes that maintenance teams need to make are small, and there is almost always pressure from the business to get the code out as soon as it is ready, so an Agile approach with small and frequent releases makes a lot of sense. If the time boxes are short enough, the customer is less likely to interrupt and re-prioritize work in progress – most businesses can wait a few days or a couple of weeks to get something changed. Time boxing gives teams a way to control and structure their work, an opportunity to batch up related work to reduce development and testing costs, and natural opportunities to add in security controls and reviews and other gates. It also makes maintenance work more like a project, giving the team a chance to set goals and to see something get done. But time boxing comes with overhead – the planning and setup at the start, then deployment and reviews at the end – all of which adds up over time. Maintenance teams need to be ruthless with ceremonies and meetings, pare them down, keep only what’s necessary and what works. It’s even more important in maintenance than in development to remember that the goal is to deliver working code at the end of each time box. If some code is not working, or you’re not sure if it is working, then extend the deadline, back some of the changes out, or pull the plug on this release and start over. Don’t risk a production failure in order to hit an arbitrary deadline. If the team is having problems fitting work into time boxes, then stop and figure out what you’re doing wrong – the team is trying to do too much too fast, or the code is too unstable, or people don’t understand the code enough – and fix it and move on. Reviews and Retrospectives Retrospectives are important in maintenance to keep the team moving forward, to find better ways of working, and to solve problems. But like many practices, regular reviews reach a point of diminishing returns over time – people end up going through the motions. Once the team is setup, reviews don’t need to be done in each iteration unless the team runs into problems. Schedule reviews when you or the team need them. Collect data on how the team is working, on cycle time and bug report/fix ratios, correlate problems in production with changes, and get the team together to review if the numbers move off track. If the team runs into a serious problem like a major production failure, then get to the bottom of it through Root Cause Analysis. Sustainable pace / 40-hour week It’s not always possible to work a 40-hour week in maintenance. There are times when the team will be pushed to make urgent changes, spend late nights firefighting, releasing after hours and testing on weekends. But if this happens too often or goes on too long the team will burn out. It’s critical to establish a sustainable pace over the long term, to treat people fairly and give them a chance to do a good job. Pairing Pairing is hard to do in small teams where people are working on many different things. Pairing does make sense in some cases – people naturally pair-up when trying to debug a nasty problem or walking through a complicated change – but it’s not necessary to force it on people, and there are good reasons not to. Some teams (like mine) rely more on code reviews instead of pairing, or try to get developers to pair when first looking at a problem or change, and at the end again to review the code and tests. The important thing is to ensure that changes get looked at by at least one other person if possible, however this gets done. Collective Code Ownership Because maintenance teams are usually small and have to deal with a lot of different kinds of work, sooner or later different people will end up working on different parts of the code. It’s necessary, and it’s a good thing because people get a chance to learn more about the system and work with different technologies and on different problems. But there’s still a place for specialists in maintenance. You want the people who know the code the best to make emergency fixes or high-risk changes – or at least have them review the changes – because it has to work the first time. And sometimes you have no choice – sometimes there is only one person who understands a framework or language or technical problem well enough to get something done. Coding Guidelines – follow the rules Getting the team to follow coding guidelines is important in maintenance to help ensure the consistency and integrity of the code base over time – and to help ensure software security. Of course teams may have to compromise on coding standards and style conventions, depending on what they have inherited in the code base; and teams that maintain multiple systems will have to follow different guidelines for each system. Metaphor In XP, teams are supposed to share a Metaphor: a simple high-level expression of the system architecture (the system is a production line, or a bill of materials) and common names and patterns that can be used to describe the system. It’s a fuzzy concept at best, a weak substitute for more detailed architecture or design, and it’s not of much practical value in maintenance. Maintenance teams have to work with the architecture and patterns that are already in place in the system. What is important is making sure that the team has a common understanding of these patterns and the basic architecture so that the integrity isn’t lost – if it hasn’t been lost already. Getting the team together and reviewing the architecture, or reverse-engineering it, making sure that they all agree on it and documenting it in a simple way is important especially when taking over maintenance of a new system and when you are planning major changes. Simple Design Agile development teams start with simple designs and try to keep them simple. Maintenance teams have to work with whatever design and architecture that they inherit, which can be overwhelmingly complex, especially in bigger and older systems. But the driving principle should still be to design changes and new features as simple as the existing system lets you – and to simplify the system’s design further whenever you can. Especially when making small changes, simple, just-enough design is good – it means less documentation and less time and less cost. But maintenance teams need to be more risk adverse than development teams – even small mistakes can break compatibility or cause a run-time failure or open a security hole. This means that maintainers can’t be as iterative and free to take chances, and they need to spend more time upfront doing analysis, understanding the existing design and working through dependencies, as well as reviewing and testing their changes for regressions afterwards. Refactoring Refactoring takes on a lot of importance in maintenance. Every time a developer makes a change or fix they should consider how much refactoring work they should do and can do to make the code and design clearer and simpler, and to pay off technical debt. What and how much to refactor depends on what kind of work they are doing (making a well-thought-out isolated change, or doing shotgun surgery, or pushing out an emergency hot fix) and the time and risks involved, how well they understand the code, how good their tools are (development IDEs for Java and .NET at least have good built-in tools that make many refactorings simple and safe) and what kind of safety net they have in place to catch mistakes – automated tests, code reviews, static analysis. Some maintenance teams don’t refactor because they are too afraid of making mistakes. It’s a vicious circle – over time the code will get harder and harder to understand and change, and they will have more reasons to be more afraid. Others claim that a maintenance team is not working correctly if they don’t spend at least 50% of their time refactoring. The real answer is somewhere in between – enough refactoring to make changes and fixes safe. There are cases where extensive refactoring, restructuring or rewriting code is the right thing to do. Some code is too dangerous to change or too full of bugs to leave the way it is – studies show that in most systems, especially big systems, 80% of the bugs can cluster in 20% of the code. Restructuring or rewriting this code can pay off quickly, reducing problems in production, and significantly reducing the time needed to make changes and test them as you go forward. Continuous Testing Testing is even more important and necessary in maintenance than it is in development. And it’s a major part of maintenance costs. Most maintenance teams rely on developers to test their own changes and fixes by hand to make sure that the change worked and that they didn’t break anything as a side effect. Of course this makes testing expensive and inefficient and it limits how much work the team can do. In order to move fast, to make incremental changes and refactoring safe, the team needs a better safety net, by automating unit and functional tests and acceptance tests. It can take a long time to put in test scaffolding and tools and write a good set of automated tests. But even a simple test framework and a small set of core fat tests can pay back quickly in maintenance, because a lot changes (and bugs) tend to be concentrated in the same parts of the code – the same features, framework code and APIs get changed over and over again, and will need to be tested over and over again. You can start small, get these tests running quickly and reliably and get the team to rely on them, fill in the gaps with manual tests and reviews, and then fill out the tests over time. Once you have a basic test framework in place, developers can take advantage of TFD/TDD especially for bug fixes – the fix has to be tested anyways, so why not write the test first and make sure that you fixed what you were supposed to? Continuous Integration To get Continuous Testing to work, you need a Continuous Integration environment. Understanding, automating and streamlining the build and getting the CI server up and running and wiring in tests and static analysis checks and reporting can take a lot of work in an enterprise system, especially if you have to deal with multiple languages and platforms and dependencies between systems. But doing this work is also the foundation for simplifying release and deployment – frequent short releases means that release and deployment has to be made as simple as possible. Onsite Customer / Product Owner Working closely with the customer to make sure that the team is delivering what the customer needs when the customer needs it is as important in maintenance as it is in developing a new system. Getting a talented and committed Customer engaged is hard enough on a high-profile development project – but it’s even harder in maintenance. You may end up with too many customers with conflicting agendas competing for the team’s attention, or nobody who has the time or ability to answer questions and make decisions. Maintenance teams often have to make compromises and help fill in this role on their own. But it doesn’t all fit…. Kilner’s main point of concern isn’t really with Agile methods in maintenance. It’s with incremental design and development in general – that some work doesn’t fit nicely into short time boxes. Short iterations might work ok for bug fixes and small enhancements (they do), but sometimes you need to make bigger changes that have lots of dependencies. He argues that while Agile teams building new systems can stub out incomplete work and keep going in steps, maintenance teams have to get everything working all at once – it’s all or nothing. It’s not easy to see how big changes can be broken down into small steps that can be fit into short time boxes. I agree that this is harder in maintenance because you have to be more careful in understanding and untangling dependencies before you make changes, and you have to be more careful not to break things. The code and design will sometimes fight the kinds of changes that you need to make, because you need to do something that was never anticipated in the original design, or whatever design there was has been lost over time and any kind of change is hard to make. It’s not easy – but teams solve these problems all the time. You can use tools to figure out how much of a dependency mess you have in the code and what kind of changes you need to make to get out of this mess. If you are going to spend “weeks, months, or even years” to make changes to a system, then it makes sense to take time upfront to understand and break down build dependencies and isolate run-time dependencies, and put in test scaffolding and tests to protect the team from making mistakes as they go along. All of this can be done in time boxed steps. Just because you are following time boxes and simple, incremental design doesn’t mean that you start making changes without thinking them through. Read Working With Legacy Code – Michael Feathers walks through how to deal with these problems in detail, in both object oriented and procedural languages. What to do if it takes forever to make a change. How to break dependencies. How to find interception points and pinch points. How to find structure in the design and the code. What tests to write and how to get automated tests to work. Changing data in a production system, especially data shared with other systems, isn’t easy either. You need to plan out API changes and data structure changes as carefully as possible, but you can still make data and database changes in small, structured steps. To make code changes in steps you can use Branching by Abstraction where it makes sense (like making back-end changes) and you can protect customers from changes through Feature Flags and Dark Launching like Facebook and Twitter and Flickr do to continuously roll out changes – although you need to be careful, because if taken too far these practices can make code more fragile and harder to work with. Agile development teams follow incremental design and development to help them discover an optimal solution through trial-and-error. Maintenance teams work this way for a different reason – to manage technical risks by breaking big changes down and making small bets instead of big ones. Working this way means that you have to put in scaffolding (and remember to take it out afterwards) and plan out intermediate steps and review and test everything as you make each change. Sometimes it might feel like you are running in place, that it is taking longer and costing more. But getting there in small steps is much safer, and gives you a lot more control. Teams working on large legacy code bases and old technology platforms will have a harder time taking on these ideas and succeeding with them. But that doesn’t mean that they won’t work. Yes, you can be Agile in maintenance.

The Goal by Eli Goldratt is a business book in the form of a novel, where the protagonist must save his factory from closing due to very low productivity. The Goal is not limited to the management of a large organization (not even to for-profit companies): you simply have to define different units of measurement, like goal units instead of making money, the default goal. In fact, from the applications of the Theory of Constraints in our field I think it applies to software development too. What follows is my translation of the themes of The Goal to our field. The Goal is... Mking money, of course. For the ones of you with knowledge in accounting, the original goal is: raising throghput, the amounts of items sold (not produced) in the unit of time. Lowering investment/inventory, all the money tied up in the system in the form of assets that could be sold or products that stay in a warehouse. Lowering operational expense, all the money that we have spent as support and that cannot be recovered. How does these measurements apply to software development? A team does not always have an impact on contract negotiation, so often talking about money is far from everyday reality (kudos to you if you can apply that point of the Agile Manifesto.) The goal for software development can be translated, in my opinion, to: raising the throughput, the amount of features delivered (deployed, not implemented or tested) in the unit of time. You can measure this amount in story points, since feature vary in size. lowering investment/inventory, all the time tied up in the system in the form of undeployed or untested features that clutter the code base. In a minor part, also investment in the form of hardware, but that's by far less important than the team's time. lower operational expense, the time spent by developers every day in order to support the development. Automation is a kind of time investment that will bring more time (and quality) in the future, lowering operational expense. Like for material products, WIP has storage and opportunity costs which goes into the operational expense. Kanban is a tool that tries to reduce WIP in order to foster the two latter points. Throughput accounting This kind of throughput accounting is emphasized in the novel, over the use of cost accounting, where each developer (ehm, factory worker) has to be occupied all the time, even if the work he is doing isn't moving towards the goal: neverending refactoring. Specification of a feature which cannot be implemented until two months are gone, and will have to be rewritten. Implementation of features which won't be merged with the main branch any time soon. With Test-Driven Development, we are getting good at moving a feature from implemented to tested directly in the same commit. Yet the missing step is getting the feature to the users: maybe that's also what Continuous Deployment is all about... Dependent events Dependent events and statistical fluctuations are production systems topics that make a balanced plant close to bankruptcy: however, we're not at the point in which we can model our team as precisely as a factory. The basic point is that a plant in which everyone is working all the time is inefficient: when an early stage (like defining a specification or implementing a feature) gets delayed, downstream step such as deployment are dalayed too. Converely, when an upstream step finish earlier, the downstream stage is already at maximum efficiency and cannot process the intermediate result faster. I wonder if this applies to software development too. In a factory, workers are specialized and can do just a few jobs across the plant. Since workers and machines have different production rates, there will be just one bottleneck: the slowest one. If products have to pass from the bottleneck, anyone producing faster than the bottleneck will just accumulate WIP in front of him. Continuing with our example, if the analyst or domain expert is churning out specifications for new features every day, most of them are just WIP in front of the development team. Once there is an established buffer, any additional specification won't raise throughput any faster; instead, it will raise the inventory (partial features) and the time spent in managing it. I think this is not always true in the most technical phases of development instead. For example, in a small team a developer may be moved to testing or refactoring, or setting up Continuous Integration or evaluation of a new library. Unless you have a DBA which can just manage databases, your developer is not fixed into a stage of the system. The bottleneck The previous example featured a bottleneck, the most famous concept of the Theory of Constraints introduced in the book. This translation to the software development case is mine and could be incomplete. A feature (or a user story) has to pass in a series of stations where different people will work on it to make it real: specification from a domain expert, implementation from a technical team, extended testing with optional customer validation, deployment which should be fast but at the same time must not kill the current version of the application. Each station has an average velocity. By definition, there is a station which is the slowest and can process fewest story points in the unit of time. This is the bottleneck. (This is not always true, as velocity may vary greatly in time with the addition of new people or hidden lines discovered in a feature. Becoming good at estimation and stabilizing a team are two objectives that help reach the assumption.) You can identify a bottleneck by looking at where is the WIP: it will accumulate in front of it. If you already have a kanban board, this phase is simpler... Once identified, the throughput of the system can only be improved by raising the bottleneck's capacity enough so that it is no more a bottleneck. You can move people to it (keeping an eye on communication costs); ensure it is used at maximum efficiency by freeing the specialized developers from other mundane tasks. Now you can restart and find a new bottleneck... Conclusions This is just a little introduction to the themes of the Theory of Constraints and Goldratt's teachings; I don't pretend to explain the whole book in an article. It is also against the Socratic method: you should reach the answers yourself, and these are just examples from my experience. There is more to Goldratt and The Goal than bottlenecks and throughput, such as continuous improvement. If you're working or managing in a software development team, I suggest you to read this book if you have the opportunity. Even when freelancing, it is an eye-opener in moving towards a Goal instead of busyworking; and it's written with a never boring teaching method.

If you're looking for a cloud expert, specifically someone who knows Amazon Web Services and EC2, you'll want to have a battery of questions to assess their knowledge.

What happens when you have no product owner at all? How does a team know what features to develop in what order? Several teams I know encountered this. They all had product managers. Most of them had BAs. All of them had a technical manager who was willing to be their product owner, but they had no real product owner. They called themselves Scrum-but. I used to think this was ok. I now think Scrum-but is a bad label. That’s because agile needs a responsible person who is not part of the cross-functional technical team to rank the backlog so the team knows the order of the work. Without that person, the team does not know what to do. So why is it so bad for a team to call itself Scrum-but? Because it’s not Scrum-but. It’s not Scrum. It’s iterative and incremental, but it’s not even close to Scrum. It’s not agile. Johanna's General Agile Picture When you have no product owner who is not outside the team, or outside the hierarchy of the team, you lose something very precious to agility, the notion of the customer or customer surrogate. You lose the person who could be helping the team understand what the customer really wants. You lose the back-and-forth about the product that the customer helps the team understand. The manager can help the team understand the requirements, but the manager is not the customer. The manager is not the person who can set the real acceptance criteria. The manager can see the demo, but the manager cannot say for sure that the team is developing the correct requirements in the correct order. So why am I so insistent that we stop calling this Scrum-but, and even stop calling this agile? Because it breaks down the separation when-and-what-to-build (responsible person responsibility from ongoing incremental delivery of product on a regular basis (the cross-functional team responsibility). The customer or responsible person explains when-to-build in my little picture. The team decides how to build it. When the team manager gets involved, that allows the “business” to be unaccountable for developing the system. How do you know what is shippable product without the responsible person? The problem is this: System development, product development is a joint venture between the business people and the technical people. We need the legal, marketing, sales, and anyone else on the “business” side of the house to help us with the what-and-when to build decisions. That’s why we need a responsible person. In Scrum, that person is called a product owner. And, we need a technical project team to deliver the value. We use agile as an approach and use the demo because it shows business value every iteration. When the business is unaccountable, the agile ecosystem breaks down. We no longer have ideas coming into that funnel, being evaluated by that responsible person. Sure that responsible person has a lot to do. And, that responsible person should develop product roadmaps and make the potential product direction transparent to the rest of the organization. That way, the next iteration or two is clear for the team, and everyone can fight discuss the product direction. But when all the discussion is in the technical organization, those discussions tend to not happen. Or the discussions go off in a different direction than the product needs to go. And, that’s a Very Bad Thing. Because, when the discussions don’t occur, the technical group takes all the responsibility for the product: for what to build, when to build it, and for how to build it. And that means we have let the rest of the business abdicate all of their responsibility for their part of the product. That’s not the partnership agile promises us, nor is the transparency agile promises us. So, when you hear Scrum-but because you have no product owner, substitute “On the road to agile.” You’re actually iterative and incremental, but not agile. You have not made one of the necessary cultural changes for transitioning to agile. Can you keep doing what you are doing? Sure, if it’s working for you. And, that’s the million dollar question: How is this working for you? (If you would like more hints as to what else to do, consider my project management book, Manage It! Your Guide to Modern, Pragmatic Project Management. You have other options, if you cannot manage the agile cultural change right now. Those other options will help you move closer to agile than trying Scrum-but and failing.) This is one of the points—the agile ecosystem and making it succeed—I’m working on for my keynote at the Agile Vancouver conference in late October.

This is what Wikipedia writes about the watermelon: The Watermelon (Citrullus lanatus (Thunb.), family Cucurbitaceae) can be both the fruit and the plant of a vine-like (scrambler and trailer) plant originally from southern Africa, and is one of the most common types of melon. [...] The watermelon fruit, loosely considered a type of melon (although not in the genus Cucumis), has a smooth exterior rind (green, yellow and sometimes white) and a juicy, sweet interior flesh (usually pink, but sometimes orange, yellow, red and sometimes green if not ripe). Watermelon (Citrullus lanatus (Thunb.), family Cucurbitaceae) can be both the fruit and the plant of a vine-like (scrambler and trailer) plant originally from southern Africa, and is one of the most common types of melon. This flowering plant produces a special type of fruit known by botanists as a pepo, a berry which has a thick rind (exocarp) and fleshy center (mesocarp and endocarp); pepos are derived from an inferior ovary, and are characteristic of the Cucurbitaceae. The watermelon fruit, loosely considered a type of melon (although not in the genus Cucumis), has a smooth exterior rind (green, yellow and sometimes white) and a juicy, sweet interior flesh (usually pink, but sometimes orange, yellow, red and sometimes green if not ripe). For my metaphor, I’ll use the one with red flesh but orange and yellow would work too. I think most of us experienced the phenomenon when the project status is red but is getting greener and greener when climbing the management ladder. The project’s core is red but for the management it has a nice green paring, so it looks like a watermelon. This is why I call this phenomenon Watermelon Reporting. But why are we creating such reports and how can we avoid it? Why? The bearer of bad news already had a bad time in the ancient world. If he was lucky, they gave him the chop but in other cases they simply chopped his head of. This hasn’t changed until now but fortunately only in a figurative sense. Some bosses aren’t interested that there are problems with a project in their responsibility because if they know about it, they are in charge. So what do they do to avoid incurring the wrath of their boss ? They tweak the project status just a bit and the melon starts growing. Another reason could be that nobody wants to be in the focus of management, thus they embellish the project status in the hope that everything turns for the better. And as we all know hope is the last to die. In the end the result is the same.. Eventually the overripe melon bursts and there is no rescue for the project anymore. How to avoid it? The answer is easy: Transparency, transparency and transparency. If there is no way to hide the current status the watermelon can’t grow. Fortunately Scrum and other agile frameworks provide tools like burndown charts and backlogs to help the team with their transparency. But there are also tools like dashboards or kanban boards to do this job, but this will be the subject of one of my next blog posts. Conclusion The nuts and bolts of any project are transparency. If the project status is transparent, the watermelons can’t arise. If anybody is able to get the information, it will be difficult to hide something.

I recently attended the Agile Coach Gathering UK in Bletchley Park near London. I met a lot of interesting people, had some great talks and discussion and learned a ton. As the gathering was an open space conference I also proposed a session with the topic “What’s your favorite Agile Game?”. The goal was to collect some great games I could play in my next Scrum or Kanban trainings. A fun fact of this session was that everybody found out that we knew more games than we expected before. We came up with the following list of games. P&Q P&Q is not really a game but a collaborative process. The P&Q is a simple process which makes just two things; “P’s” and “Q’s.” The objective of the exercise is to make a decision as to how to best maximize the profit of this process. A more precise description can be found here. The XP Game The XP Game if one of the oldest and most known games in the agile community. The XP Game is a playful way to familiarize the players with some of the more difficult concepts of the XP Planning Game, like velocity, story estimation, yesterday’s weather and the cycle of life. A detailed description of the game can be found here. There are several variations of the game but my personal favorite is the LEGO(c) XP Game. I’m a big LEGO(c) fan and use any excuse to play with those bricks. Here are some photos of a team playing this game. I highly recommend this game to any team new to agile. Scrum from hell Scrum from hell is more a role play than a game and simulates a dysfunctional daily scrum meeting. It is always fun observing the participants playing the roles. The duration of this game is only 15 minutes and a must for any Scrum training. A description of this game can be found here. The communication game As I don’t have the real name of this game I just named it this way. This game is all about communication. The following roles are part of this game: Client Business Analyst Architect Developer In the first round nobody is allowed to speak. The client describes his requirements as written document to the business analyst and the BA passes on what he did understand to his architect and finally to the developer. As the info arrives at the developer he starts to build what he understood. When he is ready the review starts. In most cases the client don’t get what he has expected. In the second round everybody is allowed to speak and ask questions. In most cases this leads to a product the client asked for. If you have some more info about this game or even some artifacts, leave a comment. The ballpoint game This game is also one of my favorites. It’s about passing as many balls as possible between the players during a given time. With this game the concept of iterations/sprints and retrospective are explained. I already posted a more detailed description of this game in my blog which can be found here. Making paper hats In this game the concepts of velocity and iteration/sprint are explained. The main goal is to map the planned amount of paper hats with the actual amount. This game can also be played by blowing balloons or any other simple task. The customer in this game tries to push the development team to build as many paper hats as possible during an iteration. The result is that most of the build paper hats are useless as the quality is quite low. The customer keeps pushing until the team realizes that they are only able to build x paper hats during one iteration in the requested quality. Now the team knows his own velocity and is able to negotiate with the customer on the maximum number of paper hat. Another outcome of this game is that the player realize that quality is not negotiable. If someone has a link to a more precise description, please leave a comment. Other games There are a lot of agile games online. During the session I suggested the following places to search for additional games: http://www.tastycupcakes.com – This is a great resource for agile games. I highly recommend to have look here http://www.kanbangames.net – On this page you’ll find some games explaining the concepts of kanban and lean software development. AgileGames on Google Groups – If you’re interested in the newest games or want to discuss about games, this is the place to go. Come and join our group. If you have any other resources or any other addition to this blog post, feel free to leave a comment.

The Agile Chronicles is a set of articles documenting my experiences using an Agile process (Scrum) in software development on my current Flex project. Part 1 – Stressful Part 2 – Code Refactoring Part 3 – Branch Workflow Part 4 – POC, Strategy, and Design Challenges Part 5 – Acceptance Criteria & Punting Part 6 – Tools, Extra Merge Day, and Postponed Transitions Part 7 – Bugs, Unit Testing, and Throughput Part 8 – Demo, Burnout, and Feature Juggling Part 9 – Scope Creep Part 10 – Conclusions This entry is about defining what the acceptance criteria for user stories are so you can confirm you really did complete them during the UAT at the end of the sprint. It’s also about determining when you should abort a task that is taking too much time. Acceptance Criteria – Is the User Story Really Done? Each of our 2 week sprints include a set of user stories each developer must complete by the end of the sprint. We naturally overload our selves to ensure we have an added sense of urgency, additional user stories to tackle of we encounter a major roadblock to completing a particular story, and to clearly articulate what is priority in a bigger picture. On the 2nd Friday, we do our UAT (User Acceptance Testing). The way we do it is go through the latest build from SVN’s trunk, and collectively try to do each of the user stories. Like, “User story #32 says, ‘The user can type in their user name and password, and if they are a registered user, they will be taken to the main screen’”. If this happens, that user story is complete, we get confirmation from the client as such, and move to the next. It’s not always that black and white, though. Some user stories, even if still simple, have certain acceptance criteria associated with them. The first, and only, acceptance criteria we used in the beginning was what I just described; clearly written user stories that are easily testable. As the application grows in complexity, and certain things are assumed to go along with the user story, we’ve had to add some acceptance criteria to certain user stories. For example, even though in Sprint #1 I did in fact get the login working, none of the fonts were correct, and the alignment on some of the graphics were off. This was obvious to all, including me. Did this mean that I still completed the user story? Yes. “Yes” according to who? The client. Therefore, user story acceptance criteria seems client driven. Some clients, those not like the kind agencies have, are more functionality oriented and they don’t notice subtle design inconsistencies all the time like Verdana vs. Helvetica LT 57 Condensed. Others are more about design, and less about functionality. Some are both. Our particular client is more on the functionality side of the fence because we are working with them to complete functionality; it’s not just us working alone. That said, it still seems like each user story has it’s own unique acceptance criteria. This isn’t always easy to do, either. You really need to announce the assumptions because if you don’t, one thing you can count on is the following Friday’s UAT pointing them out; either they were assumed, and are in there, or weren’t, and aren’t. Sometimes you don’t know what those assumptions are, and thus, yet again one of the validations of using iterative development in short sprints; getting the functionality done quickly and in front of the client so they can see those assumptions. Regardless, this is something our project manager and client have been doing every Monday, both during and after, our planning session. Adding more implied functionality to a user story implies it could be more challenging, thus more work involved, and thus worth more points. This in turn affects how many user stories should be tackled per sprint. Punt – You’re in a downward spiral, pull up! You ever attempt to code something really hard, and not quite get there? Or worse, you keep getting close, yet every step feels like you’re only getting farther away as you start running out of ideas… or you have less time to implement your new ideas? This is what I call the downward spiral, akin to an airplane which stalled from going too high too quickly, and now is in a downward spiral. It can happen to those who compensate for lack of intelligence with willpower (me) and those who are smart, get in the zone, and never come out. I did that this sprint, badly. I had a really challenging component, a sub-task in a user story, and greatly underestimated my ability to pull it off. As the days wore on, my determination only increased. Two times I had a “flashback” to my Flash days, and thought about ways of faking it as well as having a plan B. Upon taking 10 minutes to test my Plan B three days in, I realized my Plan B wasn’t going to work either. I then started to do the math, and figure out how many days I had left in the Sprint, and how many user stories I had left to complete in those days. I was over what I should of been. In short, I was screwed. There is an old investment lesson called “cutting your losses”. It’s about recognizing that your investment in a particular company or mutual fund is bad. The company could be blatantly going downhill, and you can’t sell short for whatever reason (selling a hammer to someone, and then buying it back for less than you sold it for). So, the only option is to pull your money out before you lose more money. It’s the right business decision to do. The analogy is an alligator has bitten your arm. You can run, and lose your arm, or attempt to kick it, and hope it opens its mouth long enough for you to pull your arm out. This is usually destined to be bad because you could then lose your leg… or worse. Same with bad investments. If they are bad, pull your money out. The only thing you have to show for it is the money you saved. Same with aborting coding a component you won’t complete in a reasonable time frame. By stopping your aren’t giving up. People like me take it very personally, and perceive it as giving up. Jesse Warden doesn’t give up. I really have to change my mindset that, given enough time, I could complete it. However, there are more important things left to be done, so I put it “on hold”. Whatever bs you tell yourself to pull up out of the downward spiral will do. This is what I call punting. Punting is a play that you do in American football. If you’re on offense, your goal is to carry the ball into the opponents end zone. If you don’t get far enough after your allotted 4 downs, you’re going to be in trouble if the opponent gets the ball, and is closer to your end zone than you their’s. So, you punt; kick the ball as far as you can down towards their end zone, yet not on it, in the hopes you make them travel as far as possible back to your end zone. If for whatever reason, your team cannot take the ball to the opponents end zone by 3rd down, on the 4th, you need to make the decision: Do you go for it or do you not take the risk and end up screwing yourself if you don’t make it, and punt instead? In American Football, this can be a very complicated decision. In Agile software, not so much. Do the math; how many days / hours do you have to play with to hit the rest of your user stories? Is it worth it to you to work 12 hour days just in case your risk doesn’t pay off? It is it worth it to work 12 hour days EVEN IF you end up failing? I did the math too late this time, but won’t make that same mistake again. That’s what I’ll keep telling myself as I work this Saturday on Thanksgiving weekend, and next week as I work 12 hour days.

The Eclipse Communication Framework has been a steady participant in the Eclipse release trains, continuously adding to its impressive list of features. This year’s inclusion of ECF 3.5 in the Indigo release train is no exception. In this article, I'll take a look at five key features of the release: OSGi 4.2 Remote Services/RSA Standards Support ECF Indigo implements two recently-completed OSGi standards: OSGi remote services and OSGi Remote Service Admin (RSA). The OSGi Remote Services spec provides a simple, standardized way to expose OSGi services for network discovery and remote access. ECF Indigo also implements the Enterprise specification for remote services management known as Remote Services Admin (RSA). The RSA specification defines a management agent to allow for enterprise-application control of the discovery and distribution of remote services via a standardized API. Also included in the RSA specification is a standardized format for communicating meta-data about remote services, advanced handling of security, discovery and distribution event notification, and advanced handling of remote service versioning. ECF has run its implementation of RS/RSA through the OSGi Test Compatability Kit to ensure that it is compliant with the OSGi specification. Extensibility through Provider Architecture ECF has a provider architecture, that allows major components of the OSGi remote services/RSA implementation to be extended, enhanced, or replaced as needed. For example, for interoperability with existing services and applications, it’s frequently desirable to be able to substitute the wire protocol/transport to one that is already being used. With the ECF provider architecture, it’s possible to substitute the underlying protocol...and use other frameworks based upon REST, SOAP, JMS, XML-RPC, XMPP, and/or others. If you wish, you can even define and use a proprietary provider and use it to expose your remote services. Or you can use one provider for remote services development and testing, and another for deployment. Asynchronous Proxies ECF has support for remote service access via asynchronous proxies. This allows client consumers of remote services to avoid the reliability problems that are frequent when synchronous proxies are used over a relatively slow and unreliable network. The choice of whether to use synchronous or asynchronous proxies is up to the programmer, and can be made at runtime. Here is more information about this feature of ECF’s remote services implementation. XML-RPC provider ECF Indigo has an XML-RPC-based provider, which implements the remote services API. Remote Service invocation through a proxy and/or async proxy is supported too. In addition to being usable for interoperability with existing XML-RPC-based services, it can also be used as an example of how to easily use an existing framework to create a remote service provider. Google wave provider Although discontinued by Google, Wave is an open protocol with an open source implementation of the Wave server available. This means you can still build applications that take advantage of the real time shared editing functionality from within your Eclipse environment using this provider. Already, ECF provides real time shared editing using cola. This is limited to two users on a a document at a time - using the Wave provider, you could have multiple authors collaborating on the same document. Mustafa and Sebastian created a multiplayer Android phone game for EclipseCon this year, using the Wave protocol for concurrency control. Take a look at the results in the video below. ECF on Other OSGi Frameworks You're not limited to running ECF on Equinox anymore: ECF4Felix allows ECF to run on the Felix OSGi framework. So far testing has only been done on Felix. But if you are willing to help with testing ECF Remote Services/RSA on another framework, please send an email to the ecf-dev mailing list. ECF Documentation Project ECF recently started the ECF Documentation Project. This project is an approach to improve the amount and quality of the ECF documentation with the help of the committer, contributor, and consumer communities. It also aims to use of ECF for new and existing consumers. Currently this includes a Users Guide and an Integrators Guide. As a user of ECF, the documentation effort is a huge help in getting ECF to work right within your application. Great credit is due to the ECF team for this, and all other features listed here. ECF wiki: http://wiki.eclipse.org/ECF Remote services section of ECF wiki: http://wiki.eclipse.org/ECF#OSGi_Remote_Services OSGi compendium specification (Chap 13 is Remote Services): http://www.osgi.org/download/r4v42/r4.cmpn.pdf OSGi Enterprise Specification (Chap 122 is RSA): http://www.osgi.org/download/r4v42/r4.enterprise.pdf RSA wiki pages: http://wiki.eclipse.org/Remote_Services_Admin Getting Started with Remote Services: http://wiki.eclipse.org/EIG:Getting_Started_with_OSGi_Remote_Services Asynchronous Proxies (examples): http://wiki.eclipse.org/Asynchronous_Proxies_for_Remote_Services ECF Builder: https://build.ecf-project.org/jenkins/ ECF Github site (other providers, examples, Wave, and Newsreader) : https://github.com/ECF ECF4Felix: https://github.com/ECF/ECF4Felix

Here are some Java interview questions which are un-common What is the performance effect of a large number of import statements which are not used? Answer: They are ignored if the corresponding class is not used. Give a scenario where hotspot will optimize your code? Answer: If we have defined a variable as static and then initialized this variable in a static block then the Hotspot will merge the variable and the initialization in a single statement and hence reduce the code. What will happen if an exception is thrown from the finally block? Answer: The program will exit if the exception is not catched in the finally block. How does decorator design pattern works in I/O classes? Answer: The various classes like BufferedReader , BufferedWriter workk on the underlying stream classes. Thus Buffered* class will provide a Buffer for Reader/Writer classes. If I give you an assignment to design Shopping cart web application, how will you define the architecture of this application. You are free to choose any framework, tool or server? Answer: Usually I will choose a MVC framework which will make me use other design patterns like Front Controller, Business Delegate, Service Locater, DAO, DTO, Loose Coupling etc. Struts 2 is very easy to configure and comes with other plugins like Tiles, Velocity and Validator etc. The architecture of Struts becomes the architecture of my application with various actions and corresponding JSP pages in place. What is a deadlock in Java? How will you detect and get rid of deadlocks? Answer: Deadlock exists when two threads try to get hold of a object which is already held by another object. Why is it better to use hibernate than JDBC for database interaction in various Java applications? Answer: Hibernate provides an OO view of the database by mapping the various classes to the database tables. This helps in thinking in terms of the OO language then in RDBMS terms and hence increases productivity. How can one call one constructor from another constructor in a class? Answer: Use the this() method to refer to constructors. What is the purpose of intern() method in the String class? Answer: It helps in moving the normal string objects to move to the String literal pool How will you make your web application to use the https protocol? Answer: This has more to do with the particular server being used than the application itself. Here is how it can be done on tomcat: http://tomcat.apache.org/tomcat-4.1-doc/ssl-howto.html From http://extreme-java.blogspot.com/2011/05/10-tricky-java-interview-questions.html

An automated regression suite can play a vital role on a software project, valuable both for reducing defects in production and essential for evolutionary design. In talking with development teams I've often heard about the problem of non-deterministic tests - tests that sometimes pass and sometimes fail. Left uncontrolled, non-deterministic tests can completely destroy the value of an automated regression suite. In this article I outline how to deal with non-deterministic tests. Initially quarantine helps to reduce their damage to other tests, but you still have to fix them soon. Therefore I discuss treatments for the common causes for non-determinism: lack of isolation, asynchronous behavior, remote services, time, and resource leaks. I've enjoyed watching ThoughtWorks tackle many difficult enterprise applications, bringing successful deliveries to many clients who have rarely seen success. Our experiences have been a great demonstration that agile methods, deeply controversial and distrusted when we wrote the manifesto a decade ago, can be used successfully. There are many flavors of agile development out there, but in what we do there is a central role for automated testing. Automated testing was a core approach to Extreme Programming from the beginning, and that philosophy has been the biggest inspiration to our agile work. So we've gained a lot of experience in using automated testing as a core part of software development. Automated testing can look easy when presented in a text book. And indeed the basic ideas are really quite simple. But in the pressure-cooker of a delivery project, trials come up that are often not given much attention in texts. As I know too well, authors have a habit of skimming over many details in order to get a core point across. In my conversations with our delivery teams, one recurring problem that we've run into is tests which have become unreliable, so unreliable that people don't pay much attention to whether they pass or fail. A primary cause of this unreliability is that some tests have become non-deterministic. A test is non-deterministic when it passes sometimes and fails sometimes, without any noticeable change in the code, tests, or environment. Such tests fail, then you re-run them and they pass. Test failures for such tests are seemingly random. Non-determinism can plague any kind of test, but it's particularly prone to affect tests with a broad scope, such as acceptance or functional tests. Why non-deterministic tests are a problem Non-deterministic tests have two problems, firstly they are useless, secondly they are a virulent infection that can completely ruin your entire test suite. As a result they need to be dealt with as soon as you can, before your entire deployment pipeline is compromised. I'll start with expanding on their uselessness. The primary benefit of having automated tests is that they provide bug detection mechanism by acting as regression tests[1]. When a regression test goes red, you know you've got an immediate problem, often because a bug has crept into the system without you realizing. Having such a bug detector has huge benefits. Most obviously it means that you can find and fix bugs just after they are introduced. Not just does this give you the warm fuzzies because you kill bugs quickly, it also makes it easier to remove them since you know the bug got in with the last set of changes that are fresh in your mind. As a result you know where to look for the bug, which is more than half the battle in squashing it. The second level of benefit is that as you gain confidence in your bug detector, you gain the courage to make big changes knowing that when you goof, the bug detector will go off and you can fix the mistake quickly. [2] Without this teams are frightened to make the changes code needs in order to be kept clean, which leads to a rotting of the code base and plummeting development speed. The trouble with non-deterministic tests is that when they go red, you have no idea whether its due to a bug, or just part of the non-deterministic behavior. Usually with these tests a non-deterministic failure is relatively common, so you end up shrugging your shoulders when these tests go red. Once you start ignoring a regression test failure, then that test is useless and you might as well throw it away. Indeed you really ought to throw a non-deterministic test away, since if you don't it has an infectious quality. If you have a suite of 100 tests with 10 non-deterministic tests in them, than that suite will often fail. Initially people will look at the failure report and notice that the failures are in non-deterministic tests, but soon they'll lose the discipline to do that. Once that discipline is lost, then a failure in the healthy deterministic tests will get ignored too. At that point you've lots the whole game and might as well get rid of all the tests. Quarantine My principal aim in this article is to outline common cases of non-deterministic tests and how to eliminate the non-determinism. But before I get there I offer one piece of essential advice: quarantine your non-deterministic tests. If you have non-deterministic tests keep them in a different test suite to your healthy tests. That way you'll you can continue to pay attention to what's going on with your healthy tests and get good feedback from them. Place any non-deterministic test in a quarantined area. (But fix quarantined tests quickly.) Then the question is what to do with the quarantined test suites. They are useless as regression tests, but they do have a future as work items for cleaning up. You should not abandon such tests, since any tests you have in quarantine are not helping you with your regression coverage. A danger here is that tests keep getting thrown into quarantine and forgotten, which means your bug detection system is eroding. As a result it's worthwhile to have a mechanism that ensures that tests don't stay in quarantine too long. I've come across various ways to do this. One is a simple numeric limit: e.g. only allow 8 tests in quarantine. Once you hit the limit you must spend time to clear all the tests out. This has the advantage of batching up your test-cleaning if that's how you like to do things. Another route is to put a time limit on how long a test may be in quarantine, such as no longer than a week. The general approach with quarantine is to take the quarantined tests out of the main deployment pipeline so that you still get your regular build process. However a good team can be more aggressive. Our Mingle team puts its quarantine suite into the deployment pipeline one stage after its healthy tests. That way it can get the feedback from the healthy tests but is also forced to ensure that it sorts out the quarantined tests quickly. [3] Lack of Isolation In order to get tests to run reliably, you must have clear control over the environment in which they run, so you have a well-known state at the beginning of the test. If one test creates some data in the database and leaves it lying around, it can corrupt the run of another test which may rely on a different database state. Therefore I find it's really important to focus on keeping tests isolated. Properly isolated tests can be run in any sequence. As you get to larger operational scope of functional tests, it gets progressively harder to keep tests isolated. When you are tracking down a non-determinism, lack of isolation is a common and frustrating cause. Keep your tests isolated from each other, so that execution of one test will not affect any others. There are a couple of ways to get isolation - either always rebuild your starting state from scratch, or ensure that each test cleans up properly after itself. In general I prefer the former, as it's often easier - and in particular easier to find the source of a problem. If a test fails because it didn't build up the initial state properly, then it's easy to see which test contains the bug. With clean-up, however, one test will contain the bug, but another test will fail - so it's hard to find the real problem. Starting from a blank state is usually easy with unit tests, but can be much harder with functional tests [4] - particularly if you have a lot of data in a database that needs to be there. Rebuilding the database each time can add a lot of time to test runs, so that argues for switching to a clean-up strategy. One trick that's handy when you're using databases, is to conduct your tests inside a transaction, and then to rollback the transaction at the end of the test. That way the transaction manager cleans up for you, reducing the chance of errors[5]. Another approach is to do a single build of a mostly-immutable starting fixture before running a group of tests. Then ensure that the tests don't change that initial state (or if they do, they reverse the changes in tear-down). This tactic is more error-prone than rebuilding the fixture for each test, but it may be worthwhile iff it takes too long to build the fixture each time. Although databases are a common cause for isolation problems, there are plenty of times you can get these in-memory too. In particular be aware with static data and singletons. A good example for this kind of problem is contextual environment, such as the currently logged in user. If you have an explicit tear-down in a test, be wary of exceptions that occur during the tear-down. If this happens the test can pass, but cause isolation failures for subsequent tests. So ensure that if you do get a problem in a tear-down, it makes a loud noise. Some people prefer to put less emphasis on isolation and more on defining clear dependencies to force tests to run in a specified order. I prefer isolation because it gives you more flexibility in running subsets of tests and parallelizing tests. Asynchronous Behavior Asynchrony is a boon that allows you to keep software responsive while taking on long term tasks. Ajax calls allow a browser to stay responsive while going back to the server for more data, asynchronous message allow a server process to communicate with other system without being tied to their laggardly latency. But in testing, asynchrony can be curse. The common mistake here is to throw in a sleep: //pseudo-code makeAsyncCall; sleep(aWhile); readResponse; This can bite you two ways. First off you'll want to set the sleep time to long enough that it gives plenty of time to get the response. But that means that you'll spend a lot of time idly waiting for the response, thus slowing down your tests. The second bite is that, however long you sleep, sometimes it won't be enough. There will be some change in environment that will cause you to exceed the sleep - and you'll get false failure. As a result I strongly urge you to never use bare sleeps like this. Never use bare sleeps to wait for asynchonous responses: use a callback or polling. There are basically two tactics you can do for testing an asynchronous response. The first is for the asynchronous service to take a callback which it can call when done. This is the best since it means you'll never have to wait any longer than you need to [6]. The biggest problem with this is that the environment needs to be able to do this and then the service provider needs to do it. This is one of the advantages of having the development team integrated with testing - if they can provide a callback then they will. The second option is to poll on the answer. This is more than just looking once, but looking regularly, something like this //pseudo-code makeAsyncCall startTime = Time.now; while(! responseReceived) { if (Time.now - startTime > waitLimit) throw new TestTimeoutException; sleep (pollingInterval); } readResponse The point of this approach is that you can set the pollingInterval to a pretty small value, and know that that's the maximum amount of dead time you'll lose to waiting for a response. This means you can set the waitLimit very high, which minimizes the chance of hitting it unless something serious has gone wrong. [7] Make sure you use a clear exception class that indicates this is a test timeout that's failing. This will help make it clear what's gone wrong should it happen, and perhaps allow a more sophisticated test harness to take account of this information in its display. The time values, in particular the waitLimit, should never be literal values. Make sure they are always values that can be easily set in bulk, either by using constants or set through the runtime environment. That way if you need to tweak them (and you will) you can tweak them all quickly. All this advice is handy for async calls where you expect a response from the provider, but how about those where there is no response. These are calls where we invoke a command on something and expect it to happen without any acknowledgment. This is the trickiest case since you can test for your expected response, but there's nothing to do to detect a failure other than timing-out. If the provider is something you're building you can handle this by ensuring the provider implements some way of indicating that it's done - essentially some form of callback. Even if only the testing code uses it, it's worth it - although often you'll find this kind of functionality is valuable for other purposes too[8]. If the provider is someone else's work, you can try persuasion, but otherwise may be stuck. Although this is also a case when using Test Doubles for remote services is worthwhile (which I'll discuss more in the next section). If you have a general failure in something asynchronous, such that it's not responding at all, then you'll always be waiting for timeouts and your test suite will take a long time to fail. To combat this it's a good idea to use a smoke test to check that the asynchronous service is responding at all and stop the test run right away if it isn't. Gerard Meszaros's book, xUnit Test Patterns, contains lots of good patterns for constructing tests. You can also often side-step the asynchrony completely. Gerard Meszaros's Humble Object pattern says that whenever you have some logic that's in a hard-to-test environment, you should isolate the logic you need to test from that environment. In this case it means put most of the logic you need to test in a place where you can test it synchronously. The asynchronous behavior should be as minimal (humble) as possible, that way you don't need that much testing of it. Remote Services Sometimes I'm asked if ThoughtWorks does any integration work, which I find somewhat amusing since there's hardly any project we do that doesn't involve a fair bit of integration. By their nature, enterprise applications involve a great deal of combining data from different systems. These systems are maintained by other teams operating to their own schedules, teams that often use a very different software philosophy to our heavily test-driven agile approach. Testing with such remote systems brings a number of problems, and non-determinism is high on the list. Often remote systems don't have test system we can call, which means hitting a live system. If there is a test system, it may not be stable enough to provide deterministic responses. In this situation it's vital to ensure determinism, so it's time to reach for a Test Double - a component that looks like the remote service, but is really just a pretend version that mimics the remote system's behavior. The double needs to be setup so that provides the right kind of response in interaction with our system, but in a way we control. In this manner we can ensure determinism. Using a double has a downside, in particular when we are testing across a broad scope. How can we be sure that the double behaves in the same way that remote system does? We can tackle this again using tests, a form of test that I call Integration Contract Tests. These run the same interaction with the remote system and the double, and check that the two match. In this case 'match' may not mean coming up with the same result (due to the non-determinisms), but results that share the same essential structure. Integration Contract Tests need to be run frequently, but not part of our system's deployment pipeline. Periodic running based on the rate of the change of the remote system is usually best. For writing these kinds of test doubles, I'm a big fan of Self Initializing Fakes - since these are very simple to manage. Some people are firmly against using Test Doubles in functional tests, believing that you must test with real connection in order to ensure end-to-end behavior. While I sympathize with their argument, automated tests are useless if they are non-deterministic. So any advantage you gain by talking to the real system is overwhelmed by the need to stamp out non-determinism[9]. Time Few things are more non-deterministic than a call to the system clock. Each time you call it, you get a new result, and any tests that depend on it can thus change. Ask for all the todos due in the next hour, and you regularly get a different answer[10]. The most important thing here is to ensure that you always wrap the system clock with routines that can be replaced with a seeded value for testing. A clock stub can be set to particular time and frozen at that time, allowing your tests to have complete control over its movements. That way you can synchronize your test data to the values in the seeded clock.[11][12] Always wrap the system clock, so it can be easily substituted for testing. One thing to watch with this, is that eventually your test data might start having problems because it's too old, and you get conflicts with other time based factors in your application. In this case you can move the data, and your clock seeds to new values. When you do this, ensure that this is the only thing you do. That way you can be sure that any tests that fail are due to time-movement in the test data. Another area where time can be a problem is when you rely on other behaviors from the clock. I once saw a system that generated random keys based on clock values. This systems started failing when it was moved to a faster machine that could allocate multiple ids within a single clock tick.[13] I've heard so many problems due to direct calls to the system clock that I'd argue for finding a way to use code analysis to detect any direct calls to the system clock and failing the build right there. Even a simple regex check might save you a frustrating debugging session after a call at an ungodly hour. Resource Leaks If your application has some kind of resource leak, this will lead to random tests failing, since it's just which test causes the resource leak to go over a limit that gets the failure. This case is awkward because any test can fail intermittently due to this problem. If it isn't a case of one test being non-deterministic then resource leaks are a good candidate to investigate. By resource leak, I mean any resource that the application has to manage by acquiring and releasing. In non-memory-managed environments, the obvious example is memory. Memory-management did much to remove this problem, but other resources still need to be managed, such as database connections. Usually the best way to handle these kind of resources is through a Resource Pool. If you do this then a good tactic is to configure the pool to a size of 1 and make it throw an exception should it get a request for a resource when it has none left to give. That way the first test to request a resource after the leak will fail - which makes it a lot easier to find the problem test. This idea of limiting resource pool sizes, is about increasing constraints to make errors more likely to crop up in tests. This is good because we want errors to show in tests so we can fix them before they manifest themselves in production. This principle can be used in other ways too. One story I heard was of a system which generated randomly named temporary files, didn't clean them up properly, and crashed on a collision. This kind of bug is very hard to find, but one way to manifest it is to stub the randomizer for testing so it always returns the same value. That way you can surface the problem more quickly.

In this multi-part post, I’m going to share my personal experiences while working with user stories for gathering, tracking and planning requirements. It currently consists out of three parts: What are they and why do you need them? Who writes them and how do you control scope? Where do you start and what about the planning? You can also download all parts as one comprehensive PDF for easy printing or e-reading. Where do you start? Suppose that after intensive discussions and tough scoping sessions you ended up with a list of user stories and are about to start building the system. The first story not only needs to realize some particular feature, but also involves building a skeleton implementation of the system’s architecture. How do you avoid spending way too much time on plumbing and other general purpose stuff you need for the rest of the stories? The article Managing the Bootstrap Story by Jennitta Andrea addressed this challenge in more detail and offers some alternative solutions. One of these solutions is to find and define a user story with the product owner that offers minimal functionality yet still has project value. Such a story is often referred to as the backbone story because you realize the backbone of your system in it. It’s quite common to use the backbone story to realize a proof-of-concept (PoC) that verifies the chosen architecture. Since a working PoC can give the product owner confidence that the team is able to build such a product, that fact alone may be enough project value for the product owner. More storyotypes? You might have suspected it already, but that backbone story is just an example of another storyotype. In fact, after I started looking for an approach to capture the non-functional requirements of a project or system, I ran into a slide deck that mentioned a whole set of additional storyotypes. Dan Rawsthorne, the author, tried to define a storyotype for virtually every possible thing you might need to do in a project. Personally I think he went a bit too far, but a small set of additional storyotypes proved to be very useful anyway. Storyotype Description Compound Epic A composite user story that groups a number of stories in a logical sense. Complex Epic A user story whose content and impact must be determined later in the project, but for which it is clear that it involves a significant amount of work. Setup A story that is used to setup the project environment, including a source control environment, a project website, a build server. Technical A story that involves making a technical improvement or adjustment. Examples include introducing a coding standard, refactoring a poor design, executing a performance test. Documentation A story for writing a user manual, installation manual, etc. Training A story for developing and/or hosting a training, or having a workshop with end users. Quality Improvements A story which objective is to fix a collection of related bugs, or spent a fixed amount of time to improve the quality of the code base. Spike A story that aims to do a technical investigation to determine the usability of a specific technology, or for trying an alternative technical solution. When is the story complete? So how do you know that a user story has been successfully realized? Well, if all is good, all stories will conform with INVEST and are associated with a number of acceptance criteria (typically written down as the how-to demo) specified by the product owner. That should be enough to determine if it is functionally sound. But what you still miss is a way of explaining the stakeholders, including the product owner, when the team treats the story as finished. That may differ by team, but usually includes some or more of the following criteria. The code compiles and there are no warnings or errors. The code meets the coding standards setup by the project or the organization. The code is reviewed by a peer developer. All automated unit and integration tests have completed successfully. Visual Studio’s static code analysis tool does not report any violations. ReSharper reports no potential errors (a.k.a. everything is 'green'). The daily integration build has completed successfully. The functionality was tested by another member of the team (anybody but the developer). The feature or functionality has been signed off using the project checklist. The system functionality is tested by a tester. The visual look and feel is has been approved by an employee of the communications department. Together with the story’s how-to demo these criteria are commonly referred to as the definition-of-done. Usually, a team or project will have a default definition-of-done that applies to all stories and only mentions the particulars of that story if necessary. Then what about the planning? User stories are an excellent unit for tracking progress within your project. However, purists within the Agile community will tell you that an Agile project will have no long term plan. Instead, the functionality is realized iteratively according to the priority defined by the product owner. I agree with the latter and believe that its iterative nature is essential for dealing with the changing requirements that are common in all projects. It allows deferring decisions to the last responsible moment, and that’s always a good thing. But in reality you often can’t escape from providing at least a rough schedule to your management. How should you deal with that? What I often do to get all stakeholders to join me in a number of workshops. Using use case diagrams to illustrate the context of the discussions, I try to get enough stories on paper to represent the entire scope of the project. You need to beware though that you don’t write down too much details or have too much in-depth discussions. That would give the stakeholders a false sense of precision, and consequently, will cause them to see the stories as a formal functional design. Also, if you run into some high-level chunk of functionality for which nobody really knows what it will look like, add an epic story for it and include a spike to elaborate on the epic later on in the project. Then organize a number of shorter meetings with the team or, if the team hasn’t been formed yet, with a few experienced developers. Let them discuss every story one by one and then try to estimate the size of each story in so called story points. Some people from the Agile community say you should estimate using relative sizes only. In other words, a story that seems to require twice as much work as another story should also have twice as many story points. The story point as a unit does not have value. It’s the relative differences that are important. What works for me is that every story point corresponds to the ideal day of an experienced senior software developer. In other words, one story point means that an experienced developer familiar with the chosen architecture, technology and project methodology needs to work for 8 hours without being disturbed by telephone, email, coffee breaks, or any other distractions. Mike Cohn, author of User Stories Applied, has dedicated many chapters to this estimation technique. Ideally, each story is between 1 and 8 story points, but at the beginning of the project you still may have some epics to break up. After finishing those meetings you should have an estimate of the total size of the project. Now, in order to get from those story points to a total number of hours you need to estimate the expected productivity of the team. Mike Cohn does this by creating a table with the expected roles, their availability (to deal with part time employees), and the expected productivity compared to the ideal senior developer (as a percentage). By calculating the average productivity and multiplying it with the number of story points you’ll end up with the total number of estimated man-hours. It’s only an estimate and both the productivity can be disappointing as well as the estimate in story points may appear to be wrong. But it still gives you an initial estimate that can be used for global planning and budget discussions. Obviously it is important to ensure that you keep on continuously measuring the actual productivity. Wow, now what? By now, it should be clear that a user story is not an independent concept but something that closely resonates with many of the aspects of our work in the software industry. In this multi-part post I have tried to explain a number of those aspects and to clarify the relationship between them. But even though I’ve not touched everything as detailed as possible, I still hope I've managed to convince you about the power and potential of user stories. Last but not least, if you have any questions or comments, please do not hesitate to email me at [email protected] or tweet me at my Twitter ID ddoomen.

I've blogged a few times now about Git (which I pronounce with a hard 'g' a la "get", as it's supposed to be named for Linus Torvalds, a self-described git, but which I've also heard called pronounced with a soft 'g' like "jet"). Either way, I'm finding it way more efficient and less painful than either CVS or SVN combined. So, to continue this series ([1], [2], [3]), here is how (and why) to pull an SVN repo down as a Git repo, but with the omission of old (irrelevant) revisions and branches. Using SVN for SVN repos In days of yore when working with the JBoss Tools and JBoss Developer Studio SVN repos, I would keep a copy of everything in trunk on disk, plus the current active branch (most recent milestone or stable branch maintenance). With all the SVN metadata, this would eat up substantial amounts of disk space but still require network access to pull any old history of files. The two repos were about 2G of space on disk, for each branch. Sure, there's tooling to be able to diff and merge between branches w/o having both branches physically checked out, but nothing beats the ability to place two folders side by side OFFLINE for deep comparisons. So, at times, I would burn as much as 6-8G of disk simply to have a few branches of source for comparison and merging. With my painfullly slow IDE drive, this would grind my machine to a halt, especially when doing any SVN operation or counting files / disk usage. Using Git for SVN repos naively Recently, I started using git-svn to pull the whole JBDS repo into a local Git repo, but it was slow to create and still unwieldy. And the JBoss Tools repo was too large to even create as a Git repo - the operation would run out of memory while processing old revisions of code to play forward. At this point, I was stuck having individual Git repos for each JBoss Tools component (major source folder) in SVN: archives, as, birt, bpel, build, etc. It worked, but replicating it when I needed to create a matching repo-collection for a branch was painful and time-consuming. As well, all the old revision information was eating even more disk than before: jbosstools' trunk as multiple git-svn clones: 6.1G devstudio's trunk as single git-svn clone: 1.3G So, now, instead of a couple Gb per branch, I was at nearly 4x as much disk usage. But at least I could work offline and not deal w/ network-intense activity just to check history or commit a change. Still, far from ideal. Cloning SVN with standard layout & partial history This past week, I discovered two ways to make the git-svn experience at least an order of magnitude better: Standard layout (-s) - this allows your generated Git repo to contain the usual trunk, branches/* and tags/* layout that's present in the source SVN repo. This is a win because it means your repo will contain the branch information so you can easily switch between branches within the same repo on disk. No more remote network access needed! Revision filter (-r) - this allows your generated Git repo to start from a known revision number instead of starting at its birth. Now instead of taking hours to generate, you can get a repo in minutes by excluding irrelevant (ancient) revisions. So, why is this cool? Because now, instead of having 2G of source+metadata to copy when I want to do a local comparison between branches, the size on disk is merely: jbosstools' trunk as single git-svn clone w/ trunk and single branch: 1.3G devstudio's trunk as single git-svn clone w/ trunk and single branch: 0.13G So, not only is the footprint smaller, but the performance is better and I need never do a full clone (or svn checkout) again - instead, I can just copy the existing Git repo, and rebase it to a different branch. Instead of hours, this operation takes seconds (or minutes) and happens without the need for a network connection. Okay, enough blather. Show me the code! Check out the repo, including only the trunk & most recent branch # Figure out the revision number based on when a branch was created, then # from r28571, returns -r28571:HEAD rev=$(svn log --stop-on-copy \ http://svn.jboss.org/repos/jbosstools/branches/jbosstools-3.2.x \ | egrep "r[0-9]+" | tail -1 | sed -e "s#\(r[0-9]\+\).\+#-\1:HEAD#") # now, fetch repo starting from the branch's initial commit git svn clone -s $rev http://svn.jboss.org/repos/jbosstools jbosstools_GIT Now you have a repo which contains trunk & a single branch git branch -a # list local (Git) and remote (SVN) branches * master remotes/jbosstools-3.2.x remotes/trunk Switch to the branch git checkout -b local/jbosstools-3.2.x jbosstools-3.2.x # connect a new local branch to remote one Checking out files: 100% (609/609), done. Switched to a new branch 'local/jbosstools-3.2.x' git svn info # verify now working in branch URL: http://svn.jboss.org/repos/jbosstools/branches/jbosstools-3.2.x Repository Root: http://svn.jboss.org/repos/jbosstools Switch back to trunk git checkout -b local/trunk trunk # connect a new local branch to remote trunk Switched to a new branch 'local/trunk' git svn info # verify now working in branch URL: http://svn.jboss.org/repos/jbosstools/trunk Repository Root: http://svn.jboss.org/repos/jbosstools Rewind your changes, pull updates from SVN repo, apply your changes; won't work if you have local uncommitted changes git svn rebase Fetch updates from SVN repo (ignoring local changes?) git svn fetch Create a new branch (remotely with SVN) svn copy \ http://svn.jboss.org/repos/jbosstools/branches/jbosstools-3.2.x \ http://svn.jboss.org/repos/jbosstools/branches/some-new-branch From http://divby0.blogspot.com/2011/01/howto-partially-clone-svn-repo-to-git.html

Troy Giunipero (pictured, right) is a student at the University of Edinburgh studying toward an MSc in Computer Science. Formerly, he was one of the NetBeans Docs writers based in Prague, Czech Republic, where he spent most of his time writing Java web tutorials. In this interview, Troy introduces you to The NetBeans E-commerce Tutorial. This is a very detailed tutorial describing just about everything you need to know when creating an e-commerce web application in Java. It has received a lot of very positive feedback. Let's find out about the background of this tutorial and what Troy learnt in writing it. Hi Troy! During your time on the NetBeans team, you wrote a very large tutorial describing how to create an e-commerce site. How and why did you start writing it? Well, there’s a short answer and a long answer to this. The short answer is that I was lucky to take part in Sun’s SEED (Sun Engineering Enrichment and Development) program. I wanted to focus on technical aspects, so I based my curriculum on developing an e-commerce application using Java technologies. I documented my efforts and applied them toward deliverables for the IDE’s 6.8 and 6.9 releases, resulting in the 13-part NetBeans E-commerce Tutorial. The long answer is that I had previously been tasked with creating an e-commerce application for my degree project (I was studying toward a BSc in IT and Computing), and ran into loads of trouble trying to integrate the various technologies into a cohesive, functioning application. I was coming from a non-technical background and found there was a steep learning curve involved in web development. My work was fraught with problems which I can now attribute to poor time-management, and a lack of good, practical, hands-on learning resources. So in a way, working on the AffableBean project (this is the project used in the NetBeans E-commerce Tutorial) was a way for me to go back and attempt to do the whole thing right. With the tutorial, I had two goals in mind: one, I wanted to consolidate my understanding of everything by writing about it, and two, I wanted to help others avoid the problems and pitfalls that I’d earlier ran into by designing a piece of documentation that puts everything together. Can you run us through the basic parts and what they provide? Certainly. First I want to point out that there’s a live demo application (http://dot.netbeans.org:8080/AffableBean/) which I managed to get up and running with help from Honza Pirek from the NetBeans Web Team (thanks Honza!): The application is modeled on the well-known MVC architecture: The tutorial refers to the above diagram at various points, and covers a bunch of different concepts and technologies along the way, including: Project design and planning (unit 2) Designing a data model (using MySQL WorkBench) (unit 4) Forward-engineering the data model into a database schema (unit 4) Database connectivity (units 3, 4, 6) Servlet, JSP/EL and JSTL technologies (units 3, 5, 6) EJB 3 and JPA 2 technologies (unit 7), and transactional support (unit 9) Session management (i.e., for the shopping cart mechanism) (unit 8) Form validation and data conversion (unit 9) Multilingual support (unit 10) Security (i.e., using form-based authentication and encrypted communication) (unit 11) Load testing with JMeter (unit 12) Monitoring the application with the IDE’s Profiler (unit 12) Tips for deployment to a production server (units 12, 13) Also, the tutorial aims to provide ample background information on the whole “Java specifications” concept, with an introduction to the Java Community Process, how final releases include reference implementations, and how these relate to the tutorial application using the IDE’s bundled GlassFish server (units 1, 7). Finally, the tutorial is as much about the above concepts and technologies as it is about learning to make best use of the IDE. I really tried to squeeze as much IDE-centric information in there as possible. So for example you’ll find: An introduction to the IDE’s main windows and their functions (unit 3) A section dedicated to editor tips and tricks (unit 5), and abundant usage of keyboard shortcuts in steps throughout the tutorial Use of the debugger (unit 8) Special “tip boxes” that discuss IDE functionality that is sometimes difficult to fit into conventional documentation. For example, there are tips on using the IDE’s Template Manager (unit 5), GUI support for database tables (unit 6), Javadoc support (unit 8), and support for code templates (unit 9). Did you learn any new things yourself while writing it? Yes! Three things immediately come to mind: EJB 3 technology. Initially this was a big hurdle for me. Using EJB 3 effectively seems to be something of an art form. If you know what you’re doing and understand exactly how to use the EntityManager to handle persistence operations on a database, EJB lets you do some amazingly smart things with just a few lines of code. But there seems to be a lack of good free documentation online—especially since EJB 3 is a significant departure from EJB 2. Therefore, almost all of the tutorial’s information on EJB comes from the very excellent book, EJB in Action by Debu Panda and Reza Rahman. Interpreting the NetBeans Profiler. The final hands-on unit, Testing an Profiling, was the most difficult for me to write, primarily because I just wasn’t familiar with the Profiler at all. I spent an unhealthy amount of time just watching the Telemetry graph run against my JMeter test plan, which is only slightly more stimulating than watching water come to boil. That being said, I feel that by just examining the graphs and other windows over time, critical logical associations start to jump out at you after a while. Likewise with JMeter. Hopefully unit 12 was able to capture and relay some of these. How to search online for decent articles and learning materials. The old Sun Technical Articles site was a great resource. Many of the links in the See Also sections at the bottom of tutorial units were found by adding site:java.sun.com/developer/technicalArticles/ to a Google search. Also the official forums (found at forums.sun.com) became a good place for questions I couldn’t find ready answers to. I had both the Java EE 5 and 6 Tutorials bookmarked. And Marty Hall’s Core Servlets and JavaServer Pages became an invaluable resource for the first half of the tutorial. What are your personal favorite features of the technologies discussed in the tutorial? I particularly liked learning about session management—using the HttpSession object to carry user-specific data between requests, and working with JSP’s corresponding implicit objects in the front-end pages. Session management is a defining aspect for e-commerce applications, as they need to provide some sort of shopping cart mechanism... ...and so the Managing Sessions unit (unit 8) was a key chapter in the tutorial. It’s extremely useful to be able to suspend the debugger on a portion of code that includes session-scoped variables, then hover the mouse over a given variable in the editor to determine its current value. I used the debugger continuously during this phase, and so I went so far as to incorporate use of the debugger throughout the Managing Sessions unit. What kind of background does someone starting the tutorial need to have? Someone can come to the tutorial with little or absolutely no experience using NetBeans. I’ve tried to be particularly careful in providing clear and easy-to-follow instructions in this respect. But one would be best off having some background or knowledge in basic web technologies, and at least some exposure to relational databases. With this foundation, I think that the topics covered in the second half of the tutorial, like applying entity classes and session beans, language support and security, won’t seem too daunting. I’ve noticed that the vast majority of feedback that comes in relates to the first half of the tutorial, and I sometimes get the impression that people feel they need to follow the tutorial units consecutively. Not so. The units are 90% modular. In other words, if somebody just wants to run through the security unit (unit 11), they can do so by downloading the associated project snapshot, follow the setup instructions, and then just follow along without needing to even look at other parts of the tutorial. What will they be able to do at the end of it? Naturally, anybody who completes individual tutorial units will be able to apply the concepts and technologies to their own work. But anyone who completes the tutorial in its entirety will gain an insight into the development process as a whole, and I think will also get a certain confidence that comes with knowing how “all the pieces fit together”—from gathering customer requirements all the way to deployment of the completed app to a production server. They’ll also have gained a solid familiarity with the NetBeans IDE, and be in a good position to explore popular Java web frameworks that work on top of servlet technology or impose an MVC architecture on their own, such as JSF, Spring, Struts, or Wicket. Do you see any problems in the technologies discussed and what would be your suggestions for enhancements? Well there’s one thing that comes to mind. When I started working on this project, I was studying the Duke’s BookStore example from the Java EE 5 Tutorial. A wonderful example that demonstrates how to progressively implement the same application using various technologies and combinations thereof. So for example you start out with an all-servlet implementation, then move on to a JSP/servlet version. Then there’s a JSTL implementation and ultimately, a version using JavaServer Faces. It’s great learning material, but also terrifically outdated. Right around this time, Sun was gearing up for the big Java EE 6 release (Dec. 2009), and I was also trying to learn about the new upcoming technologies, namely CDI, JSF 2, and EJB 3, for my regular NetBeans documentation work. I was getting the definite sense that JSP and JSTL were slowly being pushed aside—in the case of JavaServer Faces, Facelets templating was the new page authoring technology. So really, the E-commerce Tutorial application has become a sort of EE 5/EE 6 hybrid by combining JSP/JSTL with EJB 3 and JPA 2. Now the problem I see from the perspective of a student trying to learn this stuff from scratch, is that the leap from basic servlet technology to a full-blown JSF/EJB/JPA solution is tremendous, and cannot readily be taught through a single tutorial. Naturally, others may disagree with me here. I’m not sure if there’s a solution other than to compensate by producing a lot of quality learning material that covers lots of different use-cases. I’d suggest that the E-commerce Tutorial puts one in a very advantageous position to begin learning about Java-based frameworks, such as GWT, Spring, and JSF, which is a natural course of action for people looking to get a job with this knowledge. Planning any more parts to the tutorial or a new one? No more parts. The E-commerce Tutorial is done. Upon committing the final installments and changes last November, I rejoiced. However, I’m still actively responding to feedback [the ‘Send Us Your Feedback’ links at the bottom of tutorials] and plan to maintain it indefinitely, so if anyone spots any typos, has questions or comments, recommendations for improvement, etc., please write in! :-)

Both approaches have some advantages and disadvantages and so it’s a question of the likelihood of particular problems and business requirements and priorities.

JMX is a great way to check or change state variables or invoke a method in a (remote) running application via a management GUI such as JConsole. And Spring makes it trivial to expose any POJO as a JMX MBean with only little configuration in a few minutes. The Spring JMX documentation is very good, however there are few points that I struggled with for a while and would therefore like to record here the right solutions. I needed to monitor a command-line java application using Spring 2.5 on IBM JVM 1.4 1.5 running on a server with a jconsole on Sun JVM 1.6 as the JMX client on my PC. All the XML snippets are from a Spring application-context.xml. If you haven’t used Spring before, read a tutorial on its configuration and dependency injection. Turning a POJO into an MBean JMX makes it possible to expose getters, setters and operations taking primitive or complex data types as parameters (though types other then few special ones require the client to have the classes). You tell Spring to expose a POJO as an MBean as follows: First you declare an instance of the POJO class – the myMBean (for other reasons I’ve an old-fashioned singleton and use JobPerformanceStats.instance() to access the bean). Next you declare an MBeanExporter with lazy-init=”false” and tell it about your bean. (There are also other ways to do it, including auto-discovery.) The bean will be then visible under its key, i.e. “bean:name=MyMBeanName”, which JConsole displays as “MyMBeanName”. Notice that the MBeanExporter only works under JVM 1.5+ because it uses the new java.lang.management package. Under 1.4 Spring would fail with java.lang.NoClassDefFoundError: javax/management/MBeanServerFactory at org.springframework.jmx.support.MBeanServerFactoryBean.createMBeanServer By default it will expose all public methods and attributes. You can change that in various ways, for example with the help of an interface. If you are not running in a container that already provides an MBean server, which is my case here, you must tell Spring to start one: Enabling remote access To make the MBean accessible from another machine, you must expose it to the world by declaring a ConnectorServerFactoryBean configured with an appropriate communication mechanism. Remote access over JMXMP By default the ConnectorServerFactoryBean exposes MBeans over JMXMP with the address service:jmx:jmxmp://localhost:9875 : However this protocol isn’t supported out of the box and you must include jmxremote_optional.jar, a part of OpenDMK, on the classpath of both the MBean application and the jconsole client to avoid the exception org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘org.springframework.jmx.support.ConnectorServerFactoryBean#0′ defined in class path resource [application-context.xml]: Invocation of init method failed; nested exception is java.net.MalformedURLException: Unsupported protocol: jmxmp Remote access over RMI Alternatively you can expose the MBeans over RMI, which has no additional dependencies: However there are also some catches you must avoid: You must start an RMI registry so that the connector can register the MBean there; it won’t start one for you You must make sure that the registry is started before the connector tries to use either by declaring it before the connector or by making this dependency explicit with the depends-on attribute If you don’t set it up correctly you will get an exception like org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘org.springframework.jmx.support.ConnectorServerFactoryBean#0′ defined in class path resource [application-context.xml]: Invocation of init method failed; nested exception is java.io.IOException: Cannot bind to URL [rmi://localhost:10099/jmxrmi]: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is: java.net.ConnectException: Connection refused: connect]. Local MBean server accessed over an SSH tunnel For increased security you may want to prefer not to expose your MBeans to remote access by making them accessible only from the local machine (127.0.0.1) and use na SSH tunnel so that a remote JConsole can access them as a local application. This is certainly possible but may be difficult because normally JMX goes over RMI, which uses two ports: one for the RMI Registry and another one for the actual service (MBean server here), which is usually chosen randomly at the runtime, and you’d need to tunnel both. Fortunately, Spring makes it possible to configure both the ports: Replace STUBPORT and REGISTRYPORT with suitable numbers and tunnel those two. Notice that the REGISTRYPORT number is same in the connector’s serviceUrl and in the RMI registry’s port attribute. WARNING: The configuration above actually doesn’t prevent direct access from a remote application. To really force the RMI registry to only listen for connection from the local host we would likely need to set – under Sun JVM without Spring – the system property com.sun.management.jmxremote. Additionally, to force the registry to use the IP 120.0.0.1, we’d need to set java.rmi.server.hostname=localhost (applies to Spring too). See this discussion about forcing local access. I’m not sure how to achieve the same result with Spring while still preserving the ability to specify both the RMI ports. Check also the JavaDoc for Spring’s RmiServiceExporter . Related posts and docs: Tunneling Debug and JMX for Alfresco (A. uses Spring)- see the second section, SSH Tunneling for JMX A custom tunneling RMI agent – uses a configured port instead of a random one Monitoring ActiveMQ Using JMX Over SSH JMX 1.2 specification and JMX 1.0 Remote API specification; from the JMX spec.: “The MBean server relies on protocol adaptors and connectors to make the agent accessible from management applications outside the agent’s JVM.” On the other hand, the Oracle JMX page reads that if you set com.sun.management.jmxremote (as opposed to …jmxremote.port), you make it possible “to monitor a local Java platform, that is, a JVM running on the same machine” – thus not necessarily from the same JVM. Connecting with JConsole Fire up JConsole and type the appropriate remote address, for example service:jmx:rmi:///jndi/rmi://your.server.com:10099/myconnector, if connecting to an application on the remote machine your.server.com accessible via RMI. Regarding the connection URL, if you have a a connector with the serviceUrl of "service:jmx:rmi://myhost:9999/jndi/rmi://localhost:10099/myconnector" then, from a client, you can use either service:jmx:rmi://myhost:9999/jndi/rmi://your.server.com:10099/myconnector or simply service:jmx:rmi:///jndi/rmi://your.server.com:10099/myconnector because, according to the JMX 1.2 Remote API specification (page 90): ... the hostname and port number # (myhost:9999 in the examples) are not used by the client and, if # present, are essentially comments. The connector server address # is actually stored in the serialized stub (/stub/ form) or in the # directory entry (/jndi/ form). IBM JVM, JConsole and JMX configuration The IBM JVM 5 SDK guide indicates that the IBM SDK also contains JConsole and recognizes the same JMX-related system properties, namely com.sun.management.jmxremote.* (though “com.sun.management.jmxremote” itself isn’t mentioned). Notice that the IBM JConsole is a bit different, for instance it’s missing the Local tab, which is replaced by specifying the command-line option connection=localhost (search the SDK guide for “JConsole monitoring tool Local tab”). Further improvements JVM 1.5: Exposing the MemoryMXBean Since Java 5.0 there is a couple of useful platform MBeans that provide information about the JVM, including also the java.lang.management.MemoryMXBean, that let you see the heap usage, invoke GC etc. You can make it available to JConsole and other JMX agents as follows (though there must be an easier way): Update: There indeed seems to be a better way of exposing the platform MBeans directly by replacing the Spring’s MBeanServerFactoryBean with java.lang.management.ManagementFactory using its factory-method getPlatformMBeanServer. Of course this requires JVM 1.5+. Improving security with password authentication Access to your MBeans over RMI may be protected with a password. According to a discussion, authentication is configured on the server connector: The passwd and access files follow the templates that can be found in the JDK/jre/lib/management folder. Summary Exposing a POJO as a MBean with Spring is easy, just don’t forget to start an MBean server and a connector. For JMXMP, include the jmxmp impl. jar on the classpath and for RMI make sure to start a RMI registry before the connector. From http://theholyjava.wordpress.com/2010/09/16/exposing-a-pojo-as-a-jmx-mbean-easily-with-spring/

This post will walk through how to resize an ExtJS Panel, Grid, Component on Window Resize without using Ext.Viewport. Problem: You have a legacy page and you want to change an html grid for an ExtJS DataGrid, because it has so many cool features. Or you have a page with some design and you are going to use only one ExtJS Component. In both cases, you also want to render your ExtJS Component to a specific DIV. Also, you want you component to be resized in case you resize the browser window. How can you do that if resize a single component in an HTML page it is not the default behavior of an ExtJS Component (except if you use Ext.Viewport)? Solution: Condor (from ExtJS Community Support Team) developed a plugin that can do that for you. I had to spend some time to understand how the plugin works, and I finally got it working as I wanted. Well, I recommend you to spend some time reading this thread: http://www.sencha.com/forum/showthread.php?28318 (if you have any issues or questions, please publish it on the thread, so other members can give you the support you need). Requirements to make the plugin work: Your have to apply the following style to the DIV (the width is up to you, the other styles are mandatory, otherwise it will not work): If you have any border around your ExtJS component, you have to set a HEIGHT. And you will also have to set a height to your ExtJS component. In this case, autoHeight will not work. If you DO NOT have any border or other design on the ExtJS component side, you do not need to set height and you can use autoHeight. In my case, I put a border on the external DIV, so I have to set Height: HTML code (all DIVs): And you need to add the plugin to the component (In this case, I’m using an ExtJS DataGrid): var grid = new Ext.grid.GridPanel({ store: store, columns: [ {header: 'Company', width: 160, sortable: true, dataIndex: 'company'}, {header: 'Price', width: 75, sortable: true, renderer: 'usMoney', dataIndex: 'price'}, {header: 'Change', width: 75, sortable: true, renderer: change, dataIndex: 'change'}, {header: '% Change', width: 75, sortable: true, renderer: pctChange, dataIndex: 'pctChange'}, {header: 'Last Updated', width: 85, sortable: true, renderer: Ext.util.Format.dateRenderer('m/d/Y'), dataIndex: 'lastChange'} ], stripeRows: true, autoExpandColumn: 'company', height: 490, autoWidth:true, title: 'Array Grid', // config options for stateful behavior stateful: true, stateId: 'grid' ,viewConfig:{forceFit:true} ,renderTo: 'reportTabContent' // render the grid to the specified div in the page ,plugins: [new Ext.ux.FitToParent("reportTabContent")] }); And done! Now you can resize the browser and the component will resize itself! I tested it on Firefox, Chrome and IE6. You can download my sample project from my GitHub: http://github.com/loiane/extjs-fit-to-parent PS.: If you want to use the full browser window, use a Viewport. Happy coding!

Working together day after day is an intense human experience, with all the glories and warts that emerge from constant interaction between these astonishing, disappointing, challenging, infuriating, magnificent, normal human beings we call team members. On an agile team, especially, we see this, and in our pursuit of excellence we know that conflicts arise and that we can expect both harmony and disharmony. Navigating conflict is our new mind-set, in which we help teams move from conflict to constructive disagreement as a catapult to high performance. Editor's Note: This article has been excerpted from Lyssa Adkin's book, "Coaching Agile Teams" (Addison-Wesley Signature Series (Cohn), July 2010). The Agile Coach’s Role in Conflict Coaching teams to navigate conflict may feel unfamiliar or uncomfortable to you. It did for me, even though books, articles, and studies on the subject abound. As a plan-driven project manager, I didn’t have to “go there” in the face of conflict very often because team members joined the team and left the team as we moved from phase to phase. If my feeble attempts to resolve a conflict failed, no big loss. Sooner or later, the team members in conflict would move on to other projects. With agile, however, team members stay together throughout the project. They do not move on, nor does the conflict. Given this, the agile coach faces conflict squarely, skillfully determines the severity of it, mindfully decides whether to intervene and how, generously teaches teams to navigate it, and courageously refuses to settle for a team that shrinks from greatness by avoiding it. As their coach, you help teams navigate conflict. You show them a method. You can’t give them a full-color, waterproof chart that marks the shoals and hazards. You can give them something more precious, more powerful. You can give them a guide, a framework, so that they create their own charts, whenever they need to do so. Five Levels of Conflict An agile team humming along in the rhythm of steady momentum will display conflict all the time—minor quips at one another, rolling eyeballs, heavy sighs, emotional voices, stony silences, tension in the air. You’ll witness dry wit, teasing, “just joking” comments, or just-short-of-snide remarks, all in the range of normal for an agile team. These behaviors signal normalcy for any group of people who spend considerable time together and who create a shared history. It happens in neighborhoods, community coffeehouses, churches, and agile teams—especially agile teams—where team members sit arm’s distance apart for hours on end every day while they create products together, all the while responding to the built-in pressure of the timeboxed sprint. Conflict, ever present, can be normal or destructive, and the two can be hard to tell apart. An author of many books on conflict, Speed Leas offers us agile coaches a framework we can use to determine the seriousness of the conflict (1985). This model, well suited to agile teams, looks at conflict in a deeply human and humane way. As depicted in Figure 9.1, it forms an escalation path of conflict from “Level 1: Problem to Solve” to “Level 5: World War.” (Click for larger image) Figure 9.1 Five levels of conflict Level 1: Problem to Solve We all know what conflict at level 1 feels like. Everyday frustrations and aggravations make up this level, and we experience conflicts as they rise and fall and come and go. At this level, people have different opinions, misunderstanding may have happened, conflicting goals or values may exist, and team members likely feel anxious about the conflict in the air. When in level 1, the team remains focused on determining what’s awry and how to fix it. Information flows freely, and collaboration is alive. Team members use words that are clear, specific, and factual. The language abides in the here and now, not in talking about the past. Team members check in with one another if they think a miscommunication has just happened. You will probably notice that team members seem optimistic, moving through the conflict. It’s not comfortable, but it’s not emotionally charged, either. Think of level 1 as the level of constructive disagreement that characterizes high-performing teams. Level 2: Disagreement At level 2, self-protection becomes as important as solving the problem. Team members distance themselves from one another to ensure they come out OK in the end or to establish a position for compromise they assume will come. They may talk offline with other team members to test strategies or seek advice and support. At this level, good-natured joking moves toward the half-joking barb. Nastiness gets a sugarcoating but still comes across as bitter. Yet, people aren’t hostile, just wary. Their language reflects this as their words move from the specific to the general. Fortifying their walls, they don’t share all they know about the issues. Facts play second fiddle to interpretations and create confusion about what’s really happening. Level 3: Contest At level 3, the aim is to win. A compounding effect occurs as prior conflicts and problems remain unresolved. Often, multiple issues cluster into larger issues or create a “cause.” Factions emerge in this fertile ground from which misunderstandings and power politics arise. In an agile team, this may happen subtly, because a hallmark of working agile is the feeling that we are all in this together. But it does happen. People begin to align themselves with one side or the other. Emotions become tools used to “win” supporters for one’s position. Problems and people become synonymous, opening people up to attack. As team members pay attention to building their cases, their language becomes dis-torted. They make overgeneralizations: “He always forgets to check in his code” or “You never listen to what I have to say.” They talk about the other side in presumptions: “I know what they think, but they are ignoring the real issue.” Views of themselves as benevolent and others as tarnished become magnified: “I am always the one to compromise for the good of the team” or “I have everyone’s best interest at heart” or “They are intentionally ignoring what the customer is really saying.” Discussion becomes either/or and blaming flourishes. In this combative environment, talk of peace may meet resistance. People may not be ready to move beyond blaming. Level 4: Crusade At level 4, resolving the situation isn’t good enough. Team members believe the people on the ”other side” of the issues will not change. They may believe the only option is to remove the others from the team or get removed from the team themselves. Factions become entrenched and can even solidify into a pseudo-organizational structure within the team. Identifying with a faction can overshadow identifying with the team as a whole so the team’s identity gets trounced. People and positions are seen as one, opening up people to attack for their affiliations rather than their ideas. These attacks come in the form of language rife with ideology and principles, which becomes the focus of conversation, rather than specific issues and facts. The overall attitude is righteous and punitive. Level 5: World War “Destroy!” rings out the battle cry at level 5. It’s not enough that one wins; others must lose. “We must make sure this horrible situation does not happen again!” Only one option at level 5 exists: to separate the combatants (aka team members) so that they don’t hurt one another. No constructive outcome can be had. What Should You Do About It? The goal of navigating conflict is to de-escalate. Knock it down a notch or two. As the agile coach, the first and most important question to answer is “Do I have to respond?” First, Do Nothing Agile teams—even new ones and even broken ones—can often navigate conflict by themselves, even conflict up into the level 3 range. So, sit back for a while and witness their moves. See whether they make progress. Even if it’s not perfect or the “complete” job you could do for them, if team members navigate the conflict well enough, leave them alone. To help you live with the uncomfortable feeling of watching a team’s bumbling attempts to deal with conflict, remember these words from Chris Corrigan in The Tao of Holding Space: “Everything you do for the group is one less thing they know they can do for themselves” (Corrigan 2006). The team’s bumbling is better than your perfect plan. Remember the goal of supporting the team’s self-organization (and reorganization). Your discomfort is a small price to pay. But what if you’ve decided to intervene? If you feel you have observed long enough (which should feel like a really long time) and decided to intervene, there are a few response modes you can employ: analyze and respond, use structures, and reveal. These come in order from least to most powerful for the goal of fostering self-organization. Analyze and Respond This may be the most comfortable response mode an agile coach can use because it feels familiar and at least somewhat analytical. To use analyze and respond, the agile coach considers these questions (Keip 1997): What is the level of conflict? What are the issues? How would I respond as side A? How would I respond as side B? What resolution options are open? What should I do (if anything)? When using the analyze-and-respond mode, remember that no one has the whole story. Each person’s perspective is valid and needed. If there are ten team members, you can bet there are at least ten perspectives, each of which is true in the eye of the beholder. What is your knee-jerk reaction when conflict arises? Agile coaches must be able to name that reaction and consciously choose it or reject it in service to the team. See Chapter 3, “Master Yourself,” for ways to keep yourself solidly rooted when unexpected things happen with teams. Things like conflict. Table 9.2 provides a map of successful response modes at each level. Look to this to help answer the question “What resolution options are open?” when addressing conflict in the whole-team setting (Keip 2006). Table 9.2 Conflict navigation response modes at each level When the team de-escalates, they get more options for dealing with the conflict as the tools from the next level down become available to them. The analyze-and-respond mode for navigating conflict may feel comfort-able to you—an easy shoe to slip on. If that’s the case and if it seems the best choice for your current level of skill and confidence, use it. However, you should know that it is the weakest response mode for building high-performance teams because it puts the coach in the driver’s seat. It also relies completely on analytical thinking, which is just one small way to think about conflict. So, as you feel ready, try the next two response modes.