Re-Engineering In Agile Development Can Just Be Refactoring

If you talk to a programmer, almost all software sucks, even the programs they wrote a few months ago. This is typically due to changing technology, increased knowledge or even a market shift. The software you wrote may have been a good idea when you started, but does it serve the correct purpose now? Even when there is a question of real purpose, the software gets installed into a production environment, because nobody wants to be involved in a project that did not deploy.

Now that the system is in production what do you do? Somebody typically maintains the system, correcting defects and adding features. Slowly, that system continues to grow. However, the programmers and managers realize that the system does not really do what they want it to, or maybe there is a ton of technical debt that should be cleaned up. In many cases, people call for a complete rewrite of the system. If you are an agile development organization, this typically brings additional concerns:

Re-engineering is highly problematic from an Agile point of view. We are not producing functionality. We are not working from user facing stories. There may be business value to what we are doing (such as improving the performance of our application) but it is not directly measurable business value in most cases.

Re-engineering is also too often driven by technical concerns within the team (or its management). We didn’t build it as well as we wish we had. We have learned new things now, and we want to incorporate them.

Generally, wholesale re-engineering of a system is a bad idea, because the system still serves a purpose and its’ functionality is a moving target. If the system does not serve a purpose, then what is the point of rewriting it? So, if we follow the assumption that parts of the system were a good idea and scrapping the system is a bad idea, how do you modify the system? This is highly dependent upon what you are trying to change, and how agile your organization really is.  For our examples, let’s assume the organization is used to Scrum, and uses it in all software projects. How do we deal with the desire to re-engineer a system and resolving technical debt?

High Defect Rates

One way that a technical debt heavy application can present itself is when there is a high defect rate. This may not be a bad thing if there is a desire to rewrite the application. You can use the defect list as the list of user stories, and plan a few sprints to clean up the defects. However, the key is to include some time to analyze the areas or components that have the highest defect density. These areas will be the focus of each sprint. Your defects can become additional unit tests, so that you have your baseline tests as well as proof that the new code should also handle the defect situations. Once you have this safety net of unit tests, you can freely re-engineer and refactor these troublesome areas. If the application was “very buggy”, this type of work can quickly stabilize a system without taking on the pain of trying to rewrite the whole application and introducing different issues.

Replacing Old Technology

Let me start this by saying you should not replace old technology just because you think it is old or because there is something newer and cooler. You need a valid business reason, or management will not agree to the work. One good reason to replace old technology is when support for it is being ended (or planned), and the new version has some features that your application could take advantage of. In particular, you really want to attack this problem when there is a feature to add or there are some defects that need fixing. Hopefully, you can still use the existing unit tests, but if not, the unit tests will need to be modified (or rewritten) to work with the new technology. This is one of the more dangerous tasks because it is such a significant change. Typically, the code will be changing enough for this work to be considered a significant release, and will require all the trappings of a typical longer project. This also requires a significant amount of analysis in order to determine the system impact and the proper user stories. The risk in this type of project is proportional to the size of the technology being replaced. If you are replacing a library that is used for financial calculations, you may be able to organize the sprints in a similar manner to the defect correction example. However, if you are moving from Struts 1.1 to Spring MVC 3.0, you have a significant amount of risk and analysis to do, so plan accordingly.

New Feature Does Not Fit The Design

Sometimes, design decisions made at the beginning of a project may not have taken into consideration some requirement that will appear 6 months later. Because of that problem, designs will need to change over time. The problem here is that sometimes a new feature cannot be implemented with the current design. At one time I worked on an internal content management system. The CMS system used the concept of widgets to define prepackaged html tags with supporting javascript. This was very similar to JSP custom tags, but the original design of the system was completed several years ago, so the widgets were created, parsed and manipulated in the Java code. To add a new widget, which had only happened once after the initial design, you needed to modify the code in a few places as well as creating the new widget classes. Now, a new requirement has come along where a user of the system must be able to define their own widget with html and javascript. In this type of case, a new design was needed as the widget definitions moved from static definition to dynamic definition. As in the previous examples, the unit tests are very important as they provide a minimal set of requirements the new design must satisfy.

As in the old technology replacement example, this work will have risk proportional to the size of the components being replaced. In the CMS example, the design change of a core feature carried significant risk. One benefit was that the widget conversion allowed stories and sprints to be planned nicely. First, the basic dynamic widget design could be completed, and then future sprints would deal with the conversion of each static widget. You could actually run the two widget systems in parallel until the last static widget is converted. Once that is complete, you can freely delete the old static widget classes and marvel at your new implementation.

The key in all of these examples is finding a way to break each task into user stories so that you can plan your sprints. This allows you to get the project on a familiar path that even management can understand.

From http://regulargeek.com/2010/07/20/re-engineering-in-agile-development-can-just-be-refactoring