A Transformation Story: From Waterfall to Agile Software Development
A Transformation Story: From Waterfall to Agile Software Development
Becoming agile is time consuming, resource-consuming and risky. Learn about the transformation from waterfall to agile and why it matters.
Join the DZone community and get the full member experience.Join For Free
There exist several software development life-cycle (SDLC) models giving different trade-offs between implementation complexity and speed, scalability, software quality, cost, and adaptability. Addressing controllability and predictability in large software products, the waterfall model was introduced . Similar to the evolution of societies using bureaucratic models, software development’s evolution has been affected by the bureaucratic nature of the waterfall model since the 60’s.
In a waterfall SDLC, requirements gathering, analysis and planning are performed as an initial phase. Architectural design is then followed by software development and testing whilst deployment is the last phase. Large planning and documentation efforts are usually necessary upfront, in order that all deliverables achieve the expected quality at predefined milestones. Requirements are key to the success of waterfall software development. Clear requirements at an early stage should be available with little or no changes at a late stage. The problem is that there are customers that realize what software products suit their needs only after using several draft versions, making clear requirements at an early stage impossible. Even if the requirements are clearly defined, a customer’s major change request at the release phase may result in redesigning the software product from the beginning.
In an attempt for rapid response to changes, flexibility, and fast time-to-market, companies strive for agile-like SDLC. Whilst agile SDLC principles are known since the agile manifesto  one of the hard truths that the software industry faces is that it is not easy to become agile-like. It could be time-consuming, resource-consuming, and risky. What is also interesting is that the path to take towards agility may vary largely from one company to another. There are however basic guidelines to follow and maybe a framework to guide thinking towards the right direction.
Scaling agile is not straightforward since processes may not scale. Principles like continuous improvement and transparency may naturally evolve from small software groups to large enterprises but this may not be true for processes. Processes depend on the context and changing context may necessitate changing processes.
Going beyond scale, a newly formed startup that attempts to employ agile principles and practices is a completely different story from the waterfall to agile transformation of a software house of thousands of people who have been working for decades using specific processes and practices.
Siemens Enterprise Communications rebranded as Unify GmbH. An enterprise communications vendor transformed into software and solutions vendor with SaaS unified communications applications as a flagship. Waterfall software production changed to agile-like. This article does not propose the “right way” for transforming from waterfall to agile SDLC, and I don’t think that there exists a “right-way” for all companies and organizations. However, there could be a list of things to try and things to avoid. Parameters to be considered for transforming, factors that helped, difficulties and lessons learned are presented together with a framework that could guide towards basic decisions. As a former employee of Unify GmbH, I attest that ideas and opinions presented herein are not those of Unify GmbH. They are solely the critical analysis based on my years of experience.
The big picture
Our transformation consisted of several steps. Research and development departments for Siemens Enterprise existed in several locations across the globe. They consisted of project/product management departments, architectural departments, software development departments, testing departments, DevOps departments, sales, etc. Scrum teams were formed by using members from development, testing, DevOps and the architectural departments. Scum masters were either new hires or previous test/development managers. The waterfall SDLC was broken down into multiple waterfalls. When the multiple waterfalls model was mature enough, sprint development based on scrum ceremonies occurred. More scrum teams were created in time, leading to the consolidation of many departments. The company now has products with agile-like SDLC whilst there are products having an SDLC with distinctive waterfall characteristics.
The mini-waterfalls stage
Our waterfall-to-agile transformation had an intermediate stage where the waterfall was decomposed into smaller waterfalls (iterations) as shown in Figure 1. If this stage was too short then our transformation may have been jeopardized since the company may have not been ready to transform. If it was too long then projects would have been delayed, demoralizing the employees (can we really be agile?). Characterized by synergies throughout the SDLC, this is when continuous integration (CI), test-driven development (TDD), and test-automation were digested. When the mini-waterfalls stabilized (e.g., satisfying quality, budget, and time constraints), self-organized scrum teams replaced development departments and testing departments.
Multi-waterfall stabilization was an issue. A company that always planned for a single waterfall may find it difficult to plan for multiple waterfalls. Iterative planning, however, may be a useful achievement towards the transformation to agile. If testing is squeezed, for example, then there may be iterations where testing is insufficient or it is carried over to the next iteration. The first option may limit the quality of the product while the second may shift the whole project at a later stage. It is difficult to accommodate large differences in SDLC effort between iterations.
There may be a largely uneven distribution of deliverables per iteration, leading to largely varying development/testing efforts. CI and TDD alleviated this problem since automated regression testing helped to increase testing speed and efficiency.
FIGURE 1. Multiple waterfalls (iterations)
We found TDD and CI to be very helpful. Junior software developers found it difficult with TDD and training was necessary. Testers with a “testers don’t write code” mentality also found it difficult to fit in a scrum team. Developers with a “developers don’t test” mentality also needed training. Using Mike Cohn’s three-level automation pyramid  we constructed our automation strategy based on two realizations of our systems:
1. Frameworks of services for applications
2. Integrated components that constitute complete systems
The first realization leads to Mike Cohn’s pyramid as shown in Figure 2 (a), whilst the second replaces the middle (service) layer with integration layers as shown in Figure 2 (b). Both realizations were used to decide what to automate, together with past experience on the systems developed and risk analysis. When solutions integrate a number of software and hardware components, an incremental integration strategy may be appropriate. Smaller components (i.e., units of code) can be automated at a lower level and larger components (even complete systems) may be automated at higher levels. The number of integration levels in the pyramid may depend on the costs and benefits per level. If it takes a heavy effort to write and maintain an automated test-suite whilst it takes too long to run it, just to catch a few bugs of low impact, then manual testing should be performed at this level instead. The bug-clustering effect was evident in many cases, where most of the bugs were introduced by small parts (clusters) of the code. Automated tests were targeted towards those clusters.
Automated unit-testing produced faster results than higher-layer automation testing, making it crucial for the transformation. It allowed for easy tracking of the buggy parts of the code and coupled with performance testing it gave insights about the scalability of the architectural design. Usually, fixing a bug in the code needs to be compensated with code restructuring, otherwise code maintainability may deteriorate. Developers under pressure may skip such code restructuring in order to focus on writing more code. This generates technical debt  and we used unit-level automation testing as a countermeasure.
Running automated regression test suites at the UI level is time-consuming and requires a considerable effort for generating and maintaining. Such regression suites should only run the absolutely necessary number of tests. The rest of the UI tests can be manually covered whilst there are cases where service-layer automation testing may also be used to cover automated UI testing.
The service-level of automation testing covers more code than unit testing, making it slower since a test may access databases or other components. Functionality is tested directly from the production code bypassing the UI, making it faster than UI-level automation.
Although waterfall companies may enjoy automation at different levels and to a different extent, we found it to be an important technical factor that helped towards the transformation. Automation at all layers is important but during the transformation, fast deployment and encouraging automation results are vital. There is a plethora of automation tools with different capabilities using various programming languages. Although a comparison between different tools is beyond the scope of this article, we've chosen such tools based on three parameters: Fast deployment (easy to understand and use), best fit for our technological needs and maintainability. Moving from manual to automated engineering is a change and if this change gives encouraging results soon it will be a reason to keep changing for the transformation.
FIGURE 2. Different realizations of a test automation pyramid(a) A framework of services for apps
FIGURE 2. Different realizations of a test automation pyramid
(b) Integrated components making up systems
Using a Testing Group as a Basis for Synergies
Development teams and test teams can be independent or integrated. Test independence avoids testing bias and it is often effective at finding defects and failures. Integration avoids unnecessary synchronization meetings, it may provide a common ground for synergies and it may enforce a common-goal sharing attitude.
When the software development organization is independent from testing an “us against them” attitude is expected. To some extent this is desirable. There is usually a “civil war” between developers and testers otherwise one may start wondering if both departments are doing their job correctly. This “civil war” should not prohibit brainstorming. The company’s clear vision and strategy must be shared by all teams across the waterfall production phases. Although these objectives may seem conflicting, a balance between “civil wars” and shared common goals was another important factor for our successful transformation.
Testing should not be exercised as a separate activity that simply follows development. Quality assurance is not only about reactively finding defects and reporting. It is an ethos that should be embedded in all teams throughout the SDLC and this involves proactively preventing defects and employing software risk management. A defect in the requirements may lead to multiple bugs found during system testing which could result in major redesigns and frustration. Fixing the defect at an early stage will save time and money. Preventing requires testers to be involved in the requirements documenting phase, they must be engaged with customers, designing and development. Static testing and static analysis could be implemented by testers in the requirements, design and development phases as shown in Figure 3. Actively participating in code reviews and unit or component testing generates synergies. This requires highly-skilled testers that have a software development background and a positive mentality towards learning whatever is necessary. Knowledge about the code and the solution’s logic will help in test-cases design, exploratory testing, finding bugs or even preventing them. Customer engagement of testers should occur at both requirements documenting and release phases. This may bring the customer’s skepticism into quality assurance, another skill that we found to be critical for a waterfall-to-agile transformation. Multi-dimensional quality assurance may help to build testers with the necessary soft and technical skills for joining a scrum team.
FIGURE 3. Software testing for synergies to help the transformation
Cross-functional teams increased synergies in our case since their everyday work involved collaboration with diverse stakeholders. Team members came from multiple backgrounds and were specialized in different areas. Such teams enjoyed diverse competencies and whenever overlapping skillsets existed together with knowledge sharing, they were the backbone of future scrum teams.
Lean Processes and Mentality
Lean principles include process optimization by eliminating waste, fast delivery, people-centric working environments, high learning standards, and continuous improvement . Although we were a large hierarchical company with a focus on following rules, there was always room for people-centric mentality. We’ve found that the lean mentality was another prerequisite for a successful transformation in our case.
Metrics may have both a detrimental and a very positive effect. They are used to measure progress and they should always motivate (even if they are not good). Motivation is crucial especially during the transformation. Metrics must be interpreted at a group level and not at an individual level.
Interpreting metrics may be tricky in some cases. An increased number of defects reported by testers may have two interpretations: Either the testers are doing a better job or the developers are doing worse. In both cases, this should be handled in a way that increases productivity and does not de-motivate.
Within a multinational company, different functional groups or product groups or groups based in different geographical areas may develop their own metrics. When metrics are developed in isolation from or even in competition with other departments, silos may be reinforced. Competition is always welcome as long as it enforces a unified vision and common goals. Silos may work against such goals and they should be dealt with to favor a successful transformation.
We had multiple levels of documentation: From requirements and specification documents, to design, development, testing, installation, upgrade, and maintenance documents. There were process documentation and documentation for internal and/or external purposes. We're not fond of excessive documentation but it is often difficult to manage too much information orally, via emails or by using reminders on the walls. On the other hand, excessive documentation is expensive to maintain, it may be the source of delays or the reason to lose customers (e.g., nobody likes to read long documents for a smartphone app). Lightweight documentation was useful in our case once we’ve found the balance between completeness and accuracy, readability, and maintainability.
It is unlikely that a waterfall company will have all the necessary skills to become agile. Talented agile professionals will probably have to be hired. While technical skills are always important, soft skills (e.g., passion for knowledge, team-spirit) are also important. In an environment that demands self-organization and independence, focus should be given in hiring people with the appropriate soft-skills. In waterfall companies, however, soft skills tend to be nice to have and the focus is on technical skills (especially in low-ranking positions). Agility necessitates independence and self-organization to be promoted from nice to have to essential skills. We’ve found that hiring people with a mentality of learning and growth, as well as educating ourselves towards such issues was a cornerstone for our transformation.
It is difficult to change well-established processes especially when individuals feel that they will lose their status-quo. In our first steps towards agile, we’ve found that the roles within the company were blurred with respect to waterfall. During the transformation the new roles should be clear as soon as possible. This includes everyone, starting from the top of the hierarchy to the bottom. It is no longer the manager that decides for the team, it is the team (which includes the customer) that makes decisions, factors that may introduce uncertainty and difficulties. As teams get more responsibilities they should also take ownership. Individual ownership should be capitalized by knowledge sharing and teams should motivate team ownership. Companies should award team ownership: teams win and not individuals. Individuals make the effort, use their performance assessment metrics for improving and growing but if the team doesn’t improve and grow and if team metrics are poor in the long run then this will negatively impact everyone.
Relationship With Customers
The relationship between a waterfall company and its customers may be a typical supplier-to-customer relationship. Little interaction and feedback exists and much attention is given during contract negotiation. Agility, however, requires frequent customer interaction and constant attention. The quality of the customer’s feedback is an important success factor. As the customer was made a part of our team, the typical supplier-to-customer relationship was gradually transforming into a partnership.
Organizing Without Traditional Milestones and Committing Work
Committing work for every department was strictly time-boxed and planned in advance, long before reaching the execution date. There existed product milestones like business feasibility study (business plan), definition of product/version requirements, completion of analysis and bring-up, completion of system test/customer release, completion of customer acceptance test, end of sales and end of contractual obligations. From a software development perspective milestones like interfaces available, source code readiness, unit testing completed, integration test-cases available, completion of integration tests, synchronization milestones and checkpoints, beginning/ending of system testing and beginning/ending of field trials phase is a snapshot of the multidimensional organizing elements that we used traditionally.
Letting go of most of these milestones, performing relative estimation based on user-stories and story points, required time to get it done correctly. We found difficulties with scope creep, story-point estimation, and over-committing work. However, the experience gained from iterative planning of the multi-waterfalls stage did help us in this case.
Transforming From a Quality Assurance Perspective
The QA teams especially have had difficulties during the transformation. The challenge was two-dimensional: A mentality challenge and a technical challenge. Going from an independent QA team to an integrated agile team was a big change. The tester had to travel far beyond his comfort zone. From a technical perspective, in the agile teams, everyone developed and tested code. This was more difficult for the testers although our multi-dimensional quality assurance during waterfall (Figure 3), have alleviated this to a certain extent.
The Dynamics of Previous Attempts
Previous transformation attempts even at a different context and not towards agile, if unsuccessful they may cause unease and resistance for future attempts. New attempts will be more difficult since people within the company may be skeptic about agile (they may even campaign against it). Change involves uncertainty. People’s comfort zone usually includes stability and not uncertainty. Some people need to suffer before they decide to change and if they panic after a change they tend to revert to old habits, even if old habits are bad.
Hierarchical Levels, Rules, and Independence
Large companies tend to have many hierarchical levels, encourage a directive management style involving bureaucracy and rules, striving for control and to prevent chaos. Independence requires experience and experience implies technical and soft skills. An inexperienced employee will find it difficult to be independent and it is likely to be more comfortable in following rules. A scrum team of self-organized professionals involves highly skilled and experienced members. It involves people who like ownership and independence whilst following rules and achieving a balance between individual and team goals. Multiple hierarchical levels, where independence and self-organization could be suppressed by managerial authority, may impede agility.
A Three-Dimensional Framework: Transformation Guidelines
Despite the differences in factors like processes, practices, mentality, scale and maturity between software companies we attempt to provide a minimal framework for guidance. Three categories can be identified, namely, structure, mentality, and processes, and they may provide a basis for generating a transformation recipe.
Centralized or decentralized, flat or tall, a company’s structure defines how resources are distributed. It may be based on services, products, markets, or in geographical areas, among others. Although a detailed analysis is beyond the scope of this article we found that agility requires flattening the structure. Reducing the number of hierarchical levels may help the communication between different levels. Software architecture, analysts, development, testing and deployment departments could be the building blocks of a structure that enjoys and empowers synergies. We’ve found that synergies between all phases of the waterfall SDLC are a prerequisite for the success of future agile scrum teams.
Top-level management’s decision making should be based on a clear and stable direction whilst being dynamic. It should be both proactive and reactive. Middle-level managerial mentality involves exercising ownership whilst leading by example and embracing employee independence and self-organization. There must be a balance between following rules and independence. Which rules and how many rules, needs to be experimented and decided per company. Independence brings diverse viewpoints that are desirable, keeping in mind that everyone must be aligned on a common direction.
Focusing on projects may deviate from focusing on products, which may be detrimental. In large software projects people may focus on their function or profession and not on products. A product-focused mentality relaxes functioning and profession constraints and welcomes learning whatever is necessary in order to build the product. This mentality shift is not easy and it requires training.
Processes determine how things get done and they should be lean and flexible. This includes planning, communicating, predicting, designing, controlling, developing, testing, reporting, documenting, etc. To get things done processes are usually decomposed into rules.
There are interdependencies between the three. Large waterfall companies tend to have tall structures and inflexible processes. Although reducing hierarchy levels may allow for more flexible processes and independence, a mentality that welcomes changes is always a good start.
Waterfall stability and scale need not exist against speed and flexibility, the much desirable agile properties. Waterfall companies with the appropriate structure, processes and mentality may achieve dynamism based on stable steps. This is not an easy task and it may require substantial reformations, restructuring and lay-offs.
We explored basic ingredients of our transformation recipe dictating possible courses of action, difficulties and lessons learned. If a company can make the transformation it will do it via its own recipe, a recipe however, that could be based on guidelines. Agile is not only about following “agile processes”. It is not scrum or sprints that brings agility. They can help only if the appropriate culture and competencies exist.
Turning from waterfall to agile is a change. A waterfall company that does not possess any agile-like characteristics will find the transformation to be a big change. Big changes take time and they must be broken into smaller changes. It is important that even small improvements and contributions are recognized and celebrated.
Agile software development may not be for everyone but it is also important to remember that agility is not binary. There is no dilemma like either you are agile or you are not since the level of adoption is a continuum: some are more agile than others. Some companies may never be fit for this kind of software development whilst for others, possessing a few agile-like characteristics could be what they need. There are cases where it is not possible for a company to be agile in all products. Some products must remain in waterfall production whilst the rest of the products can be agile. In order to decide about a company’s ability for agile software development, training, and time to experiment are necessary.
The worst-case scenario should be avoided: A transformation attempt that was considered to be “catastrophic” since it had a detrimental effect to the company’s overall performance. The outcomes were a waste of resources, a demoralized personnel and a company reverting to the waterfall software development mode. Under such circumstances, it may be useful to remember that a learning organization  may learn more from failing to transform into agile rather than succeeding. Learning has to do with realizing mistakes and improving.
Companies expect tangible and sustainable improvements by transforming to agile. Although intangible improvements could also be important, a waterfall company that submits the effort and the resources to make the waterfall-to-agile transformation expects to improve. If it manages to cope with frequent requirement changes, fast releases and if it is more successful (e.g., attract more customers, increase ROI), then this is a success story. After all, it’s all about learning and improving, irrespective if you’re agile, lean, waterfall, or a hybrid.
- “Managing the development of large software systems”, Royce, W. W., IEEE WESCON, vol. 26, no. 8, 1970, pp. 1-9.
- “Managing Technical Debt: Reducing Friction in Software Development”, Kruchten, P., Nord, R., Ozkaya I., Addison-Wesley, 2019
- “The Lean Mindset: Ask the right questions”, Poppendieck M., Poppendieck, T., Addison-Wesley, 2013.
- “The Fifth Discipline: The Art & Practice of The Learning Organization”, Senge, P. M., Crown Publishing Group, 2006.
Opinions expressed by DZone contributors are their own.