Value Stream Management Essentials

Value stream management (VSM) is the lean practice of monitoring, evaluating, and continually improving an organization's ability to deliver valuable products for their customers. VSM, when appropriately scaled, can help an organization generate higher value in a variety of ways. From improved collaboration and communication across the entire software delivery lifecycle to actionable insights and metrics backed by real-time data flowing through an organization, there are many benefits for organizations to look at their own processes on an ongoing (and hopefully automated) basis:

• Development teams can get instant feedback on their day-to-day processes, helping them to identify and understand bottlenecks as they occur.
• Product managers can accurately communicate to the rest of the business and their customers about delivery dates, helping them better manage expectations.
• Release engineers can start to shift governance concerns left, helping to identify risk, minimize rework, and improve quality.
• Executives can help ensure that engineering activities are aligned with business goals.

VSM is about creating value by making work visible, reinforcing best practices, removing bad ones, and thus succeeding faster as a development team and organization. VSM is not a new concept, but in this digital world, VSM has proven itself an essential everyday tool to help ensure a high-performing DevOps organization.

One of the main outcomes of value stream management is continually measuring and improving the software development lifecycle. For this, we need to shift the conversation from only being about Agile or DevOps to a more comprehensive dialogue — one that spans from a business idea all the way to a customer deliverable. Many organizations will start their VSM journey with an honest conversation about their current end-to-end processes. Teams will typically begin with how ideas (business value) are created and tracked; then moved into the "fuzzy front end" where they are selected, designed, architected, and developed; and finally, sent into the repeatable stages to be built, deployed, tested, and released to customers.

Oftentimes, this will take place over a few hours in a process commonly referred to as "value stream mapping." The goal of value stream mapping is to come up with a diagram that accurately depicts the current stages of software delivery within an organization and is a crucial first step in being able to manage the value stream. Many organizations go through this process once or twice a year, but that simply is not often enough. Change is constant, and if we want to be sure we are working at optimal efficiency, we need to be able to track our value stream in real time. How does that work? In all honesty, it's not an easy problem to solve.

Efficient value stream management requires a value stream management platform (VSMP), which aggregates data from all the development tools your organization uses on a regular basis to deliver software. Examples of data types most commonly collected are:

• Commit information from your source control tool
• Build information from your continuous integration server
• Any and all testing data that might run as part of a build (unit testing, code coverage, static analysis, security scans, etc.)
• Deployment information to all environments
• Any automated testing data performed post-deployment

Being able to collect this data is key because it is essential to building and establishing relationships between how these tools fit together and how they drive value through an organization.

Ideally, teams using a VSMP can leverage their existing tooling's APIs to associate code changes to change requests and subsequently tie those change requests to builds, test results, deployments, and every other activity that takes place as part of the software delivery lifecycle.

VSM helps improve the three DevOps pillars of people, processes, and technology.

The People Challenge

Now more than ever, teams are working remotely and having a harder time collaborating in a fast-paced development environment. The lack of face-to-face interactions makes "water cooler" conversations a thing of the past. Standup meetings and retrospectives often suffer because it's not obvious how the team is functioning as a whole. It is difficult to know which team member is responsible for certain tasks, and even harder to know when those tasks have fallen behind.

VSM for the People

By tracking work in progress, VSM helps make the work visible, which creates a shared responsibility for the software produced and increased transparency, communication, and collaboration across development, IT/operations, and business teams. VSM, in many ways, starts to humanize DevOps by creating meaningful dialogue around current processes and encouraging a DevOps culture that embraces thinking outside of the box to solve traditional problems.

By adding visibility, teams obtain knowledge about everything that is required for an organization to deliver high-quality software and, therefore, tend to have more empathy for their coworkers. Visibility into how value is produced allows everyone from top-level executives to developers to look at the same data and have much more direct and transparent conversations.

When these groups are aligned, conversations become easier because expectations are set with data-driven analysis rather than feelings. When teams do not fully understand each other's responsibilities, this can cause a number of issues that ultimately delay throughput and can result in finger-pointing. Value stream management can revolutionize a company's culture by refocusing efforts through compassion and comradery.

The Process Challenge

When discussing processes, we tend to focus on how efficient we can make a particular workflow. This requires organizations to constantly reevaluate how they operate to identify new bottlenecks and develop a leaner alternative. Part of the luxury that comes with having a highly responsive, agile development organization is constantly juggling the requests of the business, customers, and their own teams.

At the end of the day, this leaves many with a million-and-one things that stakeholders would like to see done, and probably only 20-30 that can get done during any given sprint. Over the years, teams have used a number of methodologies to try to get better at tracking, planning, and managing work in progress (WIP). From Scrum, Kanban, Extreme Programming to Adaptive, Dynamic, and Lean Software Development, each approach takes a different twist on the same challenge — being flexible while still being predictable.

VSM for Processes

VSM helps teams across the business better understand what can be delivered and when. This is critical from a tracking and planning standpoint. It is crucial to understand what on-time delivery impediments exist and the best way to remove them. Understanding how work flows in and out of a particular value stream segment can help identify the types of bottlenecks you may have.

For example, does work move in and out of a segment in large batches instead of a continuous flow? If so, this might be a great part of your value stream to focus on and improve. By removing the flow imbalance of your epics, features, stories, and bugs, teams can decrease wait times, improve delivery reliability, and delight customers. Later, if we see negative changes in this segment again, we can swiftly identify and address the bottleneck to ensure a streamlined process.

The Technology Challenge

Technology is always changing, and with that change comes uncertainty. Recently, organizations have had to weigh the risk, cost, and value of modernizing their applications into microservices. How do we know if it is worth the engineering investment?

Other challenges in technology are around security and quality. Are our teams taking the necessary steps to ensure they are developing software without vulnerabilities? Are they adhering to our quality standards? Security and quality initiatives always have good intentions but typically wither on the vine without constant monitoring to know when teams have regressed.

VSM for Technology

VSM allows for unprecedented business agility. It provides insights into everything from high-performing teams, the effects of unplanned work, lead times, cycle times, and other key performance indicators that businesses use to determine efficiency. With the right visibility, it is easy to identify which teams are following quality and security best practices. Organizations that excel in VSM can start to manage critical business factors such as risk, revenue, and cost.

By tracking and visualizing a team's quality and security efforts, companies can identify risk earlier in the development process. This allows teams to minimize the amount of rework that takes places, which can help keep costs down. The overall efficiency of the improved value stream will result in a fast speed to market, which has a direct correlation to revenue.

In the digital transformation age, business agility is necessary for growth and survival. It's not only about velocity anymore — companies need to consistently deliver clean customer experiences, such as delivery speed and quality. To do this, you need to have meaningful conversations about trade-offs and an understanding that we only want to deliver faster software while preventing negative affects to the overall quality of the product.

The first step in better managing a value stream is to plot out precisely how work on a day-to-day basis gets done. This should take into consideration all of the curveballs that often get thrown into a given sprint or release. A team can usually plot out their value stream relatively quickly as they are the ones primarily doing the work daily. However, this first attempt is usually very development focused. It may not directly express how teams communicate their work or unspoken dependencies they may have with other teams.

Once you have successfully mapped your value stream, the next hurdle is optimizing it. Organic conversation is the best place to start. In keeping with the spirit of Agile, it is important that your development teams are having a regular retrospective to discuss what they feel they can improve. Even more important is having a culture where the team feels they are responsible for owning any corrective actions to improve their operational efficiency. 

Require teams to discuss the work they are delivering in relation to the value stream. Many teams will keep a printed or white board representation of their value stream in their retrospective room. This allows teams to discuss precisely where they think bottlenecks occur. If a particular story caused problems during an iteration, it's important to understand the cause of the problems. Where did it get held up? Could we have planned better for this situation?

Common reasons that work doesn't get completed are:

• The team was impacted by unplanned work (support, excessive meetings, etc.).
• The work was dependent on input or contributions from another team, and it didn't get done (design, QA, other development teams).
• The team was waiting on feedback from a customer before finalizing the implementation.
• The work was held up in code review.
• There was too much in-progress work, and the team couldn't contain the story.

The feedback gained from these conversations can help teams that have not automated the management of their value streams immensely.

When implementing value stream management, there are a few concepts that are fundamental for a successful implementation:

1. Value stream mapping – The act of mapping out all steps in the software delivery lifecycle. Each stage represents a transition of business value.
2. Bottleneck detection – The process of identifying where business value stalls within your value stream. To maximize flow through the process, you want to remove the worst bottlenecks. A good way to get started is to analyze the length of time it takes for a work item (story, change request, etc.) to move from one state to another (e.g., a story to moving from in progress to complete).
3. Business agility metrics– These metrics focus on ensuring work flows through the value stream smoothly, specifically:
   • Lead time – The time required for a work item to go from an accepted (backlog) to value realization (the customer). This includes both engineering time and non-value-added waiting time between subprocesses.
   • Cycle time – This is similar to lead time, except cycle time is measured from when the development team begins work to when that work reaches the customer.
   • Load – The numbers of work items active or waiting in a value stream at a given time. Load measures utilization capabilities of value streams related to productivity in the process flow.
   • Throughput – The rate of work items completed during a period of time. Improving throughput can result in better responsiveness to customer requirements and may yield lead time reductions for value streams.
   • Work item distribution – The proportion of different types of work items over time. This provides teams visibility into the type of work being completed (features, defects, tasks, etc.)
   • Wait time – An estimate of time that work items spend idle (in a non-productive state) during their journey through the value stream.
   • Build frequency – The number of builds per time period by application. This allows the business to see the frequency of continuous integration and, therefore, the faster integration of value.
4. Value metrics– Broader insights into the full value stream that indicate or predict customer value.
   • Security – The number of vulnerability severities over time and which teams are adhering to security best practices.
   • Deploy frequency – The number of deployments per time period by application and environment.
   • QA metrics – Analysis around your software's quality. This allows the business to know which teams are testing frequently and to trend those results over time.
   • Milestone risk – The work item completion score to identify which ideas are likely to miss their sprint completion dates and ultimately the scheduled release.
   • Interaction metrics – Click through, download, or install rates are all examples of high customer interests.

Using a value stream management approach, teams can make their software delivery practices smarter and provide greater value. Instead of relying on a siloed tool to provide insights into the delivery process, you should now leverage the data across all of your meaningful systems to get a true understanding of how work moves through your organization. When you are using a data-driven approach, it also allows teams to have visibility into how the whole process fits together. Aggregating this data into a value stream management platform creates an automated single source of truth for where work is, and a chain of custody to show how it has made its way through the value stream. This can completely change the meeting dynamic from reactive to proactive.

Instead of a meeting focused on determining where features are or how much longer they might have until completion, meetings can now focus on outcomes and removing potential blockers. When everyone is looking at the same data and the same metrics, something magical happens: The cultural differences between your engineering teams and your business are no longer separated, but instead, everyone is aligned on producing and delivering value. That is why value stream management is not just an instrument for management but is, instead, considered a best practice for everyone — from leadership to developers.