Technical Debt - Part 1: Definition
· document system architectures
· provide adequate code coverage in QA processes
· implement standards and conventions
· properly understand the technologies leveraged
· properly understanding the processes the technology is designed to support
· refactor code base or architecture to meet changing needs
Cunningham followed up with a brief video to further describe and clarify the concept in 2009. The debt metaphor he describes fits surprisingly well. Just as in finance, debt can provide a valuable tool to expand more rapidly than you would be able to otherwise. However, if you accumulate too much debt the majority of your resources will need to focus on servicing that debt. One can think of the principal amount of the technical debt as the cost associated with fixing the problem and the interest as any business cost associated with the shortcomings. The loss in productivity a programmer might experience enhancing an undocumented application is an example of interest.
The key difference between financial and technical debt is that the former is usually very visible and easy to quantify while the latter is not. There aren’t any financial calculators that allow you to input a few parameters and have it calculate a number that tells you how much technical debt you have. When you assume technical debt you are acquiring an asset (working software) at a lower expense (in terms of time or capital) than the true cost of the asset. The gap between the price paid and the true cost represents the technical debt.
Organizations with mature risk management practices will often track these types of technology gaps in a centralized way. However, few have formal governance processes to identify, quantify, prioritize, and resolve technical debt in an effective and coordinated way. Even if those processes exist they usually reside in different areas of the organization (i.e. internal audit, operational risk management, etc.) and coordination becomes difficult. I will talk about each of these processes in detail in subsequent posts.
There are several types of technical debt and some have even attempted to create a taxonomy to classify it. However, at a high level the most important distinction is the difference between strategic and nonstrategic technical debt. Strategic debt describes situations where tradeoffs are made in a proactive and measured way. One might wonder why anyone would choose to make compromises that will have consequences down the road. The two most common reasons debt is accumulated is to save money and/or time. It can make perfectly good sense to take shortcuts to get across the goal line when there is a limited budget or time to market is a big concern.
What makes debt strategic is the fact that the pros and cons were evaluated and an informed decision was made by the appropriate stakeholders to assume and manage the debt to help meet strategic objectives. It is important to recognize that just because someone was aware of the technical debt beforehand and made a conscious decision to assume it does not make it strategic. For example, the debt created when a development manager decides on his own to postpone documenting the system his team is building until after implementation cannot be considered strategic to the organization. It may be strategic to that particular manager’s objectives but may be in conflict with the interests of the organization as a whole.
The decision to assume the debt must be explicit and made by key stakeholders based on a reasonably accurate assessment of the risks and rewards contained therein. Additionally, process must exist to manage technical debt to ensure that it maintains visibility and is paid back at the appropriate point in time. Any debt that does not meet these criteria cannot be considered strategic. Below is a scenario which describes strategic technical debt.
An emerging opportunity has been identified and the ability to fully capitalize is directly tied to the organization’s ability to deliver a timely solution. There is also a limited amount of budget allocated to providing a production ready solution. To meet project goals, an open source platform which allows rapid application development is chosen. However, the platform has not been proven to support the volume which would be eventually required if best case projections materialize.
The potential gap the platform brings to the table is discussed with stakeholders and without the reduced cost and shorter timeline the architecture provides it will not be possible to move forward. To mitigate the risk, the company assigns resources to closely monitor usage and routines are put in place where senior leaders meet twice a month to evaluate revised projections. If and when the projections indicate that the platform might not be able to support usage, the plan to migrate to the long term architecture will be executed.
While assuming strategic technical debt to help meet objectives can be in the organization’s best interest, assuming nonstrategic debt almost always is not. Nonstrategic technical debt describes situations where gaps are created without stakeholder approval and the implications to strategic objectives aren’t properly considered. The sources of nonstrategic debt are numerous but often result from process deficiencies.
One frequent source of nonstrategic technical debt is low quality code. Sometimes code is written poorly right out of the gate because standards and conventions aren’t enforced and QA processes don’t exist. A more common reason code quality suffers is because shortcuts often must be taken to meet the time constraints usually associated with enhancement requests.
In software development, system requirements change often. This change can be due to real changes in the environment or our understanding of it. As the environment changes or our understanding of it evolves, the code we write must be modified to support that. These modifications can make systems complicated and difficult to support if routines aren’t put in place to refactor code when necessary. It is natural that systems are complex but real challenges can arise when they become complicated. There is a distinct difference between complexity and complicatedness and much has been written on why the distinction is important.
I will dive deeper into governance in a subsequent article but it makes sense to point out that that many sources of nonstrategic technical debt such as poor code quality can be managed or avoided. By adopting development processes that embrace change and put an emphasis on code quality is critical. Many Agile processes implement controls to mitigate the risk of poor code quality. Extreme programming (XP), for example, leverages pair programming where developers write code in pairs which increases code quality by having the benefit of two different perspectives on how to solve technical challenges. The Scrum development methodology requires that a “product owner” who represents the business and QA resources play active roles directly in the development process. Alignment to business objectives and quality are baked directly into the development process instead of being an afterthought.
It is important to point out that technical debt means different things to different people. How much agreement you need to get across your organization depends on what you plan to do with the inventory of technical debt once you go about compiling it. Regardless if you are creating an inventory to manage it discretely within a small development team or creating an enterprise wide governance initiative, it is important that business stakeholders buy into the definition.
The ultimate goal of defining and identifying debt is to manage it. Including business partners early in the process is key because once the technical debt is identified and measured, remediation efforts will need to be prioritized along with other enhancement requests. Business partners are much more likely to trade off implementing new features for paying back the debt if they’re included early in the process. Vetting out the definition with the business will help you stay on the right path as you go about turning over rocks.
Gartner defines IT debt as the cost of clearing the backlog of maintenance that would be required to bring the corporate applications portfolio to a fully supported current release state. While this definition is broad a lot is left to interpretation. A critical question becomes what does “fully supported current release state” really mean? If the application itself is free of technical debt but there are quality issues in the underlying data does that qualify as technical debt using the Gartner definition? What if those data quality issues are from upstream systems and you’re paying interest on someone else’s debt?
Something else to be aware of is that most definitions of technical debt available today focus on workarounds that were implemented during the development process. Your business partners may want to cast the net wider than the traditional definition and include technology risk related items such as control weaknesses. Coming to a consensus on a definition with the folks who are writing the checks (i.e. business partners) and prioritizing the backlog will pay dividends during the remediation process.
The debt metaphor is an important tool in understanding and communicating the implications of technical debt. Using this metaphor, free cash flow can be thought of as the amount of resources available to maintain and enhance systems. Cash flow is a finite resource that must be allocated carefully. Failing to understand how much of that cash flow is being spent servicing technical debt can lead to poor decisions on where to most effectively invest. This could eventually lead to technical bankruptcy where all an organization’s resources are spent dealing with the inefficiencies created by the debt they have accumulated and they can no longer keep up.
While reaching the point of bankruptcy is an extreme case, I have seen very real examples of this scenario first hand. Once an organization comes close to having this level of debt there can be devastating consequences. Even if systems are completely internal, impacts to customer service usually arise. Just like financial bankruptcy, sometimes the magnitude of the underlying issues are so severe that it becomes impossible to emerge. In the worst case scenarios, systems have to be scrapped and rewritten, entire departments are outsourced, customers and market share are lost and shareholders lose confidence. Therefore, it is critical that organizations create a plan to proactively deal with this phenomena.
In my next article I will discuss how to identify technical debt within an organization. Subsequent articles will discuss the rest of activities in managing the lifecycle including quantifying, resolving and governing technical debt.