Planning for Agility
Planning for Agility
Learn what goes into proper planning and how to make great software in the age of cloud-first, cloud-native, and coronavirus.
Join the DZone community and get the full member experience.Join For Free
We in the software industry are not only caught between a rock and a hard place, but we’re toeing the edge of a cliff at the same time. As a group, we all serve at the pleasure of the business—even if software is our business. That is, the software we write is not an end in and of itself but a means to the end of a thriving (i.e. profitable) business. But we’re also in the midst of a complexity boom in technology with cloud-first, hybrid-cloud, and otherwise distributed systems being the preferred model for software development and delivery. To top it off, we have been asked to do more with less since the great recession—a request that is going to become even more fevered in the wake of the COVID-19 pandemic.
We must find a way to partner with the business, be nimble in our approach, and do all of that with the utmost efficiency. In order to achieve all three of these things and airlift ourselves out of this precarious situation, I suggest a radical solution: planning. Now, before you scream, “Waterfall,” and run away, plans come in all shapes and sizes and can be used for many different purposes. For the sake of my argument, I am looking at plans in a very specific way and can say with years of experience over multiple successful projects that:
- Proper planning is the only way to understand the most efficient way to build something with the fewest number of resources.
- Plans can be built to absorb change and uncertainty.
- Planning does not require every last detail of the requirements to be known in advance.
- Generating a plan can help you understand a project better before you start.
And, most importantly:
- Every plan is subject to change.
However, just because a plan may change over time does not mean they aren’t still critical. How can we be a good partner to the business if we can’t tell them when we’re going to be able to ship what they’re developing marketing plans for or what they’ve promised to customers? How can we be a good partner to the business if we don’t really know how many people we need to develop a set of features or when hiring an expert consultant would make more sense than dedicating internal resources to learning a new technology analyzing a new set of regulations?
Building and running a project plan requires some work—and a new way of thinking about projects and tasks, but once that work is done, all of the needling questions of the business can be answered with something other than a SWAG made while keeping your fingers crossed behind your back. Questions like:
- When will this be done?
- Do we have time to add this thing an important client wants in the current release?
- Do you really need more people?
- When can we fit in this feature?
- Why can’t we fit in this feature?
And so on.
For anyone who thinks that putting cards on a board and shifting them to the right is planning, it’s not. Proper planning occurs at an entirely different level from individual tasks on a Kanban board and requires:
- Different task granularity
- Full dependency identification and validation
- Critical path and staffing analysis
- Risk analysis through task float/slack
Task Identification and Dependencies—Escaping the Rock
This first step stops many people before they start. It’s true that if you’re mired in a code base where pulling one thread over here unravels an entire sweater over there, estimating the time it takes to create a new feature may range from daunting to impossible. But there are some techniques that can help all of us, no matter if we’re working with clean code or something that’s passed through so many hands that each bug fix is a death-defying act.
First, let’s talk about task granularity. When planning a project of any significant size (more than a few weeks of work for a single person), task duration should be on the order of days. We have accumulated a long history of looking at tasks in terms of hours, and while that might work for tracking progress, it fails miserably when planning. Planning at that level of granularity will leave your plan shaking with every change of detailed requirements.
However, when looked at through the lens of a week and across many weeks and many tasks, plans become resilient to changing and emerging detailed requirements. Identifying more coarse-grained tasks means that they are no longer tied directly to the details of a single requirement. Given that they are now aggregations of smaller efforts, each of these tasks has some give to it and is under less pressure to have the exact-right estimate assigned to it. In addition, in a large enough plan, the law of big numbers kicks in. This says that as sample size (number of tasks) grows, the average of a subset of those samples will converge with the overall average. In short, it all comes out in the wash: within a large set of tasks with a reasonable distribution of durations, if one task is underestimated, it will probably be balanced by another task that’s overestimated.
If, even at this level, you’re having a hard time estimating tasks, you may need to engage in a bit of CYA by creating a de-risking technical proof-of-concept task that you wire up as a dependency to the actual task to execute the development work. This also creates a visible “spike” earlier in your project that you can use as a communication tool with the business.
But identifying tasks is only the first step. We also must understand what has to get done first. During planning, you must spend the time to identify what components in your codebase need additions or modifications and then untangling the web of dependencies among them to understand what tasks need to be done before other tasks can start. CAUTION: this is an iterative exercise. With each pass, you will stumble across tasks you missed the first time around and dependencies you didn’t know were there. Remember to include tasks that cover UX design, integrating components, end-to-end testing, and UAT. The plan should also encompass all activities required to ship your software, whatever that entails.
Lay out the tasks, their durations, and their dependencies in a tool like Microsoft Project and use the resulting Gantt chart to look for problems. Do you have tasks that don’t have anything that depends on them? If so, why are you doing them? Do you have circular dependencies? If so, maybe you need to add another task to refactor your code to remove that circular dependency in the first place.
The Critical Path—Addressing the Hard Place
Once all tasks and dependencies have been identified and articulated within a Gantt chart, it can be analyzed. The plan will have one (and possibly more) critical paths, which is a series of tasks that describe the longest chain of dependencies in the project. Each task on the critical path will start directly after a previous one with no float (or slack, in Microsoft Project parlance). By default, no matter how many people you try to pile onto the project, it can never be done sooner than the critical path.
The critical path will tell you immediately if the timeline someone is surely already requesting (or, worse, setting) for this project or feature is feasible or not. In my experience, this, alone, is worth the cost of creating the plan in the first place. The critical path also tells you where on your project you should assign your best resources. Not only does the critical path define the shortest duration of your project, but any delay along it will, by definition, delay the entire project’s release. A task on the critical path is not the place to put your new junior developer!
Staffing—Edging Away from the Cliff
Planning and executing a project this way is a wholly different experience from coming to a project with a set team and letting tasks flow over that group of people, which is rarely the most efficient way to approach something. Tools like Microsoft Project will help you allocate resources to tasks in a way that ensures you’re not double booking anyone. Staffing in Project is a bit of an art (hint, staff along the critical path first), but it enables you to use float on non-critical tasks to your (and the business’s) advantage. When staffing a planned project, you can reduce the number of resources required for the project by consuming float on some tasks.
Consider two ten-day tasks that are not dependent on each other, like Tasks 1 and 2 below:
Shifting Task 2 to start on 5/4 consumes 10 of the 15 days of float but now allows you to assign the same resource to Task 1 and Task 2, reducing your total resource needs:
Staffing a project this way, which is also an iterative process, is the only way to identify the smallest feasible team to safely produce the required functionality, understand how that work should be assigned, and identify the order in which it must be done. If you are asked to do more with less, having a plan like this in your hand provides the ammunition to go to the business and explain exactly how many people you need and what will happen if you’re required to execute with fewer. In a win for the business, running a super efficient team on one project may free up resources to work another project in parallel, potentially reducing the opportunity cost on a different product line.
Risk Analysis—The Airlift
At this point, the hex you may be putting on this idea has probably shifted from “Waterfall!” to “Best laid plans …” While making a plan in no way means that plan won’t (or shouldn’t) change, many of failures in “planning” come from creating a plan that is so hard to execute or so susceptible to small missed hurdles that it is doomed to failure before execution even starts. Every plan must be interrogated for overall risk and execution complexity before it is committed to—or even communicated to stakeholders.
First, risk. If we think back to dependency identification and critical path analysis, it’s easy to see that risk is directly related to float. Imagine a project where every task is on the critical path. In this scenario, if any task in the plan is delayed at all and for any reason (someone gets sick, the scope changed slightly, a resource got pulled off for a day to do production support), the project will not be completed on time. For this reason, an all-critical project is much too risky to be successful.
Risk can be quantified such that it can be compared across plans—both different plans for the same project and across different projects over time. Righting Software by Juval Lowy details a number of different ways to calculate across the floats in your project to arrive at a single value to communicate the risk in a project, including activity and criticality risk. The key takeaway here, though, is that risk must be interrogated, and if a project is too risky, steps must be taken to reduce the risk before the plan is committed to.
The easiest way to “de-risk” a project is to introduce float along the critical path. This can be done most simply by shifting a task later in the project that is on the critical path out one, two, or even three weeks to introduce float along the critical path (and increase the float among all non-critical activities). This gives the plan a chance to absorb those unexpected bumps and unforeseen circumstances that are a part of life. I recommend significant de-risking when just starting out because being able to deliver what you said when you said you would it is more valuable than trying to deliver something a week earlier and failing to do so.
But critical path and float are only one side of a project’s risk. In any sufficiently large project, there is likely some degree of execution risk as well. Consider the following network diagram for a nine-month, greenfield project I was involved in several years ago:
The bold line in the plan indicates a critical path. In fact, If you look closely, you’ll find two parallel critical paths in the top portion of the project plan. In addition to those critical paths, there’s yet another set of parallel tasks shown in the bottom section of the plan. Even if the risk quantification across all the activities’ floats in this plan indicates a project with a tolerable risk level, it will take a very mature team with active project management to keep track of all of the different threads of execution and integration points and make sure all are ushered through to completion on schedule.
Execution risk is largely subjective, though the project above should leave pretty much anyone with a sinking feeling in the pit of their stomach. If, when looking at the network diagram, it is a sea of parallel paths or massive integration points, that is hardly a recipe for success. If the shape of your diagram leaves you wondering if you’re up to it, you should consider introducing logical milestones or splitting the project up into independent sub projects that can be managed independently.
Partnership and Agility
After all this what and how, let me come back to why. We have so much on our plates already, why add the burden of creating a plan that, let’s face it, is going to change when the business decides that this other shiny thing is suddenly more important or you have a major production issue that requires all hands on deck? How is planning out a nine-month project with seventy tasks agile?
What is agile? If agile means true business agility and if it means transparency and partnership with stakeholders, I posit that you can’t have agility without a plan. Having a plan eliminates the guesswork in answering questions about staffing or timing or scope. It enables detailed tracking of progress, and it allows you to quantify the effects of change.
Let’s focus on the last two, specifically. First, progress. I’m sure we’ve all experienced the “80% done 80% of the time” syndrome that paralyzes many projects. With a plan that encompasses all the tasks you need to accomplish for the project to be complete, it’s possible to track progress with incredible accuracy and report progress to the business with complete confidence. You can also know very early if you’re ahead or behind on the overall schedule and can be proactive about communication. I speak from experience when I say that the only thing the business loves more than delivering on time, on budget, and on quality is being told far in advance that a deadline is going to be missed instead of learning that right at the finish line.
Now, let’s talk about change, which is the reason we threw away plans in the first place. If everything changes all the time, why plan? Because planning is essential for reacting to change in an intelligent way. Let’s review the bidding:
- Plans should be created at a coarse level of granularity that does not tie tasks directly to detailed requirements (that will change).
- Plans should include float and be at a tolerable risk level such that change and some degree of mis-estimation can be absorbed without changing the project delivery date.
- Plans can also include milestone deliverables to help stakeholders confirm the approach and functionality in incremental steps.
- Plans allow you to understand the cost of a change that’s requested.
The last one is what I find most interesting. Say you’re midway through a three-month, five-person plan, and management asks for a significant addition to one of the features you’re building. In any sufficiently complex network of items, how a change affects that network becomes very difficult for a mere mortal to predict, but with a plan, you can save off a copy of it, make the adjustments required to satisfy the change request, and see what happens. How much does it extend the plan? Does it require another resource for a period of time? If they’ve asked you to maintain the same delivery date, what scope do you have to cut in order to finish on time and maintain a tolerable risk level?
There is no worse feeling than not being able to deliver on a promise you’ve made to the business and no better feeling than to accomplish what you set out to do while avoiding a death march—or endless sprinting. Yes, plans change, but planning should still be a constant in your toolbox and can become a key to transforming your partnership with the business—and your career.
Opinions expressed by DZone contributors are their own.