Estimating Software Projects: A Practical Approach for Tech Leads

Introduction

Accurately estimating software projects has been a big challenge for technical leads for quite some time. While there are many established techniques in the market that explain how to estimate a task at hand, they don’t often provide a systematic process to break down tasks, account for unknowns, and track and revisit the estimates. In practice, estimation is an ongoing exercise that keeps evolving with requirements, spikes, and development.

A successful estimation requires an organized approach including collaborating with product managers and architects to clarify and align initial requirements, conducting spikes to reduce uncertainty, systematically revisiting and refining estimates, and incorporating effort for testing, code reviews, and deployment tasks into the planning. It is also important to account for buffers to handle unforeseen delays. 

This article presents a framework for tech leads to improve their estimation, ensure proper delivery as promised, and proactively negotiate resources and timelines with the leadership. 

Working With Product Managers To Define Initial Requirements

It is important to have an alignment between product managers and the engineering team on what is being built. A tech lead should always properly review the PRD (product requirement) and clarify any questions/requirements with the product manager. Improper understanding of requirements is usually the main reason behind inaccurate estimation or delays in delivery timelines. 

Once the requirements are clear, the tech lead can translate these requirements into a high-level design (HLD)/ERD. An HLD/ERD would help bridge the gap between functional requirements and technical implementation. An ERD is mostly an entity relationship diagram that usually defines the relationships with multiple components, the data flow, and the workflow. While this would be pretty high-level, it would help to get the product management and the engineering team on the same page. 

It is also important to get the document reviewed by any senior technical authority, like architects or peers, if that’s a company requirement. At this stage, the technical architects may raise concerns related to their product or suggest adjustments to design, keeping quality attributes/non-functional requirements in mind like scalability, consistency, usability, or platform constraints. Most product managers would focus on product features; however, as a tech lead, it is important to think about non-functional requirements and address any concerns. This would be the right point to get consensus before any engineering efforts get started. 

Once this agreement is reached, it would be easier for technical leads to understand the high-level effort required for the project at hand. The work can then be broken down to tasks. Depending on the complexity of the project, a follow-up detailed low-level design may be needed, covering the system interactions, APIs, and edge cases, which can also be added to the list of tasks. Although this step is often overlooked, this initial phase of requirement gathering and coming up with ERD plays a critical role in avoiding rework and providing accurate estimates. By clarifying expectations early, teams can avoid misunderstandings later during implementation.

Building Initial Estimates With Low Confidence

Once the initial requirements are clear and HLD is reviewed, the next step would be to create a rough estimate based on the initial breakdown of tasks. Since this is a pretty initial stage in the SDLC cycle, there can be many unknowns, and therefore, estimates can be provided with low confidence. Usually, the management has a scale to determine the confidence of an estimate, like low, medium, or high. And a low estimate just means that this is not a final estimate, as there are many technical unknowns and the estimate would change. It’s a ballpark figure. However, early estimations are valuable to the management as they help them frame some rough timelines and identify the need for further exploration. 

The estimation provided should go beyond development tasks. It should also account for other important tasks like integration testing, performance/load testing, spikes, prod security review, etc. These items are often missed in the first round of planning, leading to underestimation. In some cases, prior experience would help with providing more confident estimates. However, when building something new from scratch, a spike or a POC(proof of concept) becomes crucial. These should also be considered as tasks while estimating. To accommodate any unknowns and delays, it’s always advisable to add a buffer to the estimate, typically a sprint or less based on the overall sizing. 

Below is an example structure of a rough high-level estimate:

A few tasks in the above list may or may not apply to all projects; for example, performance/load testing might be done by some other team in a few companies. At this stage of planning, the idea is to just list down all the types of work items that are required to get this feature work done, and it need not be precise or granular. However, it is important to communicate this. It helps stakeholders understand the areas that still need clarification.

Conducting A Spike To Uncover Unknowns

A spike is an exploration activity intended to reduce uncertainty and provide the necessary detail to implement a feature. It's mostly coming up with high-level or low-level design that translates product requirements/ERD into concrete implementation detail. 

Each feature or component outlined in the ERD should have a dedicated spike defining the 

• API interfaces and contracts
• Data model changes
• System-level interactions
• Non-functional requirements, such as consistency, scalability,  performance, or security

For example, a new metadata storage feature would require designing CRUD APIs, data model, and caching logic, etc. While a search execution feature may involve query parsing, execution flow, and returning results in an optimum way. Therefore, the metadata storage and search execution each should have their own SPIKE.

Spikes may not be a one-time task; a few complex projects might require multiple spikes to cover each of their smaller components. These should also be called out in the initial estimate, as they would contribute to both the design and delivery. 

By having dedicated spike tasks, teams can reduce risk during implementation by clearly understanding the implementation details and what is being built.

Systematically Estimating Development Work

Once spikes are complete, you get a better clarity on what needs to be implemented. Estimations at this stage should be more granular and accurate. Each task can now be better scoped based on granular level tasks like code reviews, unit testing, etc.

For example, implementing a cache, the effort can be broken down into

• Development (3 days): Core logic and integration
• Unit testing (1 day): Writing and validating test coverage
• Code reviews (2 days): Including iterations and, in some organizations, walkthroughs with subject-matter experts
• Functional and integration testing (1–2 days): Ensuring the feature works end-to-end within the system
• Load or performance testing (1 day): Validating the caching mechanism under real-world conditions

And for implementing a data model, the effort can be broken down into

• Development (2 days): Writing scripts and creating tables
• Running migration (1 day): To apply all DDL changes
• Code reviews (1 day)

These estimates vary depending on the company’s engineering practices and the features being built. Like the data model change requires an entirely different set of tasks compared to building a caching layer. Also, code reviews may be simpler in a few companies but may require a formal walkthrough and sign-off in others. It’s important to allot time as per the process and guidelines of your company. If anything takes more than a few hours, then it should be included in the estimation.

At this stage, teams can use various estimation methods available in the market, like use case points, wideband Delphi, or story points. I have used estimates in terms of workdays/sprints.

Should Include Testing and Other Quality Work

Software quality related tasks are often overlooked while estimating a project though they are really critical and are mandatory for the final product delivery. Engineers are required to spend time on unit testing, integration testing and performance testing and bug fixing to deliver a quality product. And not estimating these tasks would only result in delay in delivery timelines. 

In some organisations, additional processes like product security (prod-sec) reviews and threat modelling might be mandatory which may require some amount of a developer's time. And therefore time for such tasks must also be accounted for during estimation. 

A common misconception amongst the leadership is that “writing code” constitutes all the engineering work and therefore the product should be delivered within a few days. And is a small effort feature. However, in reality, development is just one aspect of SDLC, and there are many other work items to deliver an entire feature, like we spoke about earlier. And it’s important that the estimation reflects all these additional work items.  

By systematically accounting for all the work and not just feature development, tech leads can set realistic expectations and advocate for the right resources while providing accurate delivery timelines.

Negotiating Resourcing and Delivery Timelines

Now that we have detailed and accurate estimates in hand, the next step would be to align with the managers and product managers on delivery expectations. This begins with understanding the release timelines and evaluating if the estimates fit that window. 

Mostly, the engineering managers would do this calculation and would allocate resources based on your estimates. For example, if the project requires 115 days of effort and the target is to deliver within 20 days, then the manager might assign 6 engineers to the project, depending on what work can be parallelized. 

Apart from parallelization of work, it is also important to understand the resource experience. These things the technical lead should call out if he sees concerns and negotiate what can be done and what cannot be done based on the resources, parallelization of work, and timelines. By proactively identifying constraints and discussing trade-offs early, tech leads can avoid surprises later.

Tracking and Revisiting Estimates

Once the delivery timelines are finalized, the last step would be tracking the progress against the estimates. Usually this step is ignored by tech leads which can result in projects going off-track. However good the estimates are this step is a crucial one to ensure things are delivered on time.

This need not be a very detailed tracking system. A simple excel sheet that maps who will work on what and when, would provide a good foundation for tracking. This tracking is not a milestone tracker like the managers or project managers do but rather a granular level development/testing work items tracker. This would allow teams to anticipate risks early and pivot in a different direction if needed. 

The following table shows a simple tracking table with each task linked to a sprint and an engineer

Regularly revisiting this tracking sheet would help to pivot in a certain direction if required, like if a particular development task is taking longer due to some refactoring(that wasn’t accounted for). Then this can be proactively caught using this tracker. This would help tech leads and managers pivot a bit by pulling in an additional engineer or moving the refactoring work for later as a tech debt item. Updating task status and reassigning or shifting it can help with overall timeline predictability. Regular checkpoints would allow teams to re-align and, if needed, re-negotiate scope, engineers allotted, or timelines.

Conclusion

Estimation helps engineering teams to commit, prioritize, and deliver with confidence. While there are many estimation techniques in the market, none of them talk about the entire estimation process. And this article tries to focus on that which would help tech leads to provide accurate estimates.

From clarifying initial requirements, conducting spikes, systematically breaking down development work, negotiating resources, to tracking estimations, all these phases play a crucial role in estimation. Approaching estimation with a well-defined structure would help technical leads to pivot as needed, call out slippage earlier, and deliver software on time as promised.