This case study is excerpted from the book, Evaluating Project Decisions: Case Studies in Software Engineering, authored by Carol Hoover, Mel Rosso-Llopart and Gil Taran, ISBN 0321544560, published in Oct. 2009 by Addison-Wesley Professional, Copyright 2010 Pearson Education, Inc. For a complete Table of Contents, please visit: www.informit.com/title/0321544560
You can read the case independently of the material from the chapter. If you choose to read the case study first, then read it once quickly in “story-reading mode” simply to set the context for the information in the chapter. Then read the chapter material thoroughly, and finally reread the case study in an analysis mode.
A software project manager must make an estimate for a job and figure out what issues are influencing the estimate. He must figure out what are good inputs into the process so as to make a good estimate.
Case Study Objective
After reading the case study, you will be able to do the following:
- Understand examples of some estimation techniques.
- Understand that there are different phases in which to estimate and how estimation during the planning phase might be improved.
- Understand how some estimation techniques might be set up to better support an organization.
Use of historical data to improve estimation, overall metrics in estimation, understanding metrics versus data issues, and how to look for similarity in projects.
- Junior engineer: Engineers with less than five years of experience
- Metrics: A relative understanding of how data can be used to evaluate some activities
- Productivity numbers: A measure of how much project work the company gets
- Senior engineer: Engineers with more than five years of experience
Marvin Saymore was being asked to estimate his first full project after starting his new job with Transad. He had to give his boss an estimate, and he was not sure what the primary estimation issues on his projects were. The question was whether estimates were driven by events that were unavoidable or were because of mistakes in estimating past projects. He began to reflect on what he had learned back in school about estimation and what might influence the Washington Transit Authority (WTA) project.
Marvin thought that when people in the company tried to estimate, they tried, in some sense, to predict the future. Marvin was not sure whether this was really true as the new WTA project was being estimated. He thought that estimates were more based on the past and that they were a best guess of what might happen, not a prediction. With that “best guess” came some risk and some probability of success or failure. He had been taught that humans are creatures of habit and tend to do things more or less the same unless a major change occurs in the environment. In a sense, Transad had more or less been the same for the last 100 years of development, so change was not necessarily a big issue for the managers. So, based on this understanding, history should be a good predictor of future activities.
However, Marvin was still left with deciding what past, or, more accurately, what data from the past, he should use. There always seems to be much data, but if you do not know what the data means or represents, then it is hard to decide how to use it. For his purpose, Marvin would accept that data used to follow the progress of a project were valid measurements. Making his decisions more difficult on how to estimate the project was that Marvin had been taught to use more than one way of estimating. This, he believed, would allow him to compare estimates and see which ones seemed more reasonable and in some ways see whether he was aware of all the major issues on the project. Dramatically different estimates for the same project using different techniques might indicate hidden issues or future problems. Marvin realized that to understand the history for projects, he would have to understand the metrics the organization collected on projects in the past.
Marvin first discovered, as he took over the reins as the new software manager at Transad, that metrics were fine for other people but not necessarily for his group. He found out that a number of metric efforts had been proposed, but none had been maintained for a long period of time or were used to predict future performance. The company almost exclusively depended on the expert judgment of the senior engineers and the senior managers in the company for estimates. This would have probably been all right if historical data had been used independently to validate the estimates, but this was never done. In addition, the historical data that was available was mainly “bean counter” data, that is, number of hours on the project, hours preproduction, hours postproduction, and repair hours in the field. The preproduction did not specifically break out any software events, only how many hours were used.
One of the first issues Marvin had to resolve was whether the “bean counter” data was usable in some way. He needed to develop a way to obtain accurate data, which he could use to compare with the historical data he had available. He considered this accurate data, which was once related to his projects, as “metrics.” He used a simple technique over a short period of time that would enable him to understand clearly how each type of engineer spent his time. Once he had enough of this information, he might then be able to use it to validate the overall estimates on the project. Marvin wanted to see whether he could establish a correlation between the bean counter data and real data on his development projects. Table 3-4 gives the resulting data that Marvin obtained.
What immediately became apparent is that accounting for all the time the engineer spent was something that would take a little work. Next, it was also apparent that there was a large difference between how much time the senior engineers and the junior engineers were getting to work on what people considered “real” project work. There seemed to be an inverse relationship between an engineer’s experience and the length of time that the engineer was allowed to work on a project. Marvin wondered whether this interpretation of the data was correct or whether there was another reason that the experienced engineers spent less time on real project work.
Without productivity numbers, these values may or may not have a major impact on the estimates, but Marvin’s gut feeling was that something was very much out of “whack.”
Just Like the Last One
Next, Marvin began to examine the company concept of “sameness” that was used by engineers to evaluate requirements. A number of recent projects had been bid to customers as being exactly the same with “minor” changes. As Marvin discussed this with his engineers, he found that this sameness was not quite the same for everyone involved. He decided to conduct a direct requirement-by-requirement examination of a new project compared with the “base” project, which was being changed. Requirements were directly compared by one of his senior engineers, and any differences other than syntax were noted. There was also an attempt to evaluate how different the requirement was from the original baseline project.
On one of the “same” projects, about half the original requirements had been modified, and about half of these modifications involved significant or moderate changes. In the eyes of management, this indicated a “same” project.
One other piece of information that Marvin began to put together for his projects was the idea of whether there might be some significant events during projects that were contributing to a large number of hours. In other words, Marvin was looking for the 80/20 effect on his project estimates. As he reviewed the data, he found that whenever the hardware group proposed a new hardware configuration for a project solution, there were a certain number of hours needed to integrate the software and new hardware and to integrate the new hardware into the system. Table 3-5 has a number of the data points that Marvin was able to develop from the historical data.
Marvin now felt armed with a number of new elements to help with his estimation efforts. He planned to update his estimation process so that he could incorporate his new knowledge. He had to figure out how best to explain to his boss that he had a new tool to better estimate the WTA project. He thought that by using the information in Table 3-5 he could produce at least two if not three more techniques to help the expert judgment that had been used by Transad to estimate projects in the past. Now all he had to do was convince Enrique, his boss, that this was the right thing to do.
Marvin encounters the following problems in the case study:
- Are the decisions to use the estimation techniques or the information that Marvin has at his disposal effective?
- What issues are most affecting the estimation decisions Marvin has to make?
- What are the major decisions that the manager must make to help him with his estimation problem?
- What would you have done differently if you were in the manager’s place?
- What estimation techniques from the case can you identify that are being used? Were the decisions for these techniques supported in the case?
- What information about estimation did you gather that you did not already know?
- What are the decisions that Marvin needs to make?
- What are the PEAK inputs for these decisions?
- What are the alternative solutions and risks associated with these decisions?
As you look at the decisions that were made concerning estimates in the case, one of the key concepts is that your best estimates must be based in accurate metrics. To ensure accurate metrics, you must make sure to verify that the data feeding the metrics is valid. Although decisions concerning estimates can be evaluated with PEAK, the knowledge can be corrupted if the metrics being used are not correct. “Garbage in, garbage out” is a well-understood concept. But garbage can be hidden in metrics very easily, and the impact on estimation along with the consequences is severe.
Key Ideas to Remember
The PEAK model can help clarify the inputs and show that a decision based on faulty input data is also likely to be faulty.
The decision to put new functionality in hardware rather than in software is not the best decision because it is based on faulty input data. The experience and the assumptions are flawed in the decision:
(P) Problem: New functionality is needed.
(E) Experience: It’s easy to add hardware to the system.
(A) Assumptions: It’s easier and less costly to add hardware than to add software to a system.
(K) Knowledge: Basically, for a hardware company, software is harder to manage.
Solution: Add a new hardware component to the system to meet the functionality.
Assumed Risk: The cost to get software running on new hardware is not well understood and may be greater than expected. The recent cost model shows the cost to be about 5,000 hours to integrate software with new hardware.
The estimation decision to split the engineers’ time also needs to be reviewed. The information being used shows all time being equal, but the concept of context switching is ignored. Machines can switch jobs back and forth, but humans cannot easily do this. Application of the PEAK model reveals the flawed assumptions associated with this estimation decision:
(P) Problem: Projects are late because engineers are overloaded.
(E) Experience: The company has many projects all in different phases of development.
(A) Assumptions: Engineers can support more than one project without much impact.
(K) Knowledge: Senior engineers will most quickly resolve problems with products in the field.
Solution: Split the engineers’ time as needed to solve problems.
Assumed Risk: Junior and senior engineers may not actually get the same amount of time on a project, and therefore using senior engineers to solve field problems may be an issue.
The PEAK model helps you evaluate the estimation decisions that were made and look for where problems with the inputs can contribute to issues with the solutions.