Are estimates worthless?

If you take a picture while on vacation, is that picture yours? You are probably thinking: “What a stupid question. Of course it is mine!” But what if a stranger had asked you to take a picture of her with her camera? Is the picture still yours?

There are two lessons in this. The obvious one is that context is incredibly important in determining what is right and wrong. The more subtle lesson is that we have a tendency to underestimate the importance of context that we are not aware of. When you hear someone making a categorical statement that you know is wrong they are often victims of this tendency. Being “experienced” is in essence being knowledgeable of many contexts.

More and more people are coming to the conclusion that estimation is unproductive and in any case so error prone that it has no real value. “Why not use the time spent on estimation on more productive tasks?” they ask. In lean startup circles few people would question this approach. Time is of the essence and it is much better to focus on getting real feedback by building a minimal product than trying to guess how long it will take to build things. Of course, there is some implicit estimation going on. When figuring out which features to build first you need to weigh expected value against probable cost.

In the corporate world things are pretty different. Even if you can get someone to accept that estimates by themselves do not create value, few people will agree with the proposition that we should stop estimating before we build. “Ah, but they are just behind the times.”, you say. Or could it be that their context is different than yours?

If you read about innovation you will invariably come across descriptions of how hard it is to get new ideas accepted in a running business. This is commonly ascribed to either company culture or managers fear of risk. Once you get involved with a company that is struggling to innovate you realize that the real problem is something else: most significant changes to a business will also affect the suppliers and customers of the business. If these outside parties do not accept the change, it will fail. From this perspective it becomes much easier to understand why many businesses are reluctant to change. They can manage the own internals but they often have little power to decide on how their suppliers or customers operate.

By now you are probably wondering where I am going with all this. What does innovation have to do with estimation? It turns out that estimation has some interesting similarities to innovation. If you visit a software house that has been around for more than a few years you will probably find that estimates are an integral part of the sales process. Customers ask for features and get (binding) estimates back. Once an estimate is accepted the software house builds and delivers it. Does this process work? Just barely. Customers typically feel that they are being fleeced and the software house struggles with profitability. But it works well enough to make it difficult to convince either party that change is necessary.

So, the core problem in getting these software houses to accept that time spent on creating estimates is a form of waste is in explaining how they can get their customers to accept this. When they read exuberant blogs of lean startups that say you should stop estimating they think: “Sure, who needs customers?”. Their context is so different from the context of a startup that the advice sounds naive.

The only way to get someone in this context to embrace the idea of not estimating is to show them an alternative way of working with their customers that these customers will embrace. The irony is that since processes based on estimates often work poorly there is a lack of trust between customer and supplier. This makes it harder to convince the customer that an alternative process without estimates would be better.

I think that estimates are to a large degree a waste of time. At the same time I accept the fact that getting rid of estimates can be extremely hard, depending on your context. One strategy I have used when estimates are part of the sales process is to start with the small things. Many customers are willing to try out a system where all changes that can be performed in under a week are done without a prior estimate. The risk to the supplier is also limited here. If a developer spends a few days on a problem and then realizes that it will take more than a week the potential loss to the supplier is acceptable as long as this does not happen too often.

(Here is some more info about Core group in english)

6 kommentarer på “Are estimates worthless?
  1. Magne Jørgensen sier:

    Sorry, that this response got a little too long. Got carried away a little …

    I support some of Niklas’ opinions. If you don’t have to estimate for a good reason, don’t do it. There are studies suggesting that incorrect estimates may lead to less efficient behavior than no estimates at all and studies suggesting that very early estimates based on very little information can be very dangerous for the success of the project. The problem with Niklas’ position, as I see it, is that it does not reflect sufficiently on the context of why we estimate. The effort estimates clearly has little or no value in themselves, the value is a consequence of their use in planning, budgets, cost-benefit analyses, bids etc. If a task has to be done and it is not essential to know when the developer(s) will be ready for other assignments, an estimate of effort and duration may not be needed. If, on the other hand, the client needs to know when certain functionality will be completed, decide whether the whole system or some functionality is worthwhile to develop, there is a need for coordination of resources and there is enough information to estimate meaningfully, it seems to be very rational to estimate the effort needed as input to such processes. I guess that my position here is not very controversial (these processes are part of nearly all engineering disciplines/projects I know about) and suspect that the disagreement about the value of estimating must be based on something else. Niklas also seems to acknowledge the value of at least rough estimation for cost-benefit analyses, which he for some reason terms “implicit estimation”.

    So, what is Niklas argumentation that leads to his statement that “estimates are to a large degree a waste of time” and “…. create no value”? (A statement I find lacking a context and for this reason difficult to argue against – is his context, software development in general, further development of existing solutions, building of new, large, complex products ….?). Let’s list what I found of assertions/attempted support related to lack of value of effort estimates:
    1) “More and more people are coming to the conclusion that estimation is unproductive and in any case so error prone that it has no real value.”
    Response: Where is the evidence for this claim? Do you think what they believe is true? The average estimation error is about 30% (according to international surveys). This clearly makes it possible to make meaningful decisions and plans based on estimates in most cases. Even if the claim was true, an increase in belief (from very few to a few more?) in something cannot be considered as evidence against estimates. This is not a matter of popularity.

    2) “Time is of the essence and it is much better to focus on getting real feedback by building a minimal product than trying to guess how long it will take to build things.”
    Response: This may look like an argument against estimation, but is mainly an argument in favor of incremental development or prototyping for planning (and other) purposes. Estimates and plans based on feedback from building a minimal feedback are not surprisingly much better than estimates not based on such feedback. The comparison, implicitly assuming that we should do either A or B not both, is invalid in case that we need an early indication of cost or some other rational purpose requiring effort estimates. But, as mentioned before, if you don’t need an estimate, do not estimate (trivial statement, but not always followed). Niklas’ statement about “it is better to” lacks both empirical evidence and context, as far as I can see. I do, however, not doubt that Niklas has experienced contexts where «better to» is likely to be true (although I doubt that the experience is very robust, since only one alternative is chosen and the effect of the other is hypothetical).

    3) “But it [the fixed price paradigm that requires early estimates] works well enough to make it difficult to convince either party that change is necessary.”
    Response: I clearly see the problems related to selection of providers based on their bid (Research finding: The less competence, the lower the bid) and not so much on their competence. The main problem with the argumentation is that even when a client does not require a fixed price, he will usually need an idea of the cost involved, which also require an estimate. Niklas makes it look like, as I read him, the client’s wish for an estimate vanishes with the removal of fixed price. This is of course not true. Clients live in a context where it is rational to have budgets, make cost-benefit analyses etc. Working incrementally without fixed price means that these budgets play another role than in fixed price paradigm, not that there is no purpose in estimating.

    A strange observation is that Niklas’ way to avoid estimates, i.e., “One strategy I have used when estimates are part of the sales process is to start with the small things. Many customers are willing to try out a system where all changes that can be performed in under a week are done without a prior estimate.”, involves estimating. How could he otherwise know whether a task could be performed in under a week? This indicates that Niklas has a quite narrow understanding/interpretation of what estimation is in his argumentation against it.

    What does Niklas really mean about estimation? It is not at all clear to me after reading his essay. I think the topic deserves more preciseness (what kind of estimates and estimation processes are we talking about) and context to enable meaningful discussion. In particular, is it the fixed price project based on no experience with construction of the system that he argues against (which would be an interesting discussion, where trust, ability to document competence and productivity, how different contract models leads to shared responsibility and better interaction with the client etc. are all factors involved)? Is it something else? From Niklas’ writing about “implicit estimation” and suggested use of categories of tasks based on estimated effort, it seems to be that he rather advocates estimation as input in the communication with the client, than warn against it.

    In short, I find it difficult to follow Niklas argumentation and failed to find evidence against the benefits of estimates in general. There are clearly contexts where estimates are not needed, where they are waste of time (even dangerous) and fixed price projects are in many contexts not recommended. It might be that Niklas’ concerns reflect his experience in his context, and that I fail to understand his context well enough. As an outside reader (asked to respond on this ;-)), it seems however that his understanding of estimation and its purposes is too narrow to support his rather general claims (or beliefs) about effort estimation.

  2. I find little value in estimates at levels smaller than approximately three months, but mostly because I have spent a lot of time working with bigger, Corporate(TM) clients who have bigger problems to solve than to worry about the difference between a task taking 4 days and 5 days.

    As part of taking a more positive approach to this issue, I try to direct clients towards focusing on exploring the product more effectively, driving down the cost of mistakes, and expanding their value of «the system». The more my clients do these things, the more value they find in other issues, and the less time they spend fixating on the accuracy of their low-level estimates.

  3. Phew! I think your reply is longer than my post…

    I don´t think I will try to comment on all the point you make, I’ll stick to the most important things. First of all, I would like to clarify a few things:

    The point of my blog was not to prove anything in an academic sense, it was meant to provoke the reader into thinking in new ways. The IT-community seems to be divided into two camps, one views the abolition of estimates as a “no brainer”, the other sees formal estimates as essential to doing business.

    The main focus of my post is not really about the merits of estimation in itself, it is more about the way many people discuss estimation without being explicit enough about context. (It is slightly ironic that I did not make my context clear enough while discussing the importance of context here.)

    When I say estimate, I mean a formal and in some way binding document that describes a problem to be solved and the cost of doing this. I was primarily thinking about estimates as they are used in many businesses as an integral part of a purchasing process with their customers. When a developer makes a quick guess at how long something will take it is an “estimate” but the cost of producing it is negligible. In many software houses I have worked with developers will spend up to 20% of their time creating formal estimates.

    I feel you miss an important point when you criticize the scheme I propose at the end.The developer only needs to make a quick initial guess at the size of a task. If this points to the task being smaller than the threshold work can commence without a formal estimate. If the developer realizes after a couple of days that the task is larger than expected, work is halted and a formal estimate is created. If the customer opts to not go ahead with the task the work that has already been done is “lost”. In practice this will not happen often. The point of doing things in this way is that less time is wasted on formal estimation.

    Another thing I do not agree with is that estimates are necessary for a customer’s budgeting process. In advertising it is much more common to use retainers; the customer pays a fixed fee per month and this is used on whatever is deemed as most important.

  4. Truls Unholt Truls Unholt sier:

    First of all, I like the introduction of the term «context» into this context :-). However, I am not sure that it is developed far enough. Let me take the last point – the capital budgeting process – as an example. The capital budgeting (often lumped together with financing in a software project context) focuses on the needs of the investor(s), not the software development team (which in this context hardly ever is considered). Every single investor I have ever met have wanted to know the cost side of the project. This provides the context for software development, whether developers like it or not and whether it facilitates optimal software development processes or not. Moreover, I do not think that leaning on the business model is a viable strategy since the business model is defined in a business context, not a software development context, and are not (and should not be) driven by developers needs.

  5. Magne Jørgensen sier:

    OK, so the question is not «Are estimates worthless?», but rather how to optimize the investments we make in effort estimation work, or perhaps how to avoid wasting time on writing committing documents on price and solution (bids). I have experienced in many companies that the client’s trust in competence and efficiency of the provider is the key factor for the last question. The more trust, the larger proportion of effort is spent on the development work and the less on costly price-related documents there to make the client feel safer in his investment.

    Assume that the client has a set of people (or companies) as candidates for conducting a job. If he knows that the person (or company) selected, preferably based on competence, is very competent and motivated AND that the effort threshold for negative cost-benefit of the job is very unlikely to be reached, he would in many cases not (at least rationally speaking) need more than a very rough estimate of the cost. The problem is how he should know this. Reading CVs or looking up reference clients do not typically give enough confidence for the desired level of trust in competence. I suspect that the clients wish for committing prices and documents in many ways reflects that they lack the trust/documentation of competence and efficiency of the provider/developers. The question of not wasting time on fixed price estimation work is then related to the question of how should software companies document their competence and productivity? If they were better at it than currently is the case, many problems would be solved, I think.

  6. We are still focused on different contexts. I was describing the context where a supplier and customer have a contract – a common case is that the customer has bought a software system from the supplier – and the customer regularly wants changes and enhancements. In this case I would argue that it is relevant to discuss the possibility of getting rid of estimates. There will always be some kind of estimation but if estimation work is reduced from 20% of capacity to nearly zero I think it is correct to say that we have «stopped» estimating

2 Pings og/Tilbakesporinger for "Are estimates worthless?"
  1. [...] Are estimates worthless?& Magne’s response – interesting discussion of the value and cost of estimation and its role in contracting w.r.t. trust – a nice addition to the discussion: Agile not suitable for governmental IT?. [...]

  2. [...] planning is guessing. My experience leads me to conclude that estimates are essentially worthless when employing a highly iterative development process. If one accepts that estimating a task does [...]