Group processes in software estimation

Tom Breur

12 March 2017

Agile practitioners resent BDUF: Big Design Up Front. The effort of estimation itself is considered waste, and moreover: this effort gets squarely in the way of actually delivering value early. However, there are many settings in which an estimate is expected. Personally, I prefer this to be a “lightweight” one, as I am apt to point out that spending more time estimating in no way guarantees that the accuracy of it will be any better. But alas, I am aware how counterintuitive that may sound, so I am always ready/willing/able to give up that fight, and choose another one.

Delivering software and then subsequently analyzing (in retrospect) how fast that went during the preceding iteration, is a much more valid way to arrive at an estimate for the bigger project. Once the delivery team have seen (measured) where time and delays went, they are much better equipped to “estimate” (data based) how much they can deliver in the next iteration (which Michael Mahlberg would probably consider analysis, rather than estimation). I resign to the fact that it is usually low trust that requires us to spend time estimating rather than delivering. But you can still try and take the working relationship forward from there.

Many aspects to software estimation have been researched and documented already. Mike Cohn’s Agile Estimating and Planning is one of the classics in this area. Confounding of “estimation” with “delivery commitment” is one of the hackneyed but pernicious causes of dysfunctional dynamics. Another one is group processes among team members that gather to arrive at a “sound” estimate. Two Norwegian researchers, Moløkken-Østvold and Jørgensen, have made a great contribution with original research on group processes. One of their findings hopefully sheds some light on the well-known bias to estimate overly optimistic. Maybe this has contributed to the old wisdom: “It always costs more, and takes longer than you thought.”

Besides strategies to negate bias, it is also clear from this research that estimating “unknowns” is intrinsically hard, and a basis for the optimistic bias. Doing “spikes” in Agile is meant to address some of this uncertainty. Rather than “guessing” how long it will take, and what you will encounter, you do some work, but expressly not with the purpose of creating the solution. Spikes solely serve the purpose of surfacing unknown aspects of the solution. It’s a rational compromise between working towards analysis (“retrospective” estimation), and doing more fact-based estimation.

Last, but certainly not least, there appears to be a management dynamic where project responsibility tends to get assigned to whomever can (dares to…) come up with the most optimistic estimate for completing a project. Needless to say, that obviously feeds into “optimism” of a special sort. Since after the fact it isn’t possible to compare between project leaders (only one got the job), this mechanism can go unchallenged for a rather long time. And for good, but terrible reasons: “heroism” is another well-known (but dysfunctional) pattern in software development. But don’t we all just love our heroes?