Why is writing software so hard?

Tom Breur

24 December 2017

“Software is eating the world”, people say. If my sources are correct, it was Marc Andreessen who said this first in 2011, and with that statement he directed attention to the ever-growing importance of software and technology in the economy. Increasingly, businesses are being run on software and delivered as services, as opposed to the traditional attention for tangible goods. This even holds for industries that have a significant material component like agriculture, or the defense industry. In that light, Andreessen argues that software and technology companies are being undervalued by the stock market.

But as the infamous Standish Group Chaos reports point out, time and time again, gigantic losses are being reported. Every year we see that for the larger software projects, so much bigger disproportionately (!), their reported losses are. So how do we prevent this death spiral from running amok? And what makes writing software so incredibly hard?!?

When a sculpture artist sets out on a project he “sees” the fruits of his labor turn into an object. Writing software is different, yet software engineers create something, too. Based on imperfect specifications, and input from partially informed business experts, he (or she!) tries to create a text file that behaves in ways that meet their creators’ needs. In some odd way, that software simultaneously doesn’t do what it is supposed to do, precisely because it does exactly what it has been told to do…

As you craft a software solution, it doesn’t “break” like a sculpture does. In fact, the cracks are often largely invisible! Software tends to break far out in the future, when a myriad of edge cases lead to outcomes that were maybe never specified, but that unmistakably diverge from business needs. And when it breaks, paradoxically, the software may well be running fine. Like a rogue stock trading algorithm making irrational trades, albeit strictly following instructions. How can this be?

Sometimes, the developer –under time pressure– lacked the time required to think through possible future states that might occur. Then “common” issues like a null-pointer or buffer overrun may crash the system. But at a slightly higher level of abstraction advanced software behaves like a complex system (in the Cynefin sense), and if you buy into that notion it becomes apparent why nobody could ever anticipate all future states – not even in an ideal world. This may surface as a lack of imagination by the developer to understand what the system is supposed to do, but that would be an unreasonable and limiting perspective on this dynamic.

What makes large software projects especially daunting, much like the construction of Egyptian pyramids must have felt like, is business demands call for efforts at a scale and scope that (far) exceeds what a human brain is able to grasp, intellectually. Our minds cannot comprehend these solutions holistically, so we work at small parts at a time, in the hope that the total solution will behave as planned.

Every piece of software in production is “alive”: the sheer fact that it is being used, is guaranteed to trigger new demands. Software maintenance never stops until code gets decommissioned… Every time the solution behaves in undesirable ways, we fix that little defect, but can we realistically expect the engineer to master the solution as a whole? Not really. The complexity is mostly hidden, as it should, or else we wouldn’t be able to build the intricate machines that have come to rule the world…