After Forty Years, Still No Silver Bullet
As always, technology can help with production. What’s scarce is orientation.
Forty years ago, computer scientist Fred Brooks published a paper called No Silver Bullet: Essence and Accident in Software Engineering. As its title implies, the paper argues there are no technological shortcuts to making software radically easier, simpler, or more reliable. You may think AI is the ultimate silver bullet. It isn’t.
Moore’s law was in full force in 1986. Hardware was getting more powerful, faster, and cheaper. Surely, some technology would come along to do the same for software. Brooks argued this wasn’t in the cards, since software is fundamentally different from hardware. For one thing, it’s of a different order:
The essence of a software entity is a construct of interlocking concepts: data sets, relationships among data items, algorithms, and invocations of functions. This essence is abstract, in that the conceptual construct is the same under many different representations. It is nonetheless highly precise and richly detailed.
Specifying, designing, and testing this construct is difficult. The challenge isn’t implementation but design: “We still make syntax errors, to be sure; but they are fuzz compared to the conceptual errors in most systems.”
Technical advances usually make development easier. No Silver Bullet traces the history of time sharing, unified programming environments, and high-level languages. Object-oriented programming was a promising new technology at the time and there were even rudimentary AIs in the form of expert systems. Brooks examines them and concludes they’re not enough.
Why? Because coding isn’t the hardest part of making software. Instead, the hard part is knowing what to build:
The hardest single part of building a software system is deciding precisely what to build. No other part of the conceptual work is so difficult as establishing the detailed requirements, including all the interfaces to people, to machines, and to other software systems. No other part of the work so cripples the resulting system if done wrong. No other part is more difficult to rectify later.
What will the system do? How will it serve strategic objectives? How will it enable better judgment and allow people to derive meaning from data? These aren’t implementation questions, they’re design questions. Somebody must define the “construct of interlocking concepts” that define the system, aiming for good fit between the system and the context it serves. LLMs can help, but they can’t replace human understanding and judgment, at least not yet.
Brooks calls out four inherent properties of modern software systems:
Complexity: Software systems are among the most complex human constructs. They’ve only gotten more so as computers and operating systems have grown more powerful and capable.
Conformity: Software solutions must conform to the goals, needs, constraints, and interfaces of the organizations that bring them forth. This is true whether it’s bought off-the-shelf or developed bespoke.
Changeability: Anything that lasts does so because it’s able to adapt to changing conditions. Software is inherently more malleable than other complex designed systems, such as buildings.
Invisibility: Whereas complex physical systems (again, think of buildings) can be represented with mechanical drawings, software specs are inherently abstract. This makes them hard to design.
There’s been progress in the last four decades, but these properties remain fixed. LLMs haven’t changed that. Non-deterministic components add immense complexity and unpredictability to software systems. The ease, speed, and volume of code generation make software more malleable and opaque than ever. And LLMs promise to ease bespoke development, tempting orgs away from one-size-fits-all solutions.
Which is to say, LLMs haven’t changed the nature of software. Instead, they’ve made it more so. So far, the technology’s killer application is developing software: teams can now produce more software, faster. (It’s unclear yet whether it’ll ultimately be cheaper, especially when you consider maintenance costs.)
What LLMs haven’t done yet is replace software wholesale, at least not for tasks that require predictable behavior. And as their true costs and constraints become evident, it’s increasingly doubtful they will. Instead, LLMs will likely become part of systems that include traditional deterministic components — both inside the systems and as part of the development process.
The resulting systems will be more complex, malleable, and abstract than prior ones. They’ll also be better fit to purpose than off-the-shelf solutions. But that requires design, which remains primarily a human challenge. And it’s hard:
it is really impossible for clients, even those working with software engineers, to specify completely, precisely, and correctly the exact requirements of a modern software product before having built and tried some versions of the product they are specifying.
Replace “software engineers” with LLMs, and this sentence still stands. But it also hints at where LLMs come closest to being a silver bullet: in their ability to spin up rapid prototypes. Good software is grown, not built. That is, it evolves from an initial core to a more complex system through an organic approach that respects Gall’s law:
The building metaphor has outlived its usefulness. It is time to change again. If, as I believe, the conceptual structures we construct today are too complicated to be accurately specified in advance, and too complex to be built faultlessly, then we must take a radically different approach.
Let us turn to nature and study complexity in living things, instead of just the dead works of man. Here we find constructs whose complexities thrill us with awe. The brain alone is intricate beyond mapping, powerful beyond imitation, rich in diversity, self-protecting, and self-renewing. The secret is that it is grown, not built.
So it must be with our software systems.
What was true then is true now: technology moves the bottleneck from production to orientation. LLMs make coding easier, much like high-level languages, IDEs, and compilers did in the past. But without shared models, structured context, feedback loops, governance, and clear interfaces, they won’t provide the results leaders expect.
As always, how to build gets easier — knowing what to build doesn’t. AI can help with that too — but it needs steering. The question isn’t “Which systems can we replace with AI?” Rather, it’s “How can AI help us grow systems that better fit our needs?” The answer will consider AI as a system component and a production tool. But forty years on, there’s still no silver bullet — just better ways to find good fit, faster.