Releasing AI-generated software by Mark Seemann
On software engineering, epistemology, and liability when code is generated by LLMs.
Robert C. Martin recently generated much interest with a tweet:
"My current strategy is to not read any of the code written by my agents. That's the only way I can take advantage of their productivity. What I do instead is to surround the agents with extreme constraints. Unit tests, gherkin tests, QA procedures, quality metrics, mutation testing, test coverage, and a plethora of others. In the end, I have very high confidence in the code they produce because they've had to run the gauntlet of all of my constraints and tests."
The author of Clean Code no longer reads code? What implications for software engineering does this suggest?
To be clear, how thought leaders interact with LLMs change so rapidly these days that what someone wrote months ago may no longer apply. And, indeed, Robert C. Martin recently walked back on his harness use.
The point of this article, however, is not to perform an exegesis on Robert C. Martin tweets, but rather to ponder the implications for software engineering.
Looking at the code was never foolproof #
My first reaction to the notion of not even looking at LLM-generated code was: If you're no longer reading the code, then how do you know whether it's correct?
And a second later, I realized that that's just a knee-jerk reaction from an old programmer. I know full well that looking at the code was never foolproof.
All programmers make mistakes. We write code thinking it does one thing, but it turns out that it works slightly differently. It may exhibit liminal behaviour where it throws null-reference exceptions, has off-by-one errors, perform overlooked state mutation, or is otherwise just plain wrong.
Some research seems to indicate that code reviews are one of the more effective methods for catching bugs (see e.g. Code Complete, 2nd ed, chapter 21). Even so, these techniques don't detect all defects. Some errors slip through, even when subjected to critical and systematic inspection.
In other words, humans looking at code is no foolproof software-engineering technique.
A new notion of software engineering? #
Dave Farley's book Modern Software Engineering presents a view of software engineering congruent with my own. Ultimately, software engineering boils down to rapid feedback on falsifiable predictions about the software.
Even if we give up the notion of writing and reviewing source code, there are checks we can perform. Some people experiment with 'teams' of agents, each agent having a particular role: You could have one agent write automated tests, another performing code reviews, etc. While I have previously been sceptical of such seemingly infinite regress, I have recently come to the understanding that our previous, human-centred practices, aren't foolproof either.
Even though I believe that test-driven development (TDD) is the most scientific approach to software development that I know of, you can still make mistakes. The most common sort of mistake in TDD is the test(s) that you don't write. Missing test cases leaves the System Under Test underspecified, but you may not even realize it.
LLMs seem to produce code, even automated test code, using other heuristics. People who use 'teams' of agents report good results.
Ultimately, demanding perfection from LLM-based software engineering sets a bar we human programmers never cleared. Rather, we seem to arrive at the same argument that you often encounter related to self-driving cars: The system doesn't have to be perfect: It just has to be better than the human-centred system.
My claim is not that we have already arrived at that state. The point is rather that there seems to be no inherent barrier to a fully LLM-driven process of software engineering.
Where does liability fall? #
You may, then, imagine a fully LLM-driven software release process. Perhaps humans are involved, but not as programmers, and not as code reviewers. Imagine that no-one looks at the code, but rather relies on other quality-assurance techniques to decide whether or not to deploy. Imagine, even, for the sake of argument, that the frequency of defects is lower than any comparable human-driven process.
Lower. Not zero.
Errors still make it to production. Most of these will probably be of a benign nature: A button's alignment is off. A text is shown in the wrong language. A message arrives twice, or out of order. Perhaps a bit more serious than that: A shopping-basked discount is applied incorrectly. The wrong song plays on your phone. A traffic signal is stuck at red.
Benign errors are, in this rough categorization, errors that are somehow rectifiable. Once you've discovered it, you can fix the problem. Perhaps you'll have to apologize to people, or if the discount was miscalculated to the seller's advantage, you may offer the user some kind of compensation.
Other kinds of errors will, however, be more serious: You lose a million-dollar spacecraft. Your car accelerates when you step on the brakes. You receive a deathly radiation overdose because of a race condition.
Such errors don't even have to be that bad. It could 'only' be that people miss their flight, are denied the hotel room they already reserved, receive incorrect tax bills, or lose their insurance coverage.
When software defects are serious enough, there could be repercussions. Where does the blame land?
Consequences #
Consider, at first, bugs that 'merely' cost its owners a lot of money. The Mars Climate Obiter fiasco falls in that category. No laws where broken. The only problem was that 327.6 million (taxpayer) dollars were wasted.
It would be an extraordinary software organization if such an error didn't have consequences for someone. When people were still writing code by typing it into the computer, it was easy to point and exclaim: Dave wrote that code.
Not that such simple ways of assigning blame are fair, or even productive, but they do happen. Someone suffers the consequences, even if he or she is only a scapegoat.
As long as the problem is mostly internal to the organization, if you're the scapegoat, the worst that may happen is that you're fired.
If, on the other hand, the error causes a legal liability, then the software organization responsible may have to go to court. As we have seen in e.g. Dieselgate, individual software developers may have to take the fall. Again, whether this is fair, or reflects the actual reality of what happened, is not my point. The point, rather, is that judicial branches around the world are likely to have a view on where responsibility goes that is decoupled from how software is actually produced.
- But, your honour, the LLM wrote the bug.
- So you say, son, but you signed off on the release, didn't you?
- Yes...
- I can't send an LLM to prison, but I can sentence you.
Any software organization that considers going 'full AI' need to think hard about what that means when it comes to liability.
People will play the odds #
Let's consider a recent development in a different field to illustrate what I think will happen: In June 2026, the Danish traffic authority, Færdselsstyrelsen, approved Tesla FSD (self-drive), with the proviso that the human behind the wheel is ultimately responsible for the car. Drivers are expected to pay attention to traffic, and take the wheel in case a dangerous situation threatens.
What I think is going to happen is that drivers will become increasingly inured by the effectiveness of the system. The better it performs, the more drivers will zone out. After a few weeks or months, drivers will be on their phones during most of the ride, paying little to no attention to traffic. It will mostly go well.
Then, once in a blue moon, an accident will happen. As Danish law currently works, if the accident involves loss of human life, the driver faces a sentence for involuntary manslaughter. This usually means prison time.
It's not going to matter if the 'actual' cause of the accident was a software error. Legally, the responsibility lies elsewhere. Often, the underlying reason is that legislation lags behind technological reality.
I think that the same will happen with LLM-driven software engineering. People will play the odds because they look so compelling.
How much you invest in safeguards and various quality-assurance technologies may not matter. You could establish the most elaborate system of verification and still be liable for any errors. How do you feel about releasing LLM-generated software? Are you ready to go to prison for an AI?
Conclusion #
Software engineering may not be what we thought it was. Or, rather, it may not be limited to a particular set of methodologies centred around human programmers and testers.
It's conceivable that the future holds fully automated software-engineering processes that are at least as safe as those we currently know. Even so, the human notion of liability and responsibility may not keep up.
When things go wrong, will organizations write off the error as nobody's fault? Will courts of justice consider serious defects as essentially another accident? Or will someone have to take the blame? Who will it be, if no-one wrote any code?
