Secret agentic AI by Mark Seemann
A scenario.
Here's another speculative scenario about the future of software in an increasingly hostile international environment.
It seems indisputable that the future of software development involves substantial use of LLMs. Some people experiment with vibe coding, but I find it more plausible that we'll see widespread use of LLM-based agents that produce code, with continuous, but superficial human supervision.
Can we trust those LLMs?
Economics #
In other articles, I've discussed whether LLMs deserve our trust. The perspective in these articles have mostly been on the inherent non-determinism of these models, as well as their lack of 'understanding' of what it is that they do. The point I will make here, on the other hand, does not depend on assumptions of that kind. You may, if you will, imagine a future in which LLMs are far more reliable than today.
Even so, there's a fundamental aspect of the ecosystem that most developer-cum-futurists seem to ignore. The current systems, operated by companies like OpenAI and Anthropic, run with colossal deficits. While their valuations are measured in billions of dollars, they are losing money every year.
This implies that the way agentic software development works today isn't representative of future directions. But even if today's systems are too inexpensive to last, my social media feed regularly showcases examples of what can best be termed token angst: the dread of running out of tokens.
Investors ultimately want return on investment. While the AI companies currently run on deficits, they can't keep doing that. Sooner or later, they'll either turn a profit or go out of business. How will they generate profit?
There's been some talk of 'showing ads' in chatbots, but that isn't compatible with agent mode. If no-one is looking at the output of an LLM, then ads are an unlikely revenue stream. Perhaps my imagination is poor, but the most realistic scenario is coding agents as paid services. You buy a subscription with a token budget, or alternatively, you pay for tokens as you use them.
This suggests that in the future, price is going to be a competitive factor. You may decide to use Company X for agentic coding, rather than Company Y, because Company X is less expensive.
Subsidized LLM services #
For practical purposes, today's AI companies are American. It'd be naive to think that it will stay that way. When a technology becomes sufficiently strategically important, other states subsidize national enterprises to catch up. To Silicon Valley ears, this may sound derivative and unfit for competition, but such a strategy can work. Historical evidence exists.
You can find examples in another capital-intensive industry, aviation. Airbus probably wouldn't exist without European governments taking an active interest. And I find it fair to argue that Airbus is currently doing better than their main competitor in civil aviation.
Other aviation-related examples may be found with certain airlines that operate out of resource-rich countries. Air fare is surprisingly cheap, while service is top-notch.
In other industries, Chinese electric cars are, too, notably less expensive than competitors. Even if you're a dyed-in-the-wool liberal capitalist believing in competition and free markets, current world trends are moving away from that. We see increasing protectionism and focus on reshoring strategic manufacturing capabilities.
Subsidized AI companies will exist, too.
Secret agents #
Most of the modern information infrastructure has turned out to be porous. Even though top companies are American, adversarial state actors use social-media platforms to wage information warfare. I'd be surprised if something similar doesn't happen to LLM systems.
Granted, some clandestine way of injecting data directly into American LLMs seems difficult. What state actors can do, however, is to offer alternative, subsidized systems. Use this new LLM-based coding service: It's much cheaper than the one you currently use!
You may counter that you'd never use a Chinese, Russian, or pick-your-own-enemy LLM system. But some people and organizations are more price-sensitive than security-conscious. Besides, a dismissal of this scenario assumes that ownership is transparent.
An adversarial state actor could set up a shell company in your country, while keeping technical control. Much evidence seems to indicate that a major video-reel-serving app company already works that way, and although that example is not exclusively related to generative AI, I'm sure that you can extrapolate.
Once a foreign state establishes such a clandestine beachhead, it can use it in several ways. It can passively spy on users' code bases (and other content), and it can actively inject backdoors, viruses, etc. into customers' code bases. If that sounds far-fetched, you may not have heard about the xz utils backdoor. Such attempts are already being made.
Untrustworthy systems #
What can you do to avoid this risk?
You may consider only using systems of known origin. You may decide to stick to OpenAI, Anthropic, or other American companies. Perhaps, but I think that you should consider at least two things. The first is that, as already covered, these companies run huge deficits. Where do the money come from? Investors, you say? Indeed, but which investors? Is it conceivable that some of the investors are already, through chains of shell companies, controlled by foreign governments? And if not now, then in the future?
The second consideration is whether you should consider a US-controlled company benign or hostile. If you work in the US, then you probably consider the US government to be the good guys. Over here in Europe, we used to think that, too, but currently, we are not so sure.
You may, then, think of another countermeasure: Buy hardware powerful enough for in-house LLM hosting. This might be a viable option for some organisations, particularly because continued advancements in both hardware and software will make this increasingly feasible. Still, where are you going to get the LLM?
A few organizations are big enough that they may be able to train their own LLM, but most are not. They'll need to use a copy of an existing LLM. In that case, we're back to the issue of trust. Where do you get the LLM, and why do you trust it?
To be clear, you can't 'review the LLM code'. It's a system trained on massive data sets, the result of which is billions or trillions of weights (i.e. numbers). All you can do is trust it. That said, since training of these systems is a non-deterministic process, you could argue that adversaries can't train malign behaviour into them either. If systems come with spyware, it's more likely that it's embedded in the code that surrounds the model itself. Thus, you could, theoretically, review that part of the system. You'll probably not get the source code, however, so your review would have to be of the machine code. I don't find that realistic.
Countermeasures #
What can you do to protect yourself against such threats?
Some of these are more problematic than others. When you give a third party access to your source code, there's ultimately no way to detect if the third party takes a copy of all the code. Pragmatically, you either have to trust that third party enough, or else behave as though you had already made your code base open source (which there are other good reasons to do, in many contexts).
It's possible that research into zero-knowledge proofs could help address such issues, but this is not something I know much about, so perhaps I only display my ignorance.
The other large problem is whether adversarial LLMs may inject malicious code into your code base, installing backdoors, spyware, or similar. This threat seems more tractable: Treat your code base as if it was open source, and each contribution as a drive-by pull request that requires rigorous scrutiny. We're back to the discussion about trust in software development. As I've previously discussed, you can't even trust yourself to write code without errors, perhaps you can't trust your colleagues either, and I don't think it's wise to trust LLMs to write code.
Fortunately, we have decades of experience in making software development a safe and manageable process. Parts of Code That Fits in Your Head are about working together as a team, and how to ensure that you ship quality software. Treat LLMs as randos on the internet submitting pull requests to your open-source project, and review accordingly.
"But Mark, the LLMs produce code much faster than we can review it." Yes, I know. You are the bottleneck. You and your brain. It was always like this. Typing was never the bottleneck, although it's embarrassing so quickly the industry seems to forget that.
Conclusion #
The economics of AI companies in early 2026 seem unsustainable. In the future, these services will become more expensive; perhaps to a degree where price becomes a competitive advantage. If that happens, adversarial state actors might set up and subsidize shell companies that run LLM-based coding services at more attractive prices than competitors.
Once such companies have customers of interest, they can spy on the source code, and plant backdoors and spyware into code bases.
This is just one of many reasons to be wary of AI-generated source code.
