David Buxton

So what’s the bet you’re actually making with Harriet?

He started with a principle. “Whenever you’re doing anything these days, you’re betting that you can do something the foundation models can’t. And as the foundation models’ capabilities have moved, that bet has moved a bit. My expectations were maybe a bit too conservative.”

He is genuinely surprised by the pace and admits, with a grin, that he finds it almost annoying. “macOS, once a quarter push out eight new emojis. And then Claude, five times a day they’re pushing out really nice, meaningful features.”

The fancy way to say it, he warned me, is governance architecture. In simpler words, it is “A better control plane, so that companies can deploy AI in a way that’s safer and has a better user experience. Not for the first tier of AI power users, but to enable that tier to bottle what they’re doing really well, the processes they’ve automated, the apps they’ve built. You take that and scale it at organisation level, so you bring everyone up to the level of your best people.”

What do you think about the popularity that Anthropic has gained?

I put the popular narrative to him, that everyone loves Claude and Anthropic is the new dominator, but he picked it apart before answering. “It depends what you mean by Claude. It’s a family of models, and some of them are very, very good. It’s also a family of apps.” He likes the harnesses, the desktop app especially, which he thinks has become the anchor for how non-technical power users think about AI on the ground.

But he thinks people are making a mistake. “They’re bundling together the models and the harnesses themselves. If they were more enlightened about this, they’d be thinking about how to unbundle the models. It’s really not obvious that the Claude cost-benefit ratio is the right one for most organisations.”

He pointed out, with some relish, that you can now run the desktop harness with models that are much faster and, in most cases, as good or better. “It’s a real pain to work out the configuration, but it is actually possible. Ten times as fast and a tenth of the price, which is really nice from a user experience point of view.”

If there’s no moat, where does the value go?

This is where the scepticism comes out fully, and he is the first to say it about his own industry. “I’ve been saying for a while that these businesses are not particularly defensible. At every point I’m just run over by the speed of growth of the businesses themselves. So I don’t think I’m the right person to listen to on this.”

Then he talked about how the big names in AI are making money nowadays. “It’s very difficult to see how the frontier companies make a great business selling tokens. Honestly, I think it’s AGI or bust for a lot of these big model companies. It’s very hard to see how they repay the massive investments they’re making in data centres otherwise.”

His reasoning is structural, and it’s the same reasoning behind Harriet. He thinks defensibility was never one big idea. “What I learned at my first company is that defensibility comes from lots of little fiddly things, rather than from one big idea.” He even admits this is the place he’s been most wrong. “I’ve been very consistently wrong about how much appetite there is to bet on the labs. I just assumed there’d be some natural slowdown in how fast they could move on product. I didn’t expect them to keep moving like that.”

Should we be scared of the Chinese models?

There’s a lot of fear in this conversation generally, so I asked him directly. He was relaxed about it, with caveats. “The cost-benefit ratio is very good. They’re very high quality and very low cost.” He wouldn’t connect to hosted services run by Chinese companies right now, for security and privacy reasons, and he’d be more careful with anything that touches people or has to be compliant with the EU AI rules. “But if you’re doing programming with this stuff, you can be pretty confident it’s going to perform as well or better than the hosted American versions.”

What he’s actually excited about is the user experience, and here he made the case for why this matters more for normal knowledge workers than for engineers. “The cost differential is just staggering. Similar performance to the frontier models for something like 3 to 4 percent of the cost.” Opus is good, he says, but it’s slow, and Anthropic’s answer to that is to tell you to set off twenty things overnight. “I think that messaging is essentially working around a flaw in their own product. For engineers, fine, kick off twenty features and go to the pub. But most knowledge workers don’t have workflows like that. They want to iterate and tweak. So that user experience, the speed of iteration, is going to be particularly valuable for non-programming knowledge work.”

Can AI fix something as broken as the state?

I brought up the NHS, and the bureaucratic sludge that sits under so much of Europe. This is where he got most animated, and where the conversation stopped being about products. He reached for a book, James C. Scott’s Seeing Like a State, and for the Romans.

His argument is that government is a pyramid we basically inherited from the legions. “You have a pyramid where the bottom has to feed through several layers to get to the top, and each layer gives policymakers a layer of abstraction.” The cost of those layers is real. “A lot of politicians complain that when they get into government, the first thing they find out is that none of the levers they’re trying to pull are connected to anything.”

He sees a genuinely different opportunity here, separate from the automation story most companies are chasing. “There’s a world in which AI lets policymakers get information more or less straight from the edges of the organisation, in all the rich texture it needs.” We’ve been promised this before, with big data, and he knows it. “But those always relied on tabulation, on abstractions, on good decisions about how you gather the data in the first place. For the first time ever, I think we can almost connect directly through.”

He’s careful not to oversell it. There’s an optimist version and a pessimist version. “The pessimist view is that the whole state capability gets destroyed by a load of people who come in and try to blow everything up and promise you can do everything with a computer. I think there’s a very realistic case in which that happens.” But he doesn’t think the worst case is inevitable, and he suspects there’s more of a bottom-up opportunity than people assume, because even the people inside the system are frustrated that it doesn’t work for them either. His frustration is with the lack of ambition.

“At the moment, the main thing people in government are doing is writing press releases with AI. There’s a very significant lack of vision.”

What is everyone else getting wrong?

His answer was specific. “The main thing people are systematically underestimating is how much context is necessary for non-programming agents to operate effectively.”

People assume programming is the hard case, so if AI works there it must work everywhere. He thinks that’s backwards. “Programming is kind of easy when it comes to the structure of how you engineer it. You have tests you can deterministically run. There’s tons of training data in open source. And most of the context for a programming task exists in the repository you’re working against, which is usually available to everyone.” None of that is true for the rest of the economy, where context is scattered across systems and locked behind permissions.

That, for him, is the next shoe to drop. “There’s a massive amount of optimism around toy use cases where people haven’t thought through the contextual challenges. Bringing all that context together, and distributing actions outwards from it. That’s really the next thing to drop from an enterprise perspective.”

How do you feel about all this for your kids?

We ended somewhere quieter. I asked, half about AGI, half about physics, whether AI could get good enough to solve everything. He wouldn’t take the bait on AGI. “It almost starts to disappear as an objective the closer you get to it.” But the question about his children clearly sat with him.

He went back three hundred years, to Newton, who could be a polymath because the fields were shallow enough that one person could stand at the frontier of five or six at once. “He ran the Royal Mint. He made coins.” That’s gone. “Now you basically need to be fifty before you’ve got to the frontier of any given scientific field. So I worry about what it means for human achievement if any given human is basically never going to be at the front of any research field, and even in creative things it’s going to be very difficult for humans to distinguish themselves.”

He reached for photography, which didn’t kill painting, and the phonograph, which didn’t stop people performing music. “But they massively changed the value of those activities. They basically became luxury goods. To be able to perform music, or to be able to paint.”

There is something in it that scares him. “I do wonder a lot about what the world looks like when any sort of creative endeavour, potentially even scientific endeavour, is basically a luxury good. Something any computer can do better.” A pause. “Very dangerous.”