> All requests to an LLM are idempotent, for every API call you need to send it the entire conversation history

A more appropriate term is “stateless”. LLM responses are certainly not idempotent, as they are not even deterministic.

Google figured this out with their paper from 2023, We have no moat and neither does OpenAI. The moat now is the harness and being able to recursively self improve from RLHF, a great example is how Grok used to be pretty bad but since SpaceX bought Cursor, they used that data to train Grok 4.5 which is now very competent at coding and even exceeds frontier models in certain benchmarks.

https://www.semianalysis.com/p/google-we-have-no-moat-and-ne...

> that makes me wonder if the trillion dollar valuations for OpenAI and Claude are even justified.

They aren't, not even if we forget about the capable Chinese models.

I suspect Anthropic will implode soon when employees are unable to get the cash-out that they expected. Having so much compensation locked up in company stock is risky on a good day.

They have 2 moats.

The first is the compute. OpenAI and Anthropic secured huge amounts of compute, Google, Meta and xAI have their own huge datacenters. Now anyone can rent some cloud machines and start serving Kimi K3, but it's going to be impossible to get to a similar scale as the big 5 above. And inference has economies of scale: the more people you serve in parallel, the more efficient you are.

The second is the data. By now (and maybe even by one year ago), all the data on the internet has been used for training. You need new data. The big AI companies sit on top of trillions or quadrillions of tokens that they have generated over the years. They can use that to train new models. That data is gold, and the proof is that SpaceX was happy to pay $60B to acquire Cursor.

If you want to overtake the frontier labs, you have 2 options: use their models to generate synthetic data, and provide lots of (cheap, maybe below cost) inference to generate your own new data. The frontier labs know about the first, and I'm sure they try to limit how much others milk their models. As for the second, that's the "honest" way to compete, but it's not easy.

> This makes me wonder if AI companies even have a MOAT in the first place.

They don't. The moat will mainly be the tooling around AI, not the AI itself. You don't hear any company claiming their moat is the Internet.

China has the moat that they are cheap/free/open. The US corps have the moat that the other option is Chinese models. At least for some time.

Burdensome regulatory compliance is a moat.

These companies have AI and enough money to lobby the Pope. They can afford to reanimate members of congress and push some tactical legislation through.

But all the money in the world cannot move government too quickly.

Other moats exist too. OS or browser can undermine performance and availability of alternatives.

> This makes me wonder if AI companies even have a MOAT in the first place.

Generally speaking they do, at least from my experience when switching from one model to the other - their performance decreases, and they often do large refactors outside of the requested scope as they try to bring the code closer to 'their' style.

Which makes sense imo - they'v been trained to iterate over the code they wrote, and not code that was modified by someone else in the interim.

With their current API approach they're essentially a commodity. They need to start moving parts of the harness behind the API, otherwise they'll remain a commodity.

Recursive self-improvement changes the parameters a bit, especially for the market-leaders, and it's the one thing that makes me wonder if they'll be able to extend their lead faster than the smaller labs can keep up, but it's an option available to everyone.

> Cause if that is justified, then Kimi, Qwen, Deepseek etc are also valued at a trillion dollars

It’s more like a bunch of people are placing different bets. Only a few bets are going to generate a return, possibly only one, but the profit on that one bet will make it all worthwhile. That’s the theory, anyway.

This isn’t technically true. Most model providers don’t send the thinking tokens anymore, so if you switch from one provider to another, you will be missing large parts of the conversation.

fine tuning runs of models the size of gpt 5.6 are absurdly expensive. I'd guess at least $100k in cloud gpu time for a single run, and you have to do a few iterations to get things right

The major labs don’t allow assistant prefill, so you have to “summarize”

Mamba/SSMs could change this picture.

The moat is the US government.

I think you're right and I think it's why Google have taken their pedal off the metal for model releases to focus on integrations and tools. And why Microsoft have backed off from the OpenAI partnership to do the same. Anthropic and OpenAI are going to massively struggle to maintain their pace and reach profitability just selling commodity tokens.

Fine tunes are a possibility but I think it offers very little uplift for the vast majority of uses beyond just stuffing enough context.

They have a moat; they don't have $1 trillion valuations.

Which anyone who hasn't been sitting in the SV echo chamber could have told you years ago after applying even the smallest bit of thought.

From my experience these open source models are nowhere near the performance offered by Fable/Opus/GPT-5.6. Whenever I tried Qwen, Kimi, Deepseek, the results were much worse and it just took much more time to get something usable. When you consider that, the frontier offerings are still much cheaper.