Having tested K3, Qwen 3.8 max preview, Fable and Sol for the past few days, I partially agree. Don’t trust the benchmarks, and the Chinese models really are slow and token-inefficient.

However they do seem very close to SOTA : I’d say roughly equal to previous gen (Opus 4.8, GPT 5.5). It’s yet another silly benchmark, but compare them here: https://senko.net/vibecode-bench/

I also had K3, Qwen3.8 and Fable (using Kimi Code, Qwen Code and Claude Code harnesses respectively, and the official APIs) create a simple but far from trivial web app (zero shot, from a detailed spec). In user testing all three results looked/behaved more or less the same. I had Sol (via codex) do code reviews on all three and it concluded all three were solid, with some room for improvement. Fable was slightly ahead of the pack.

In my own work I still prefer Opus 4.8 (until Antrophic come to their senses and allow 100% Fable usage on Max plans) and Sol, but if I had to find an alternative, I could live with both K3 and Qwen3.8 just fine.

Fable works very well for me on a moderately large codebase. I have had to correct it a few times or point it on the right track, but given how much faster it is at programming than I am that's a very minor issue (and most of these errors are because I underspecified what I wanted in the prompt, I can only think of two cases where it was genuinely wrong... that's a lot better than me in my professional career). Code quality is equal to what I would come up with (and better in areas I'm not familiar with) and the overall software engineering bar is higher because it doesn't get bored when I tell it to do refactors or write integration/regression tests that I would otherwise put off. Also makes it easy to audit code for things like missing audit logging or error notifications that a human would get bored doing.

The product is a fairly standard Ruby on Rails webapp with postgres as the DB. Application complexity is probably a bit higher than average for a webapp. So it's nothing that pushes the boundaries of software engineering, but it is a real product. Token budget has not been an issue for me. I pay for the Max plan ($200/month) and it is well worth it.

Yes, they are all benchmaxxed, but the question is how benchmaxxed they are relative to each other.

We run an evaluation that only compares models in open-ended multi-agent environments where agents affect each other, primarily testing writing code. It's designed to be less vulnerable because there's no solution set, and it's been pretty effective and tends to rank Chinese models lower than their advertised model cards (relative to US models). Kimi K3 is a bit of an exception there -- it truly is a near-frontier model. But it's so slow. Ironically, Muse Spark 1.1 is one of the strongest models we've tested after Fable and Sol while also leading the cost efficiency curve. Big turnaround from Llama 4.

Data at https://gertlabs.com/rankings

All models are benchmaxxed, period. ”Jagged frontier” is the euphemism du jour, I believe?

Anthropic/OpenAI were touting PhD-level intelligence three years ago. And they’re still shipping models that aren’t smart enough to realize things such as the need to drive the car to the car wash (because they hadn’t yet hill-climbed that particular brain-teaser).

It's telling that in their release post, Moonshot themselves said that K3 lags Fable and GPT-5.6 in "user experience". I took that to mean the stuff you can't push directly via RL, what some people call "big model smell".

"Despite being a highly competitive model overall, K3 nonetheless exhibits a noticeable gap in user experience compared with Claude Fable 5 and GPT 5.6 Sol."

https://www.kimi.com/blog/kimi-k3

I have been testing the various models, and I would not call Fable SOTA.

I can't actually get Fable to do anything. I only work on back-end code, and the moment Fable notices the jwt scope checks on the endpoints it's game over, it refuses to do anything because security is involved.

So for me, Fable is completely useless, the bar is very low, any llm that will actually attempt the task beats it every time.

Clearly the best way to test these is a completely scientific and controlled process of prompting for a creature on a bicycle. Much like testing the fitness of a fish by making it climb a tree, the LLM will show its true colors when presented with the impossible task of rendering an image.

didn't read the article, but just the title felt fishy because of course an inference provider wouldn't mind claiming to provide access to a mythos-class model. So far, GLM 5.2 is still my go-to, even at less than a third the parameter count compared to Kimi K3

There are some benchmarks I personally trust. But the most reliable indicator other than public benchmarks is the quality of products people are working on, and the model they use for the work. Not a benchmark scoreboard or a one~few shot demo, but the product they have been building for weeks. The reality is, even the diehard advocates who build and sell tooling for open weights, still use proprietary frontier models for their jobs.

Yeah on internal non coding benchmarks at my company, they are around last years models and don't hold up well.

Totally agree with all of this. Most of the problems we solve with AI are not one shot, all the bench marks also miss the human components when testing. Thinking alongside with a human and coming to a solution is what Fable does better than most other models.

So they aren’t as good as 2024 models? 2025?

I encountered the same issue. Kimi 2.7 looked impressive on paper, but in practice, the code was so riddled with errors that I ended up using GPT-5.5 to fix it.

I've seen this phrase repeated over and over again and I don't think it is true, or at least, to the degree that "benchmaxxing" claims are true, American frontier AI companies are probably about as guilty of it as Chinese AI companies are.

For both GLM 5.2 and Kimi K3 I feel the rough average of the benchmarks gives you a rough idea where they stand. GLM 5.2 was sitting somewhere behind Opus 4.8 but it didn't feel very far away. I've used Kimi K3 via OpenRouter and while I've had limited experience so far, it sure as shit feels like it's right up there with Sol and Fable to me. I am happily able to believe Fable has the edge still, but on a request by request basis it would be pretty easy to get an impression one way or another.

The existing Kimi models were already pretty good so I really just don't find this new model to be that hard to believe. Maybe I'm naive.

Well that sucks that everything is full of misinformation about this subject.

It's better to not even follow the news or benchmarks and just use whatever is available. I make my own judgement.

I think this one has greater impact than deepseek, next one for sure, China will take the lead without any question.

Why is token efficiency a concern with free models?

[flagged]

All you've said needs the qualifier - "for now!" Look at the trend line. It's clear that if they're not yet at the level of being "good enough" for coding, they will be soon. Sensationalist headlines aside, we all need to be preparing for a world where open models can do pretty much any software tasks you need them to.

You got baited by bad sampling settings.

It's exactly the opposite. Go turn on min_p once it's available post July 27th and most of the problems you describe will go away.