It will be very interesting to see what kind of 'slow' performance people get from running it on a no GPU, but tons of RAM server (like a dual or quad socket xeon with 1.5 to 3TB of RAM). For the purpose of giving it longer duration tasks to generate a piece of something and come back and check on what it has done in 4 or 6 hours. Even if the output is like 5-6 tok/s, that might be usable for some purposes.

Huge price difference in what you can do with buying a used 4U rackmount server and putting 3TB of RAM in it (64GB DIMMs x quantity 32 in a quad socket xeon, you can see some benchmark prices on eBay for sets of 16 or 32 matched 64GB ECC DIMMs) for <$30,000, vs the cost of trying to run it on real GPU hardware.

Now obviously, as of the time I write this, the full precision hasn't been released nor has anyone like unsloth run it through quantization yet to produce a "Q8" or "Q8-XL" variant of it. But I think it's going to need more than 1536GB of RAM, with a usable and large amount of context, more like 2TB and preferably 2.5 to 3TB.

I also predict that people who try to run it in Q4 and Q6 will get the worst of both worlds, less precision/lost knowledge but also not reliable output that comes out too slow. In my personal opinion if I'm going to deal with something that is smart but slow and running on limited budget hardware, I need it to be Q8.

Speaking of finetune, currently a common practice is LoRA over bnb 4-bit base model, but I think it's time to replace bnb with GGUF as the base model format. GGUF is actively supporting new model architectures and more aggressive quantizations.

I've made some proof of concept in https://github.com/woct0rdho/transformers5-qwen3.5-recipe . We can finetune Qwen3.5-35B-A3B in 16 GiB VRAM, and DeepSeek-V4-Flash (284B-A13B) in 90 GiB VRAM, without CPU offload. This works well on unified memory machines like Strix Halo.

Even so, larger models like Kimi-K3 still require multiple GPUs and nodes, and there are a lot more to do compare to single-GPU training.

> Then we'll be able to guesstimate if "labs are subsidising tokens on API pricing".

No, you don't. Without training cost you can infer only the marginal cost of serving this kind of models.

Moreover, you don't know the actual size of closed models (what if Fable is a 10T model? What if it's 1T?)

It's 3/15 - https://openrouter.ai/moonshotai/kimi-k3

If you're going to open source your model, why would you set your own price high enough that other providers could easily and profitably undercut you?

Since the model is natively MXFP4, I think it'll be even more interesting on the hardware front. It'll comfortably fit on a 8x AMD MI355X node. I suspect that'll drive token prices down, further.

Say a single Kimi K3 is deployed on 16 x B200s: how many concurrent users can that handle? I realize the question assumes a major simplification that everyone's prompts/sessions are the same.

AISI is capped at 100M tokens and K3 is less token efficient than Anthropic/OpenAI models. There is an argument to be made, looking at AISI results, that with uncapped tokens it would be just slightly behind the closed weight players.

This is a very insightful eye opening take. I haven't even thought of it this way. This really is the first open model to be as big as the frontier has been until now.

I think this release is actually great both ways when you think about it. We gonna be able to learn knowledge that labs have been hiding from us (e.g. cost like you mentioned). And labs could learn from whatever optimization techniques people come up with when trying to host this model.

It's honestly just good for everyone in my opinion.

If it is a mixture of experts (MoE) model like the 2.x models, won't this reduce the hardware needed to run the model?

The Kimi-K2.6 model is 1.1T parameters with 32B active parameters. With light quantization (Q6_K) that's enough to run it (slowly) on a single 5090. On a single B200 you can have 5-6 experts loaded into VRAM at a time. Realistically that would be 3-4 to account for the context. [!]

[!] With this and other MoE models it looks like an interesting area for research would be to detect or predict which models would be needed ahead of time. That way you could schedule the load into VRAM step before the weights are needed. That way you shouldn't lose much/any performance from offloading the weights to RAM.

If it takes so much resource to run, how does the sharing of a single llm works? There is some interface that basically submits context/cache plus current promt, from each user, doing basically time-sharing compute?

You're assuming inference providers are going to sell tokens at cost. You're also assuming that the inference providers have will optimized inference engine. I haven't seen that to be the case so far, to be honest.

Take a look at GLM 5 vs GLM 5.2 pricing -- GLM 5.2 cost more despite being the same model.

Take a look a look at DeepSeek, which hosts DS v4, profitably, yet others aren't able or willing to match the price.

I think cursor will likely do a grok fine tune rather than a kimi one for the next composer.

they noted in their blog post they didn't focus purely on coding for grok 4.5.

We have a lossless compression codec (working on open sourcing it over the next couple of weeks) that reduces it down to its minimum entropy -- it cannot be compressed further. On all tested large models, it's a ratio of 1.34-1.23 -- and smaller models up to 3.76x. It also increases the effective bandwidth by the same rate.

So nowadays the hardware and hosting providers must be in an optimization race, whoever can make the model just a bit smaller or more efficient (to fit on fewer/less powerful cards) will have a huge advantage and can make a lot of money.

I am curios what's the most profitable thing to "plant" (agriculture analogy) on the land (cards) that you have have: web hosting, vps, llms, image/video generation, etc

>realistically you'll need 16x for context / throughput optimisation

Sounds like I'm buying a lottery ticket this week so I can drop $800k on hardware.

> if "labs are subsidising tokens on API pricing"

> SemiAnalysis estimates that Anthropic's current blended gross margin has risen to the mid-60% range, with the API business gross margin exceeding 80%

Of course, people will insist "they are lying", "why should we believe them, it's well known they subsidize API pricing", ...

https://newsletter.semianalysis.com/p/anthropic-3q26-profit-...

https://finance.biggo.com/news/02d45650-b569-4d12-b44d-8d6d8...

> The latest AISI benchmarks on cybersec place it above glm5.2, but still way way behind SotA closed models.

Sota closed models don't even answer cybersecurity questions lol.

pardon my ignorance but is fine tuning still considered viable in face of rapid model releases. Is it really worth it?

my friends whove tried in their companies gave up on it.

Anyone who thinks that the labs are not profitable on per token API pricing is delusional and hilariously wrong.

[flagged]