Models this size aren't targeting hobbyists running models in their bedrooms on gaming/surplus hardware bought with disposable income.

This model is for researchers who already have racks of equipment, inference providers who sell access to models, and for businesses who must (or choose to) use on-prem models. All of which have to pay to use the model if their revenue rises beyond certain thresholds: https://huggingface.co/moonshotai/Kimi-K3.1-Pro/blob/main/LI...

We're running Kimi 2.8 on a 4x AMD MI300A single server on-premise that cost about $110k.

We will break even from our old Claude spend in about 2 weeks.

It services 30-40 analysts and staff concurrently and is faster than any of the SaaS out there for us.

The benefit is that you can use these models on a variety of different providers. You're not stuck with a single proprietary provider. Aggregators like OpenCode Zen and OpenCode Go can shop around providers on these. OpenRouter shows you the costs of a bunch of the providers, and you can route to the cheapest ones.

I regularly use together.ai with models like this, where they're fully self hosting them and I can turn off all data retention, at a fraction of the cost of proprietary models. I recently discovered abliterated models (models edited to not refuse to work on security stuff) and hosts for them like refuseless.com, and that's an option for open source models, not for proprietary ones.

You could easily run it with SSD streaming on as little as 192GB RAM (The dense portion is around 115GB or so at native precision, and the remainder would fit KV and a cache for the most commonly referenced experts.) This fits Gorgon Halo, which is a reasonably priced platform. It would be slow, but still usable for high-value tasks.

Possible data center bans incoming + tons of GPUs sitting in warehouses due to lack of power infrastructure to power them.

Perhaps a flood of 8x B300 will be homelab ready in a few months

"Pacing the liquidity frontier" :D brilliant. Will be using that.

Due to costs I am unfortunately years away from ever seeing hardware at this level, but if you just want to study LLM's and AI then there are still plenty of models that work fine on crappy hardware, and you don't have to worry about them becoming sentient on you :-)

It's not an open weight model (check the license, and the definition of that term if you disagree).

> Black Friday is coming soon though

A few weeks before Black Friday a couple years ago, I noticed a listing on Amazon for a high-end AI card for around $40,000 (maybe a B100, don't recall). The listing had the automatically inserted Xmas season note "Free returns until Jan 31." I spent a serious minute pondering if I could float the cash for the ~40 days between the credit card bill being due and returning the card. :-)

AND developer cloud has an image for K3 running it on a single MI300x. Offloading to system ram can be quite decent.

I cant say how well that image works as I'm still waiting for them to approve my gpu access

Depends on number of users - for a single user with a couple of sessions, you don't quite need that much. Bit of a waste of hardware to only serve this for single users though.

It’d be interesting to think if device swarm serving of these models ever becomes possible - are a few projects that seem to be just starting on this.

It seems more plausible to me than wherever the frontier is at any given time getting small enough to serve on one local card directly

Do you need 450k context?

I usually keep under 200k if I can help it for Claude/ open ai

And lower locally on my Mac to save on prefill.

I have managed to boot up these models for 1-2 users on just a couple B300s. Which costs a lot still but it's not that bad.

wish they just said "we are better than openai and anthropic, fuck you sam and you dario"

2 gb300 node actually. Or 8 gb300 cards since 1 node has 4.

you are aware that hardware gets better and cheaper, right ?

[dead]