Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

Reminds me of Deep Thought from Hitchhiker's Guide to the Galaxy

You currently can't run a 2.8T locally; there's just no way. So, it's a good start.

A medium prompt in only 11 days.

There's a tendency to cite only decode speeds, but in practice, an LLM generates far fewer tokens than it has to read (unless you ask general knowledge questions). So the effective performance is much slower than the decode rate suggests, because a 512-token prompt already takes 6 minutes to load

Should LLMs be designed to be modular, so that instead of needing access to the whole model, for a given prompt, only a small subset of the model would be used? If knolwedge was sufficiently modularized, most of it could be ignored.

Maybe a hyopthetical model of 5T of indexable weights could be used with only 50 GB of GPU ram, efficiently, because it stays resident in the GPU.

The SSDs are necessary because Apple's architecture doesn't allow RAM upgrades. Reminds me of someone who said "640KB ought to be enough for anybody".

I missed the explanation for how the SSDs are connected.

Maybe a dumb question.

Now imagine the token/s rate decline after context fill at 200,000+ context.

I wonder if faster SSDs would help?

In particular you can still get used Optane SSDs on eBay, although they’re fairly pricey. (not the bogus m.2 ones that are slower than a halfway decent consumer NVMe)

I think this is cool not for kimi but for sth like glm flash

not sure ive ever seen a #1 post on HN with only 5 stars

That's next level masochism.

That's actually pretty neat.

These slop readmes are painful to read.

make it 4x40 raid-0 ssds to achieve 40 tps.

or 40 macbooks with each 4 ssd. to get 40 tps.

Good start.

We’re reaching quadratic slop. Slop projects that don’t understand what they’re shipping built on top of slop projects that also don’t understand what they’re shipping. Magnificent.

Even if that's impressive, the README is low SNR slop as usual...

Author here. Some context and the caveats up front. The model is Kimi K3, 2.78T parameters, ~1.45 TB of expert weights. It does not fit in memory, so the experts stream from disk: one 17.5 MB file per (layer, expert), read with pread + F_NOCACHE, 16 of 896 per layer. The machine is an M5 Max MacBook Pro with 128 GB and three Thunderbolt 5 enclosures plus the internal SSD. Expert weights are untouched at their released MXFP4 precision; the resident attention trunk is int8, which upstream labels non-weight-exact, so I don't claim bit-exactness against BF16 — I claim token-identical output against my own reference on the prompt of record, checked on every promotion. Numbers, with the unflattering ones in the same paragraph as the good ones: 1.00 tok/s steady over a 512-token completion, 1.13 over 128 tokens, and 0.96 median on the 17-token benchmark from the upstream repo's issue #15 against the 0.684 posted there. Time to first token on a 512-token prompt is about 6.3 minutes — prefill is currently read-amplified 6.2x, which is the biggest open problem in the repo and is described in the results directory. What I think is actually interesting isn't the number, it's that four of the gains came from defects in the read path that instrumentation found and I would never have guessed. The instruments are in a second repo, ARGODRIVE — a 10 ms per-device read monitor, a per-read barrier trace that records which drive served each expert and which one landed last in every pass, and a config assertion harness that refuses to record a benchmark unless the setting under test actually fired. They're deltafin-specific today. The four findings: • A constant capped the reader threads at 16 and bounded both the demand and prefetch pools with the same value. Separating them was +14%; demand queueing went from 70% of blocked time to 7.5%. • Splitting each hot expert's read across two replicas on two devices was +10% — after the same knob had measured negative six times on layouts where every expert had one home and there was nothing to split against. • The prefetch path had no balancer at all: it walked a fixed directory order and took the first hit, so on any replicated layout it dumped everything on one enclosure. Giving it least-expected-completion dispatch with in-flight counters shared with the demand path was +11% and turned every replicated layout I had previously measured as a loss into a win. • A recorded "law" that a given draft depth was worse turned out to have been measured against a drafter that no longer existed. Re-testing it was +8%. There's also a drive-count ladder in the repo — same layout, one to four drives: 57% / 78% / 92% / 100% of the four-drive decode rate. And a catalogue of about a thousand timed runs of things that did not work, with the numbers: RAM expert caches from 8 to 40 GB (-4% to -48%), striping a single copy (-7 to -25%), two drives sharing one Thunderbolt link (-11%), streaming the attention trunk from SSD (-60%), Metal's file-loading API (-19 to -22%). That catalogue is the part I expect to be most useful to other people. The engine is a fork of gavamedia/deltafin, which is MIT and did the hard part; I've told the author about all of this and the upstream-relevant fixes are going back as PRs. Two things I'd genuinely like help with: whether anyone has done expert-major prefill scheduling on an MoE (read each expert once per layer and run its kernel over all rows routed to it — it should take prefill from 6.2x amplification to about 1x), and whether the drive ladder reproduces on other hardware.

[dead]

[dead]

[dead]

But why though? Cannot possibly be useful at such slow speeds, and costs a ton to perform that badly