logoalt Hacker News

Argonautlabsyesterday at 9:33 PM1 replyview on HN

Bandwidth doesn't multiply like that here, and we measured it rather than assumed it. A MoE layer needs 16 expert reads and can't proceed until the slowest one lands, so a layer costs the max over its reads, not the sum.

Going from one drive to four (13.6 → ~33 GB/s of combined ceilings) took decode from ~52% to 100% of our number — not 4× — with

Every drive already at 90–100% of its own ceiling. RAID-0 was one of the first things tried and it lost: striping makes every read touch every drive, so the slowest drive sets every barrier.

What moves this is per-read latency and read scheduling, and for long prompts not re-reading each layer's experts eight times.

Numbers in results/SCALING.md and results/PREFILL.md.


Replies

lowbloodsugartoday at 12:35 AM

Would the 40 Mac’s work with pipelining though?

show 1 reply