alt Hacker Newsmake it 4x40 raid-0 ssds to achieve 40 tps.
or 40 macbooks with each 4 ssd. to get 40 tps.
Replies
Bandwidth doesn't multiply like that here, and we measured it rather than assumed it. A MoE layer needs 16 expert reads and can't proceed until the slowest one lands, so a layer costs the max over its reads, not the sum.
Going from one drive to four (13.6 → ~33 GB/s of combined ceilings) took decode from ~52% to 100% of our number — not 4× — with
Every drive already at 90–100% of its own ceiling. RAID-0 was one of the first things tried and it lost: striping makes every read touch every drive, so the slowest drive sets every barrier.
What moves this is per-read latency and read scheduling, and for long prompts not re-reading each layer's experts eight times.
Numbers in results/SCALING.md and results/PREFILL.md.
yeah raid or not you still get the hard limitation by the pcie lanes
it is even worse with 40 macbooks.
if 40 macbooks is all that take to serve a 1TB model with decent speed then you would see everyone selling the models for very cheap right now.