I wish someone would ship a reasonably high end GPU with a lot of highly parallel NAND storage attached.

Solid state NAND has 100x the latency of RAM but in parallel it's just as capable of the bandwidth to feed a gaming GPU as RAM since bandwidth scales. Much much cheaper per byte too. The latency isn't a blocker in inference because the weights needing to be processed are predictable. You can actually work around latency, and in general that only affects time to first token not total tok/s, which on local inference is much faster no matter what you do compared to the internet. For local AI all you need is storage/memory bandwidth and that's actually perfectly scalable for any storage mechanism.

The bottleneck today to running Kimi 3.1pro at home is storing the weights in a way that you can get them into and out of the GPU for processing fast enough. A good gaming GPU can process the ~100billion active weights fast enough for a reasonable tok/s for a single user but we cannot get the weights from either system RAM or high bandwidth storage into and out of the GPU fast enough atm and we can't keep the weights in GPU RAM because that would cost too much. The way it's done in datacenters is just to have many GPUs working together and all the weights always in GPU RAM across a cluster of GPUs.

The datacenter approach of all weights in RAM always is great for serving 1000's of people at once, each with 100's of tokens per second but for home use it's a waste. You'd never individually maximally use a cluster of GB200's.

So for home use what we really want is a lot of high bandwidth storage of any kind (it can seriously be solid state NAND, just needs a 10x wider interface for bandwidth) hooked up to a moderate amount of compute. That'd get us Kimi on a single card.

I'm going to assume someone will make it at some point in the near future. At that point we're free!