I hope I'm not the only one who misread it as 292 tokens/s and got excited momentarily

I do think these “run a bigger model than will fit in VRAM” projects are necessary steps, but are they functionally useful or helpful to anyone currently? For example, is anyone out there running a big Qwen for coding on a 16-32GB machine with these techniques?

At that point, how does this compare with simply running the model on the CPU?

It matches my coding speed...its ok.

Ahaha thank you, I naively assumed the unlabeled graph in the readme was tps, not spt!

I wonder what this measures in J/token.

How many is that in tokens per Scaramucci?

that's 0.003 tokens/second. To get an hour's work done that's normally 30 tokens/second (108k output tokens in an hour) will take 416 days at this rate. And if you're using 100 watts, during that time you will spend $124.61 in electricity, as well as not being able to use your device for something else, plus the noise and heat from your device.

For $124, on Moonshot's official Kimi K3 API rates ($0.30 per 1M cached input, $3 per 1M fresh input, $15 per 1M fresh output), you can purchase 42 million fresh-input tokens, or 8.3 million generated output tokens, in whatever mix you want.

So what you get is 80x more expensive and you wait 416 days to get it.

Wow, you could do a lot at 292 tokens a sec—oh.

I have all praise for those taking this on and in my idiom would call it *the lord's work."

The image I reliably summon to mind is that compilation video showing the progress of Boston Dynamics bots. The curve between technically functional, to comically slow, to too slow for "real" work, on to, OMFG, may prove a (rough) curve.

It's work like this that moves things forward.

Hah I was looking for it and couldn't work out how many years/token. 292s is pretty good.