alt Hacker NewsFor a model this small it’s also a way of trading supply-constrained VRAM for inference time, which for self hosting consumers (and probably Chinese companies subject to export controls) is mostly good.
I can imagine a curve where for a given level of “intelligence” you either need model size, or inference time (“test time compute”), and can somewhat trade one for the other.
> I can imagine a curve where for a given level of “intelligence” you either need model size, or inference time (“test time compute”), and can somewhat trade one for the other.
This is exactly what was shown on Luna/Terra/Sol tradeoffs - Luna requires much higher reasoning efforts to approach Terra/Sol on lower reasoning. Which is fine, of course, no complaints - but true.