alt Hacker NewsBaking the base models on to ROM makes a lot of economic sense. SRAM for the KV cache & fine-tunes, not so much. Sure you’d get incredible speeds but it’s not scalable from a die-size or cost perspective.
Rather base model on ROM + KV cache on DRAM is much more scalable. Also this would work great for edge devices that have a 2-5 year lifecycle.
Replies
adrianN • today at 3:00 AM
It is my understanding that just baking the model itself into silicon only gives moderate gains because memory bandwidth remains a bottleneck.
➕ show 2 replies
Baking the base models on to ROM makes a lot of economic sense.
Less so for consumers though, because it'd mean the phone is out of date in 3 months when a better model comes along.