alt Hacker NewsI wonder if it would ever start to make sense to burn an AI model into ROM, replacing a large portion of an inference machine's RAM with ROM. (Probably not, since I'm sure those machines do dual-duty and run training when the inference workload slows down)
Replies
Companies are building chips specialized for inference, so dual use for training isn't necessarily a consideration, but there are other considerations such as:
Weights need to be loaded into the accelerator's processor fast, which means they need to be physically adjacent to it, but there is limited physical space for that - not enough to fit the all the weights of a 1T+ param model, so weights get loaded into VRAM dynamically according to what part of the model is being run.
ROM (I guess we're talking Flash memory) can be dense, since it is built vertically - many hundreds of layers, but this comes at the cost of poor performance, so even if you could fit enough ROM next to the processor it would not be fast enough.
That's the idea behind Taalas (https://taalas.com), except as silicon rather than ROM. They run a demo at https://chatjimmy.ai/ which serves an old open weights model (Llama 3.1 8B) at something like 15,000 tokens per second.