alt Hacker News200B of those 552B is PLE, which works more like a database that is read for each token, thus can be offloaded to a fast SSD.
Replies
Id love an ELI5 for PLE. Im trying to work it into my back of the napikin math for compute vs memory bandwidth limitations on tok/s in PP vs TG work.
My attempt at a simplification of this article on it https://sebastianraschka.com/llm-architecture-gallery/per-la... into a couple of sentences is that they are linear embeddings of the input token space projected per layer, which are then gated by the transformer outputs per layer.
This would mean that the only one set of weights for the ple path needs to be pumped across the memory bandwidth as they are the same linear weights for all layers?
Sheit, maybe im trying to simplify something that i need to look at in detail. but id love to leverage others understanding if possible
[flagged]
Unfortunately no, it's 200B + 552B. It's not as bad as it sounds though, because most of 552B is in 4bit natively.