alt Hacker NewsWhy is there still a hard VRAM requirement that's dependent on the model size? Isn't that exactly what this project is supposed to solve?
Replies
MakazhanAlpamys • today at 4:19 PM
[flagged]
Why is there still a hard VRAM requirement that's dependent on the model size? Isn't that exactly what this project is supposed to solve?
[flagged]
Because streaming only removes the decoder stack. The embeddings and lm_head stay resident, that is 2.10 GB of the 3.32 GB peak on 8B. And the logits tensor scales with batch x seq x vocab, not with depth.
So it goes from "whole model must fit" to "embeddings plus one layer plus logits must fit". That is why 8B works and why I did not try 14B.
The table on the site is the normal resident path, streaming is opt-in and BETA. Should be clearer, my fault.