alt Hacker NewsPTQ and vector quantization aren’t used for this because part of the point of ternary LLMs is to make them faster. In a ternary LLM every weight is an add, subtract, or no-op so it is fast on CPU.
If you’re just using a code book to reconstruct a f16 model the only savings you can get are in sending it over the wire.
Replies
mitxela • yesterday at 11:35 PM
which is important though since sending it across the wire over and over and over is actually the main bottleneck.
om8 • today at 1:40 AM
> If you’re just using a code book to reconstruct a f16 model the only savings you can get are in sending it over the wire.
That’s why you need to use efficient gemm kernels like FLUTE for inference. They are ~as good as what you can do with ternary quantization.
And storing it in memory. Memory is expensive.