alt Hacker NewsSo this compression is only pertinent to the LLM file format? In memory it'd have to be expanded into the 1.58-bit form - 5 trits per byte.
So this compression is only pertinent to the LLM file format? In memory it'd have to be expanded into the 1.58-bit form - 5 trits per byte.
It also means you can read them faster, more parameters per second during an inference which tends to be memory bandwidth limited on most systems. Thus faster inferences