alt Hacker NewsIndeed. If you're going for a catchy generalization, at least write it correctly. Most compression is history, and only extrapolates under the assumption that "nothing changes".
Indeed. If you're going for a catchy generalization, at least write it correctly. Most compression is history, and only extrapolates under the assumption that "nothing changes".
there are dictionary compressors (decent compression, most common, fast), and statistical compressors (better compression, slower). Statistical compressors are much closer to LLMs in that an llm is learning statistics about the data too.
And yes, compression is history, that's what statistics are all about. Statistics can only measure the past to make a prediction about the future. And LLMs work in the same way. The context is the history, and given that history, it predicts the next token. An LLM can, almost trivially, be dropped into something like the PPM statistical compressor (it's just replacing one implementation of a markov chain with another).