alt Hacker NewsThere was this post a few days ago https://news.ycombinator.com/item?id=49797323
It had this to say in the linked post:
This led to the natural question: can gzip do language modeling? (...). Here’s some real, unedited output after priming it on tiny Shakespeare:
gzipt --corpus data/tinyshakespeare.txt --prompt $'MENENIUS:\n' --length 200
MENENIUS:
'Though all at once canq
MARCIUS:
Pray now, nocamest thou to a morsel.
LARTIUS:
Hence, and
I' the end admire, where G
again; and after it ag .
The other is what thinking does, it tries to predict the missing sequence of tokens from a high entropy source, the prompt, in order to increase the likelihood of correctly decompressing the desired results from its weights.
Replies
How can things be compressed without losing information or structure?
Like for text, what would that involve? How do you compress a string or multi-line string without losing information and hopefully structure (paragraphs, would it be like replacing periods and the following space with just sticking the starting capitalized letter of the following word to the previous sentence's last letter and when it decompresses theres some kind of note that converts that back into the. First letter of the next sentence
I don't know how this is related, but it reminds me of how I have always believed compression to be the ultimate sign of intelligence. If you can reduce something while keeping comprehension, you are finding more abstract symbols to represent the information of the original source.