Will Wright once described the same connection from the opposite direction: compression as procedural content generation.

In the 2023 discussion of "Demoscene accepted as UNESCO cultural heritage in The Netherlands" I posted a transcript from a video of Will Wright discussing the demo scene:

https://news.ycombinator.com/item?id=36599415

Will Wright Discusses the Demoscene:

https://www.youtube.com/watch?v=m7iuFVmTJus

>You can take any piece of content in the game, and imagine an algorithmic solution to it. Or also, you know, a way that the player could customize that object of thing.

>There's this group in Europe called the Demoscene that make these very elaborate demos for a computer that fit into very tiny little memory blocks, you know like 64K of memory, and you run the thing, and in fact it algorithmically generates about 100 megabytes worth of data, you know these rich 3D environment, generated music, generated wave files, generated animation.

>And they're developing techniques to generate, you know, huge amounts of interesting data, with very very simple, elegant, compression algorithms.

>And this is a skill that game developers used to have, back in the 8-bit days. That was the only ways to do a game like Karateka(?), was to find all these little tips and tricks to compress things and generate them algorithmically.

>But since the CD-ROM came out, and very cheap hard drives, storage is cheap, so basically we've lost that skill set, and now we attack all those problems with brute force. I think we've lost something by dropping that skill set.

[...]

https://news.ycombinator.com/item?id=36613058

[...] Here's a simple low-tech pre-LLM example that shows the equivalence of compression and procedural content generation:

Take a huge text file of HN postings, and compress it with gzip or compress or some other robust compression algorithm. The better the algorithm, the more the output will look like random noise. Then slice the compressed file in half, and replace the second half with random numbers. Then uncompress it. You'll find that at the point you sliced it, it keeps on writing out almost plausible text for a while, consisting of highly probably snippets of commonly encountered words and phrases, then goes downhill towards incoherence. It's not as coherent or confident as an LLM, but the point is to show how low the bar is for using compression for procedural content generation.

LLMs are essentially a form of compression of the world's knowledge or whatever they're trained on, not just word frequencies or pixel patterns, but also concepts and ideas. [...]