alt Hacker NewsThis is the thesis behind the "Information Theory, Inference, and Learning Algorithms" course that was taught at Cambridge University.
> Why unify information theory and machine learning? Because they are two sides of the same coin. In the 1960s, a single field, cybernetics, was populated by information theorists, computer scientists, and neuroscientists, all studying common problems. Information theory and machine learning still belong together. Brains are the ultimate compression and communication systems. And the state-of-the-art algorithms for both data compression and error-correcting codes use the same tools as machine learning.
Book (creative commons): https://www.inference.org.uk/mackay/itila/book.html
Lectures: https://m.youtube.com/playlist?list=PLruBu5BI5n4aFpG32iMbdWo...
Replies
As much as I want to, I sadly don't think Information Theory makes sense in this setting, and I really wanted to believe this. When Shannon made his theory of information, he was always dealing with informational representations on the abstraction level of bits. At Bell Labs, a lot of the work was on the compression of data for transfer over telephone wires. Entropy coding, later codexes like algorithmic coding, and all compression on this level assumes that you have a bit-based X, and you compress it.
However, in deep neural networks, you are dealing with compression on different levels of abstraction. How do you decide what shared features a peacock and a palm tree have? At what scale should they be represented? How do you deal with invariance under affine transforms? Do you want to open the box of invariance under non-affine transforms?
When you start looking at what it would mean to compress feature representations, you immediately get to the question of data. You realize that Shannon simply was given a form of a very low abstraction data and that information theory came out to handle data at this level, but it's not suited for the data representations of many higher level modalities.
If you read Society of Mind by Marvin Minsky, which has aged well to about 80%, you can get the hint of the kind of abstractions that humans make and what would be needed to represent them, this is not representable in bits, you need to go to higher level shared features, and then you open all of the questions above as well as credit assignment, mutual information approximation, Fischer information between bayesians, etc.
Ah Sir David MacKay. I so respect him. Great explainer and speaker. He had built this text entry tool called Dasher [0] - that I'd heard him introduce at Princeton around 2006. It was basically an early language model that predicted which characters are more likely than others, given what you've already types and it would adjust the sizes of the available next characters based on their probabilities.
This is basically a thesis supported by Shannon’s information theory.
Any rigorous CS program should cover this in depth.
It also helps explain to people that LLMs are as likely as bzip to develop "consciousness".
Better prediction being better compression is Shannon 1948, and the link to machine learning is MacKay 2003 at Cambridge.
I had a long ranting comment I deleted. I just don't like this trend of people presenting work in a way that makes you think some combo of 1) they discovered from scratch themselves 2) it's new 3) they didn't try to cite or acknowledge where they learned it/point to good sources 4) they don't really care about trying to teach something deeply, they want shiny stuff that makes them seem deep.
This post references specific parts/calculations, but you'd never know it was not news if you didn't know better.