As much as I want to, I sadly don't think Information Theory makes sense in this setting, and I really wanted to believe this. When Shannon made his theory of information, he was always dealing with informational representations on the abstraction level of bits. At Bell Labs, a lot of the work was on the compression of data for transfer over telephone wires. Entropy coding, later codexes like algorithmic coding, and all compression on this level assumes that you have a bit-based X, and you compress it.

However, in deep neural networks, you are dealing with compression on different levels of abstraction. How do you decide what shared features a peacock and a palm tree have? At what scale should they be represented? How do you deal with invariance under affine transforms? Do you want to open the box of invariance under non-affine transforms?

When you start looking at what it would mean to compress feature representations, you immediately get to the question of data. You realize that Shannon simply was given a form of a very low abstraction data and that information theory came out to handle data at this level, but it's not suited for the data representations of many higher level modalities.

If you read Society of Mind by Marvin Minsky, which has aged well to about 80%, you can get the hint of the kind of abstractions that humans make and what would be needed to represent them, this is not representable in bits, you need to go to higher level shared features, and then you open all of the questions above as well as credit assignment, mutual information approximation, Fischer information between bayesians, etc.