Sounds like you are describing a quantized model which is a naive form of compression, not a model that is trained more efficiently.

Additionally the information theory angle is for information storage, but a model can access resources and tools to gain information and what we are really seeking to train is reasoning not information retrieval. We reduce the needs to the right capabilities and we don’t get upset if it does not know the lyrics to every song ever written.