The smaller we go with good models, the more I wonder about tiny animals.

By memorizing the fact that the sun rises in the East and sets in the West, you no longer have to make complex predictions and draw arcs every day. By memorizing the fact that the Earth rotates, and that generally most bodies in solar systems do, you can extrapolate this fact to any planet or solar system.

This example is a rough comparison to what actually happens, but it does attempt to demonstrate what compression and knowledge are, and how patterns, symmetries, and extrapolation/interpolation can make describing/modeling a complex system much more efficient.

Models can even go a step further, offloading entropy to the input. An efficient model exploits predictable structure; the remaining details can be expensive to compress further. The input can supply whatever additional information is necessary to produce the desired output within some margin of error. The information required depends on the model’s efficiency over a given domain, the structure of that domain, and the required precision of the output.

Consider generating an image of a pebble driveway. Most models today will generate a plausible arrangement of pebbles using their learned distributions. Reproducing a particular arrangement requires the input to convey the details that distinguish it from other plausible arrangements. The more closely you want to reproduce it, the more informative the input needs to be—potentially including a reference image of the arrangement itself.

This certainly raises a lot of open questions that are fun to pursue, such as the smallest possible model that can recover a complete description of our universe from minimal input.