alt Hacker News> I'm not sure I agree that LLMs solve for the statistical mean
The good thing is that you don't have to agree with that, as the fundamental technical reality does it for you. LLMs do work like that. They are just statistics and probabilities.
> I'm not sure I agree that LLMs solve for the statistical mean
The good thing is that you don't have to agree with that, as the fundamental technical reality does it for you. LLMs do work like that. They are just statistics and probabilities.
Doesn't the universal approximation theorem apply to typical LLM architectures? LLMs solve for the shape of the input data, as well as they can approximate it in the constraints of their architecture, the optimizer and the amount of compute spent in the training process
And where those constraints prevent us from matching the shape there is no guarantee that we match anything like the mean. Though maybe we can agree that that's where the optimizer would tend to steer towards when it can't do any better