> So Jev does actually know which distributions are correct, it just fails to produce them

Claims like this needs to be deeply analyized. Models do have some emergent capabilities[1], and I think there's a lot of evidence to show that semantics is actually learned (Word2Vec), and some math seems like it also might be learned (e.g. modular arithmetic). But it's hard to exactly say where there's some internal mechanism generating a true answer and where we're just getting lucky with some distribution so the answer just seems right.

[1] https://arxiv.org/pdf/2502.00873