I don't think that's it. The context that LLMs work with is simply too small, and can't get bigger. Harnesses and the models themselves are bolting structure over it manually; real people are inventing design machines that are a lot more deterministic than they look, and LLMs are just filling them in.

The design machines aren't good enough yet; and their progress has little relationship to LLMs at all. Designing computer programs and the proper level of abstraction for the easiest maintenance, flexibility and debugging is simply an open problem. There's nothing to train LLMs on that is smaller than the size of an entire program, and its history of code changes.

Get a few of those, figure out a way to analyze and reduce the patterns and lessons in them, and train on that maybe. Until then, there's no way for an LLM to figure out what lacking a particular abstraction is going to end up costing.