They can over time, but A) they can only improve to the human ceiling because they only have the set of apps that we have already created as inputs and B) creating new apps requires testing in the real world, which requires either humans or robots to go out and use the apps to do things.

You could try to make plausible apps that you never use, but then you run the risk of drifting. And without new apps to build on, you will be stuck on an artificial maximum that works well for the apps of today, but may not for the apps of tomorrow.

Imagine if AI had never seen a GUI app, if they had existed in the era of the CLI as the primary way of interacting with the server. How would they have known that Futures would be so important? They could try to guess, but most people thought Engelbart was crazy before he demoed his ideas. Hard to imagine an LLM making progress in a vacuum.