That “after a few years’ experience” bit is the catch. What’s happening during that time is a whole bunch of skill acquisition, building mental domain models, building context- and culture-specific soft skills, etc.

All of it relies on learning and information processing capabilities that LLMs simply do not have. The idea that any of it will automatically emerge from the current architectures simply by adding more parameters is about as sensible as thinking that if you keep counting long enough eventually you’ll get to complex numbers. We’re going to need to invent new technology to fill these gaps, and there is no way of knowing how long those developments will take.

It is unclear to me that "transform society" is automatically a good thing. Nuclear war would transform society too.