It's quite trivial to show that an LLM doesn't have underlying intent, and that it can only emit direct textual convolutions of its training and not combine tokens in truly novel ways. This is the very thesis of the world-model folks, e.g. LeCun et al, that LLMs are a general intelligence dead-end because they lack any inner concept of the world around them, and do not reason from that.

Furthermore, LLMs clearly do not "reason", despite the marketing around this term; their "chains of thought" are the nothing more than the result of having been trained on explicitly verbalised multi-step processes. There are many cases where the putative result arrived at in the <think>chain of thought</think> does not match the result emitted.

Whether they are "better" at software development than I am greatly depends on whether one is asking them to retrace worn technology paths that are well-represented in their training--in effect, to copy prior art--or to do something in quite obscure technology, or something quite novel altogether. (However, I will happily concede that most everyday business programming involves neither.)

Still, if LLMs were actually intelligent, let alone superhuman in the sense you suggest, then we would expect major scientific breakthroughs to be raining from the sky. If, say, an Einstein, could transform physics with only the knowledge gleaned from a human's feeble capacity to retain the literature of the time, I'd expect LLMs, who retain orders of magnitude more information with far greater fidelity and precision, to have offered at least a small slither of evidence of their superhuman capabilities.

I would also expect the objective progress and capabilities of this galaxy brain to be accelerating, not substantially slowing down as it has. GPT-2 to GPT-3 was truly a quantum leap, GPT-3 to GPT-4 was a substantial jump, GPT-4 to GPT-5 was meh, 5+ is basically unimportant, and so it goes for the other models. There are, of course, holes plugged and benchmarks where these evolutions have been, in various niche ways, consequential, but in the plainspoken meaning of model capability, the low-hanging fruit of pretraining was clearly exhausted quite some time ago. The carnival has been running on "agentic" / MCP / RAG / tool-use fumes since. This is moderately impressive and adds quite a bit of runway, but intelligence it is not.