I think the missing idea for AI is in Iain McGilchrist's The Master and His Emissary. The book's thesis, very roughly, is that the hemispheres aren't redundant processors: the left specializes in focused, decontextualized representation (sound familiar?), while the right maintains broader, contextual engagement with the world (McGilchrist fuzzily reduces it to "what is out there?" which is fundamentally about ascertaining uncertainty as a mode of attention). And crucially, the corpus callosum connects the two processors and crucially helps preserve their functional separation through certain channels (as both sides are connected).

Transformers look suspiciously like very powerful left hemispheres. They are phenomenal at manipulating representations, but increasingly weird when the representation gets mistaken for the thing itself. Hallucination, sycophancy, context collapse, overconfident completion, etc. all look less mysterious when you compare it to the lists of pathologies that those with right hemisphere injury face (e.g. anosognosia, confabulation).

So what seems missing isn't a Gödelian strange loop but the other side of the brain and a mediator between the two sides.

The way I think about it is like Kriegsspiel, the ur-game for roleplaying and wargames. To truly "simulate" war (or a fantastic medieval adventure game), the simulation requires three sides: The Blue Team with a goal, the Red Team with its own goal (typically to stymie the Blue Team), and the Umpire, whose only job is to objectively simulate the effects of the orders of the two sides.

The Umpire has the true state of the world; Blue and Red only get observations through the Umpire. Neither can directly inspect the other's state. Each side must provide orders to the Umpire, who, ideally, being an SME, is able to process them both according to the rules of Kriegsspiel but also their real world experience in war. And that's the whole point of it in the first place: the Umpire maintains the fog of war between the players, with the ultimate purpose being improving the generalship (i.e. learning) of both the Blue and Red sides. And what is generalship really? It's being able to discern resources and operations in a world of Knightian uncertainty where one must evaluate whether there is an active intelligence attempting to thrwart your model of the world and your goals.

(An aside: I think this is why Dungeons & Dragons fails fundamentally as a rules/rulings-bound game. The dungeon master and referee are the same person, while each role has a fundamentally different goal. With split roles, the DM can actually try to kill the players using the Dungeon with the same level of asymmetric information as the players and the GM to actually referee the game neutrally with a view to what actually happens.)

Nobody actually gets the God's-eye view. The Umpire gets objective reality but lacks insight into the minds of either side, both of which are pursuing unique ends. This triad allows for genuine Knightian uncertainty in games where judgment (i.e. creating and using precedent) is fundamentally required. This triad is also shared by law, government (and from my perspective trinitarian theology). Intelligence becomes qualitatively more powerful when the architecture prevents any one component from possessing a complete, self-consistent description of the whole, while allowing the components to interact through constrained third parties.

I suspect this is also why human cognition seems to somehow avoid Godelian incompleteness by sidestepping the requirement that, as a formal system, it contains a complete model of itself. Just give the system multiple partially informed perspectives coupled through an epistemic boundary, and incompleteness becomes a source of entropy sifting and uncertainty rather than failure modes.

And if what I suspect is true, at some point the qualitative bottlenecks that pure transformer-style systems possess should begin to disappear when this approach is applied.

But I'm generally an idiot, so take all this with a huge grain of salt.