First, two things can be true: the models can be amazingly good, and they can also be insufficient to run a company, at least now and maybe forever, because they are not "intelligent" in all the ways we are.

IMO the invention of AI is redefining what we think of as "intelligence". You would probably describe Albert Einstein as highly intelligent, and you'd probably also describe Bill Clinton as highly intelligent. Even before AI, you can easily look at that and see we aren't describing the same things even though we're using the same word. Einstein's intelligence is highly abstract, observational, mathematical. Clinton's is interpersonal, pursuasive, articulate, etc.

The AI we have today is extremely good at intellectual tasks like Einstein. It is not good at social intelligence and social agency like Clinton. Also, before you say that they can be really empathetic, persuasive, people are already socially dependent on them etc, that's not what I'm talking about. They are extremely knowledgeable about human socialising, but they cannot actually perform it. You could not set an AI out into the world to go and make a few friends.

I think the key moment will be if a single AI can get more than one person to care about what it has to say and what it's doing. Think about today how when someone tells you about their conversation with their AI agent, you probably roll your eyes because who cares, it's a highly individual experience.

Companies need a lot of this. They have to form relationships, become trusted amongst society, etc. AI cannot do this right now, and it's increasingly clear "intelligence" is not sufficient to do it, at least not in the way LLMs are intelligent. Maybe one day they will, but I'd imagine that will be another enormous leap in what AI even looks like akin to how ChatGPT was an enormous leap.