It's also interesting because in humans the existence of "Aha!" moments that are not preceded by or are only loosely related to a chain of thought is taken as the proof of the fundamental mystery and irreproducibility of human intelligence. Now the same argument is made to deny that LLMs actually think. Go figure.
It feels apparent to me that LLMs don't do what is colloquially thought of as thinking.
What is less apparent is that humans do.