This rhetoric (and it is rhetoric, in that it is designed to persuade) too often lacks nuance. Generating a 3 minute stand alone video is impressive, but it is a different task from generating code that must be consistent with an existing mountain of code. Generating code that implements a straightforward feature within an existing system is different to generating code that must make architectural decisions in novel domain or with a novel architecture. The Navier-Stokes result is impressive but 1) they spent a bajillion tokens / currency units to achieve it and 2) AFAIK all the recent mathematics results have been counterexamples, which are a different beast to proofs.

My own experience, pootling around with Claude like a peasant, is that it is simultaneously amazing at some tasks and an absolute moron at others. The more abstract the reasoning required, the worse it is.