My biggest gripe about AI is that very few people actually understand that, and many think that LLMs are "thinking" and capable with "coming up with a novel solution".

They are not. The only reason one might think the solution is novel is because they never saw it before, but what they are actually receiving is an excerpt from someone elses blog post or stack overflow answer. [1]

A bit terrifying thought experiment is to accept for a moment that programming is dead and all its left prompt engineering. Fast forward 5-10-15 years and whos left to actually produce new code and ideas to feed LLMs?

[1] one thing I like to do from time to time - especially when I'm asking for something I know little about - is to copy and paste the answer back to google and look where did that answer originated from.

One time I asked a very specific linux shell command and the answer didn't sit right with me. I googled it and it pointed me to a stackoverflow question. It was the first answer with ~1000 upvotes. But it also had a comment with ~700 upvotes explaining why you never ever should do that. :)