> I don't think that follows. This is just LLMs being, for a lack of a better word, "gullible." How is it different from a person believing whatever they read on the Internet? People fall for spam and scams all the time, doesn't mean they are just glorified searches ;-)

The important difference is the AI has been mass-produced and commodified at low cost.

If you scanned my brain, uploaded and ran me as a simulated mind, no matter how good the simulation was, the ability for an attacker to try a million variations to see which one slips past my cognitive blind-spots would enable them to convince me of, if not literally anything, a lot that would normally never be so.