alt Hacker News> Ever heard of social engineering?
Oh wait, I thought these things were super smart. I didn't expect "social engineering" to work on them.
> models nowadays are way sharper than they were even a year ago.
You are missing the point. If the thing can solve complex math problems and at the same time be so dumb as to fall for "social engineering", then that means that it is not "smartness" or "reasoning" that is helping it to solve those problems. Just some form of advanced, but yet dumb, search algorithm.
Replies
> If the thing can solve complex math problems and at the same time be so dumb as to fall for "social engineering", then that means that it is not "smartness" or "reasoning" that is helping it to solve those problems. Just some form of advanced, but yet dumb, search algorithm.
I'm not just trying to be snarky, but I have no idea how to read this without taking the implication that humans are advanced, yet dumb, search algorithms.
By "heard of social engineering?" I meant that humans are vulnerable to malicious input too. Prompt injection is basically a simplified form of social engineering for language models. It looks different because models operate over much smaller and more explicit contexts than humans do and are explicitly trained to follow instructions, but the general idea is similar: malicious input tries to manipulate how the system interprets trust and instructions. This is why we need protocols, permissions, and opsec for both agents and humans. That said, I’m not criticizing how you choose to use, or not use, these models, though.