I find this fascinating because - let's say that I want to simulate the scenario for myself that a language model finds itself in. I'll put aside my deep dislike of anthropomorphizing AI for a minute.

So if I'm an LLM, the only "sense" that I have available to me is the incoming stream of tokens. I can emulate that by forcing myself to imagine evaluating incoming requests by putting myself in a completely empty room with an unlimited supply of blank paper, a typewriter, and a mailbox slot.

Incoming requests (aka "context") would enter into the mailbox slot, as sheets of paper with printed content all consistently formatted in monospace font. I would collect the paper, evaluate the request, and type a response with my typewriter - feeding the reply back through the same slot.

When I imagine this, I wonder - what signals could I possibly use to determine if an incoming request is "legitimate"? I could, for example, type some response back to ask clarifying questions. But with such a limited input space and no ability to reach "out of band" of my current context, how could I evaluate the truth of what I receive back?

In the "real world" of course, I have five senses to rely upon. Critically, I have the ability to collect additional context "out of band" of the conversation and interact with unrelated entities to either confirm or refute claims. And as others have pointed out, even the addition of more signal doesn't make humans immune from social engineering attacks.

Anyway, I find this to be an interesting mind experiment to kind of imagine myself as an LLM, especially to explain to lay-people some of the challenges in "aligning" AI systems ("why doesn't it just do the right thing?")