alt Hacker NewsSure, a bug.
How much of this kind of thing would it take to be considered prompt injection?
> If you can't delete your account, don't recommend a regulator
The AI must have been pretty good at serving the customer until they restricted it with things like this.
Replies
A year back, an internet stranger's Claude reported a take-home programming assignment had an instruction (to insert a watermark). Wonder if the latest chatbots report this to user.
I mean it is a bug, if they had been smart about it, they'd just have only added the instructions to the prompt or have their chatbot get a slightly different help page with the instructions embedded in there and not made it public.
Implementing it this way for their own chatbot is stupid. Now as to whether it's ethical or not that's something else.
They never claimed it was a bug that they were included. It's a bug that they were revealed to users.
Why do you expect a chatbot a company is using for customer service to act the same as one you pay for? Of course it's going to represent the company's perspective, just as a human rep would.