alt Hacker NewsTo the best of my knowledge (which, admittedly, is far from comprehensive), that kind of attack on a model shouldn't actually be possible in a deterministic fashion.
If you can't stop them from sometimes telling customers things like "yes, I will give you a penthouse suite at our hotel for only $3/night", why would you be able to guarantee that, with some specific set of tokens, they would produce a perfect and undetectable backdoor customized to the code at hand?
I don’t think it has to be perfect or deterministic that way. It’s enough to have a bias towards implementing a backdoor in some circumstances. Something like, if the machine seems to be used in a Chinese environment, the model is biased towards missing some security issues, or towards implementing the type of bugs that can be used for an RCE, or similar.
Anthropic has done such checks at the agent level: https://cybersecuritynews.com/anthropic-claude-hidden-code/
Their excuse was defending against distillation attacks but you can see how that can be abused