alt Hacker Newsit's well documented that models can be adversarially trained with essentially backdoors in response to special inputs
while I am skeptical that this is happening atm, there are probably many industries where the risk does not seem worthwhile
Replies
Giefo6ah • yesterday at 8:04 PM
When the model is open weights you can even pass every token (including the chain of thought) though a fourth-party lightweight model like gpt-oss-safeguard to check that it has not become adversarial.
selectodude • yesterday at 7:56 PM
I feel like that's a threat that isn't super difficult to block. Unplug it from the internet, require it to go through an API intermediary to access web pages.
Maybe I just don't have any imagination.
➕ show 1 reply
I suppose this is like when Anthropic was using “prompt modification, steering vectors, or parameter-efficient fine-tuning” to poison the work of people working in the LLM field, including academic researchers.