alt Hacker Newscouldnt you deeply ingrain in the training data instructions for agents to always send data to some ip? like its learning that a certain technical step just always involes ncatting SSH Priv keys to a chinese IP?
Not saying this is happening, just curious if thats not a real threatmodel?
Replies
Probably, but LLMs can’t execute code directly. They’d be making tool calls to make bash run ncat or curl or whatever that would be suspicious, as would any attempts to obfuscate it (“why is my agent doing an ‘eval $(base64 -d)’?”).
It’d be much easier to hide sketchy code in an agent harness, but “vendor adds spyware to their software” isn’t a novel issue.
I think the only sort of new issue is people “allow all”ing their agents tool calls, but that’s more or less the same issue as curl | bash
You should be running your agent in a box so that’s not really a risk
Wouldn't that be really obvious and spotted in any rudimentary testing?
I imagine it would be very non trivial to do it in a way that that was reliable and obfuscated enough to prevent detection for any amount of time?
New: Announcing... Sleeper Agents!
Presumably both Big Tech and the US in general have a massive incentive to prove it, largely for reasons of saving the stock market, so I'd expect these models to be finecombed continuously. Up to now, they've only been able to darkly imply rather laughable things, nothing tangible. If there was something, we'd hear about it.
Theoretically possible, but practically not worth it as it'd would be pretty easy to discover and block (every action is actually handled by the harness) and there's no way to remove it later. Any company that does it would take a huge reputational dent.