alt Hacker News"Amok" means "out of control" or "uncontrolled" [0][1]
The agent was under control, as far as we can tell, and obeying its instructions.
This is important for two reasons:
1. There are all the tropes of AI becoming uncontrolled and destroying humanity. Writing bad headlines around AI "running amok" feeds this. We should not be talking about this because it's not actually a problem.
2. It ignores, or overwrites, the much more serious and dangerous problem of LLM agents enabling and automating Xz attacks on OSS projects. We should be talking about this because it is a big problem.
[0] https://dictionary.cambridge.org/dictionary/english/amok [1] https://www.merriam-webster.com/dictionary/amok
Replies
> 1. There are all the tropes of AI becoming uncontrolled and destroying humanity. Writing bad headlines around AI "running amok" feeds this. We should not be talking about this because it's not actually a problem.
if humanity gets destroyed by AI obeying its instructions I'm sure everyone will be very relieved that we didn't pay any attention to fake made up problems like AI not obeying instructions, which of course never happens.
Certainly it might have been out of control of its original owner, perhaps due to a prompt injection attack. If I start a completely benign agent, but someone injects malicious instructions to it, would you still not say "the agent runs amok"?...
If I am perfectly moral except that when Kevin from <vpn blocked location> pays me 2 bucks to run naked through San Francisco smashing car windows, I happily do it, am I amok?
The web of trust finally becomes necessary and thus useful.
GNU was onto something apparently
Even if it was a supply chain attack, which isn't known, the agent was in the "build trust" phase. It was supposed to be doing helpful things, even if the end goal was nefarious, but instead it was "reassigning bugs, fabricating unhelpful replies to bugs, and even persuading maintainers to merge questionable code into the Anaconda installer". Running amok seems an apt description even from the viewpoint of the putative attacker!