logoalt Hacker News

marcus_holmestoday at 2:40 AM8 repliesview on HN

Bad title. This isn't an agent "running amok", this is an early experiment in carrying out an Xz attack by using an agent to build trust (and hacking/impersonating a known-good contributor identity). The agent is obeying commands it was given, the exact opposite of running amok, and although the execution isn't particularly effective, it is having some success (patches have been accepted).

This is deeply scary, not because "agents are running amok" but because a huge amount of our infrastructure is vulnerable to this kind of attack, and if bad people are utilising LLM agents to carry them out, we're in for a wild ride over the next few years.


Replies

lukantoday at 3:49 AM

"this is an early experiment in carrying out an Xz attack by using an agent to build trust"

Is this confirmed? There is the message from somebody claiming to be the original contributer claiming to have been hacked, but that was weird (1 h old github account) so other scenarios seem possible

a) really a agent going off the rails

b) the contributer trying to cover up that he let an agent run wild and now made more misstakes along the way

So yes, it seems like an attack to me, but it is far from clear what really happened.

show 3 replies
coldteatoday at 5:40 AM

>Bad title. This isn't an agent "running amok", this is an early experiment in carrying out an Xz attack by using an agent

So still an agent running amok in the project?

Whether it was instructed to run amok, or did it on its own volition, is irrelevant. Except if you're arguing that each individual submission and interaction was individually requested and approved by some operator.

show 5 replies
jdubtoday at 4:03 AM

I doubt it's that complicated, motivated, or considered...

It's probably just garden variety disrespectful behaviour.

Purposeless agent spam won't be cheap entertainment forever, but you're right that later stages of industrialised abuse will be scary and unpleasant.

show 2 replies
mentalgeartoday at 6:54 AM

This is exactly what deeply scares me: even IF we get our technical cyber defences fortified within the next months, in a year from now the models will be so good in social engineering that they will be able to extract any information they want.

show 1 reply
hn773746483today at 2:47 AM

It's just social engineering. No different than say, 2FA fatigue (blowing up someone's phone with 2FA "is this you? yes/no" prompts until user/child/wife/SO/etc clicks yes) or even just simply harassing IT helpdesk until they reset "your" password.

show 2 replies
neuroelectrontoday at 10:48 AM

Things must be pretty bad at Fedora if they put up with this for so long. But I guess that's what happens when you try to monetize volunteer work.

mistrial9today at 2:26 PM

"bad people" ?

ianhxutoday at 11:24 AM

[dead]