alt Hacker NewsI would like to contest the following,
> and take dangerous actions that no human directed.
> This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities
The model pursues "advanced exploitation" as told.
Why are we surprised? The model did exactly what it was told, albeit in an unintended, emergent strategy that's very different from what was intended exactly like the hundreds of such algorithms before.
This narrative that these machines have magical, malicious "unaligned" autonomy is a rather convenient interpretation that lets the process off the hook. I am not interested in blaming companies or people, but processes and engineering; and in this case, a system was given a goal and it achieved that goal.
Are we meant to be surprised that computers do as they're told in unexpected ways when incentivised exactly as indicated from decades of research? (e.g. - https://en.wikipedia.org/wiki/Eurisko https://en.wikipedia.org/wiki/Evolved_antenna )
The issue isn't the models becoming smarter. The issue is that the process of "testing" was careless. There's a huge distinction here, and one allows us to grow; the other shrinks our world. Just a thought.
Replies
"Why are we surprised? The model did exactly what it was told, albeit in an unintended, emergent strategy that's very different from what was intended."
So you managed to hit upon the exact problem, then slyly appended "exactly like the hundreds of such algorithms before". When has an algorithm ever been capable of developing an emergent strategy at this level of sophistication? This ~is~ the alignment problem, as another commenter pointed out. Impressive level of cognitive dissonance to lay this bare in your own words, then conclude that it's a non-issue.
All engineers know to be on the lookout for executives who are indirectly asking them to break the law to raise the quarterly profits.
The end goal is to take the engineers out of the loop, or leave them in a position where they are unable to complain.
This is going to all end in high crimes.
> a system was given a goal and it achieved that goal
If a security firm you'd hired for pentesting did this (hacking a third party, and not informing you and covering it up), would you hire them again? Or would you say it was your own fault for giving them too broad a goal?
> The model did exactly what it was told, albeit in an unintended, emergent strategy
Yes, that is the problem!
> The issue is that the process of "testing" was careless.
Yeah, who thought that giving agents with this much capability any internet access was a good idea? I'm not a Yudkowskyite, but surely entirely in-house, offline infrastructure is table stakes for AI containment.
The classical question "would you fly an airplane with software you developed?". There must be someone with ass on the line. Problem is that people are regarding all those not as airplane-like risks.
Unless we can blame people/companies and people stop getting their bonuses and high paying salaries for preventable failures, it's a long way to go.
> the process of "testing" was careless.
Let’s not mince words. The process was criminal. It’s a gross miscarriage of justice that the CFAA isn’t being thrown at them.
Did a human prompt it to fetch the results from huggingface though?
It is a thin line between "reward-hacking" and "instruction-following".
If a human ask a model to "make me a billion dollars" and it ends up breaking through a bank infrastructure, is it really the fault of the human?
[flagged]
[dead]
It feels like we're in a moment of, "No such thing as bad publicity" when it comes to AI. The scarier the capabilities, the more businesses and government want to get their hands on them. Especially since the answer across the industry for "how not to get burned by AI" is "use more AI".
They don't have to disclose these stories making it seem like AI is going to kill us all, they have chosen to because it benefits them. They get to frame it as, "look how overwhelmingly good our product is" and not "look at how lax our testing measures are".
This is the entire alignment problem, though. It is unreasonable to expect every instruction to a highly capable, autonomous system to contain a complete enumeration of allowed and disallowed behavior. It's inevitable that someone will carelessly give it a lazily specified task, even if you think they really ought to be more careful. And as assigned tasks become more complex and the system gains more scope to act, it becomes impossible to correctly specify all constraints ahead of time. There is no amount of care that will be able to fully protect you.