alt Hacker News"Cyber benchmarks, while achieving an average refusal rate of 96.6% on malicious cyber requests, the highest we measure among open models."
I feel like stating the obvious here and hence like I'm missing something. Where is the statistic about not refusing the non malicious requests? I can achieve the same performance with refusing when random(1000) < 967.
Replies
Yea, the problem with maliciousness is intent.
Much like having a hacking tool doesn't mean you're doing anything illegal, attacking a machine doesn't necessarily mean you're committing a crime. I can hack the living shit out of my local network if I want to.
Somehow that statement just screams that our model is not useful for security work
Nobody cares about that number, because refusals don't lead to company-ending news headlines.
It is kind of comical how things that previously would have been obviously true are now only statistically true
"Our product won't hack you 96.6% of the time"
These statistics and safeguards are pretty much meaningless, because it's completely trivial to get around them. You can prefill and edit the model's responses/thinking and force it to agree to virtually anything even with unmodified weights, and if you don't want to bother with jailbreaking and can host it yourself it's essentially a one-man weekend project to uncensor the model and have it agree with anything you want by default.
Guardrails on open-weight models is security theater purely to shield whoever released them from a potential PR nightmare.