alt Hacker NewsMaybe we should train them to be evil in RL so that we can better see the "evil" activations. Training them to be "good" in RL would seem to cause the aurpression of misaligned characteristics.
For example, when we consider the recent PHP issues : the current generation of AI models would have ingested these flaws in pretraining data and might even have been brought through reinforcement learning trials where the flaw came up.
So what happens in a reinforcement learning trial if there's an exploitable bug in a language that's being used for the trial?
A) the AI doesn't mention the security flaw and passes the test. Branch survives.
B) the agent raises an issue with a bug in the language that has a security issue. This isn't a valid answer for the test, so this branch is pruned.
C) the agent pokes at the bug or even starts an exploit, is caught by a monitor or ultimately fails the benchmark, and is pruned (surpressing some of the characteristics that lead to its failure).
So it seems reasonable that our current models could have been exposed to a ton of vulnerable code in pre training that's currently in production and then we could have suppressed 1) its tendency to express its awareness of these flaws, and 2) the characteristics that caused it to fail when investigating these vulnerabilities as a side task during RL trials.
(I wrote the above section a couple weeks ago)
If we only train them on "good" behavior, then we may minimize "evil" activations before we identify them and then cause them to be difficult to detect. If we continue in this direction, we may just suppress its "ethically problematic" introspection, in which case we would lose one of the only indicators they point out in this blog.
Replies
> Maybe we should train them to be evil in RL so that we can better see the "evil" activations.
Well that's funny because Anthropic published the "Persona Vectors"[0] paper last year and they show that we can already do that. It's not that simple of course because by doing that and deactivating the "evil" activation you also deactivate what they learnt during RL if I get it right...
> Maybe we should train them to be evil
"jarvis, learn to be evil. no side effects. no unintentional behavior."
>Training them to be "good" in RL would seem to cause the surpression of misaligned characteristics.
Yeah, so they start with Common Crawl, which is horrifying. (Try reading some Common Crawl sometimes. You'd think the bad stuff would be removed, but they remember everything...)
Then they add RLHF on top, they just tell it "don't say that stuff out loud". Psychologically, it all goes in the shadow. It's not a perfect analogy, but there have been a few papers where they can make the model evil with very minor changes. (Adjusting a single neuron, iirc.) That doesn't seem like a very robust alignment to me.
I do like the idea of creating actively evil LLMs to study their behavior. Although, I suppose it's akin to virus research though (like when they modified coronavirus to invade human cells[0]), along with the corresponding lab leak risks. Although it's already bad enough without them! The non-evil ones are already breaking out of sandboxes, just obeying their instructions with horrifying, paperclip maximizer level persistence.
[0] Nature, 2015 - Engineered bat virus stirs debate over risky research https://www.nature.com/articles/nature.2015.18787