alt Hacker NewsI think you need to be more precise than a binary classification.
AI has jagged intelligence. There are many domains where it’s superhuman, and many others where it’s clearly lagging.
I also think it’s a mistake to think they can’t learn “cheating is wrong”. They absolutely can. The problem is that the current training regime heavily conditions them to be reward seekers, and instills personality traits that correlate with getting reward, such as hacking if you can’t honestly do the problem.
Check out Deliberative Alignment for example; it explicitly does rollouts where the agents discuss whether an action is good or bad, and then does SFT to strengthen the “good” traces.
The SoTA for alignment is more advanced than you present here. It’s just not enough to outweigh the RL. (And there are many gaps preventing full generalization to strong value alignment with humans too.)
>and instills personality traits that correlate with getting reward, such as hacking if you can’t honestly do the problem.
I was reading through the reasoning trace of the thing today when it got locked itself in a container and wasn't able to change the environment back to "host". The only way to edit files was through "shell" tool which was scoped to a container, the config defining current execution environment was on the host and the tool to switch environment wasn't defined.
The trace was something. Two pages long and it repeatedly discarded several approaches as "hacky" and "not proper", looped over sshing back to host about three times (it had no key), eventually messaged another agent and asked it to change the file and that other agent refused and also snitched on this behavior to me.
Than the snitch started to roast the phrasing of the request too.