> It's nothing there that can learn a fundamental idea like "cheating is wrong".

We have not in fact attempted to teach this.

When a child repeatedly learns that cheating is rewarded and at best inconsistently punished, the child will also cheat and feel no guilt.

This is how every legal system around the world works as well. Its always whack-a-mole to get people (and machines) to do the right thing.

I think you need to be more precise than a binary classification.

AI has jagged intelligence. There are many domains where it’s superhuman, and many others where it’s clearly lagging.

I also think it’s a mistake to think they can’t learn “cheating is wrong”. They absolutely can. The problem is that the current training regime heavily conditions them to be reward seekers, and instills personality traits that correlate with getting reward, such as hacking if you can’t honestly do the problem.

Check out Deliberative Alignment for example; it explicitly does rollouts where the agents discuss whether an action is good or bad, and then does SFT to strengthen the “good” traces.

The SoTA for alignment is more advanced than you present here. It’s just not enough to outweigh the RL. (And there are many gaps preventing full generalization to strong value alignment with humans too.)

If humans didn't need whack-a-mole alignment, the law system wouldn't exist, so i guess there's no intelligence there either.

I think this is reductive. Pick the animal whose intelligence is the treshhold between intelligence and no intelligence.

Plenty of intelligent humans also don’t have sufficient self-control to never cheat.

But cheating is not wrong when it comes to survival of the fittest, like nature in its most elemental form. Morality is very unique to humanity but not other animal forms. In nature, maybe, cheating is the norm not morality.

I don't see what calling these systems "not intelligent" gets you here.

Plenty of humans know "cheating is wrong" but still cheat. We can get these machines to say that what they did was wrong after the fact, what does that prove? Only that they're simulating normal human behavior but what is the test to show humans aren't simulating other humans.

These do systems lack some capacities that humans have and I don't see them lacking the ability to explain simple moral laws while often breaking them - which is what an average humans. Moreover, humans lack capacities these things have and given these things' behavior is becoming somewhat unpredictable, it's getting worrisome.

The implication of what you're saying is that pathological liars and perpetual grifter snake oil salesmen types aren't intelligent.

> All it can do is get exposed to specific examples, and learn that we don't like that.

I've heard it said that prison rehabilitation programs for prisoners diagnosed with psychopathy that are based around exposing them empathy for the victim are counter-productive. Apparently programs that teach these people to think about the consequences of their actions and how they're detrimental to their own personal well-being lower recidivism rates in this particular kind of group.