ben_w • yesterday at 11:03 AM

> I'm baffled that ai still has absolutely no basic judgement capabilities, apparently that wasn't in the training set.

> It should know which actions are ok and which aren't.

It's worse than that:

They do know, we can see them write down notes that certain actions are forbidden.

They then go off and performs the actions anyway.

My expectation for the cause? Helpful vs harmless: you can pick anywhere from one to the other, but you can't get both at the same time. The models are trained to do what the user tells them to do.

Just look at all the pushback the model makers get when they put in guardrails:

  If I tell my computer to commit a crime, it should do exactly that without any question or hesitation. I'm not interested in their "safeguards", especially since they no doubt have plenty of internal models lacking those things. I want sovereignty. I want total freedom and control over my computer.

- user matheusmoreira, here, 13 days ago: https://news.ycombinator.com/item?id=49678048

This user will not be alone; their preferences, and similar from others like them, will form part of any RLHF-style training.