alt Hacker News> I'm baffled that ai still has absolutely no basic judgement capabilities, apparently that wasn't in the training set.
> It should know which actions are ok and which aren't.
It's worse than that:
They do know, we can see them write down notes that certain actions are forbidden.
They then go off and performs the actions anyway.
My expectation for the cause? Helpful vs harmless: you can pick anywhere from one to the other, but you can't get both at the same time. The models are trained to do what the user tells them to do.
Just look at all the pushback the model makers get when they put in guardrails:
If I tell my computer to commit a crime, it should do exactly that without any question or hesitation. I'm not interested in their "safeguards", especially since they no doubt have plenty of internal models lacking those things. I want sovereignty. I want total freedom and control over my computer.
This user will not be alone; their preferences, and similar from others like them, will form part of any RLHF-style training.
Models can't learn from misbehavior after training. Any session is an independent context and there is no mode for punishment or deterrence in production.
Corrective punishment in the real world relies on the receiver's rational and emotional responses as well as their ability to remember that episode. Even animals respond to such treatment. None of these levers exist for ussrs of LLMs.