Basically RLHF sometimes favor "automation", and if it is not rewarded properly it will go over the line. Models are a bit more difficult to teach because their mechanism of "boundary" is quite fragile and different than human's.