I’m not convinced removing RLHF would really make the probabilities generator give us distributions that can diverge from the mean while remaining useful.

In other words, this might not a problem that can be overcome in LLMs alone.