I feel like RLHF has a pretty obvious ground truth, human feedback is used as an (albeit noisy) signal of average human preferences. Same thing with RLVR and "solving the problem".