alt Hacker NewsThis is not true. Think of the problems AI has with creating appetizing food imagery, for example. Images that every human annotator marks as unappealing with a high degree of consensus.
The problem is in the rewards, but it's more how they average across domains and contexts, so AI misapplies them. Actually, in human preference evaluations, most of Gen Z in particular extremely adverse to the common look of AI images.
So it's not the annotators involved in RLHF, it's the models failing to map to human preferences at the task level. The failure rate is also uneven across domains.
Where there are challenges with human annotation, it's not the ability to gauge what's good but rather how to correct what's wrong. Most people can tell when something is off but struggle to articulate why. That's where more domain expertise is required.
Replies
>This is not true. Think of the problems AI has with creating appetizing food imagery, for example. Images that every human annotator marks as unappealing with a high degree of consensus.
I think the way they'll solve it in the end is to create an ultra-high-fidelity simulation of the world including modelling taste and other senses on the level of physics, and then incorporate all of this into the training rewards.
I think this is how the machines know what tasty wheat tasted like.
> Think of the problems AI has with creating appetizing food imagery, for example.
For posterity, the memetic ugly food can be controlled and mostly filtered out with prompt engineering. Here's a quick ad hoc test example which (hopefully) uses the new model: https://chatgpt.com/share/6aebf030-9d38-83e8-a103-3509883905...
First image is fine, but attempts to make it more wacky turn it to the AI food slop. So then followup prompts deslopify it, mostly successfully.