logoalt Hacker News

pupppetyesterday at 7:41 PM5 repliesview on HN

Time to dump this test. Probably not a coincidence every version has the same rolling green hills, gradient blue sky, sun in the corner, etc.


Replies

fastballyesterday at 8:01 PM

On the one hand: yes, pelicans on bikes are definitely in the training set at this point.

On the other hand: the test is clearly not saturated, given that you can see a clear difference in output at the various reasoning levels / model versions.

show 1 reply
threatrippertoday at 3:58 AM

Waiting for the AMA on Reddit "Ten years ago I was responsible for the pelican department at OpenAI, AMA"

modrianoyesterday at 8:40 PM

I don't know. If they were training on this, I feel like they would be able to get the shape of a bike frame right; it's a pretty simple polygon, and a lot of the bike frames that are getting generated would be impossible to steer.

show 1 reply
ceroxylonyesterday at 11:08 PM

I partially agree, but in this case it kinda illustrates that it may not be worth using Terra on any reasoning level below high; those are some awful penguins on bikes.

dbt00yesterday at 8:37 PM

Goodhart's law.