logoalt Hacker News

vunderbatoday at 1:59 PM0 repliesview on HN

Haha yeah, the site automatically assigns the term to any benchmark that fewer than 25% of the tested models are able to pass.

What’s more surprising to me is that, unlike the “pelican riding a bicycle” whose objectivity has been slightly compromised as newer models have incorporated it into their training data, the arbitrary-point star has been wiping models out ever since the early days of Flux back in 2024.

I personally love the test because it's something that even an elementary school child with no artistic experience at all can do, but state of the art models struggle heavily.