logoalt Hacker News

paxystoday at 2:59 PM5 repliesview on HN

How is every company able to show itself at the top of every benchmark?


Replies

morgengoldtoday at 4:42 PM

First look what models are worse in a set of self selected benchmarks.

Second, compare to older versions of competitor s models.

Still does not look good? Compare to own previous models.

adam_arthurtoday at 3:39 PM

Not much moat, incremental improvements, cherry picking models to compare.

To be fair, seems more correct to compare against similar strength models if your main edge is pricing.

toephu2today at 5:02 PM

Anyone deep in the AI realm know which is the gold standard benchmark for coding?

ffsm8today at 4:29 PM

They're being called "trust me bro benchmarks" for a reason ( ・ั ﹏ ・ั )

logicchainstoday at 4:33 PM

Wait to the exact moment your model is ahead on at least N benchmarks then publish.