logoalt Hacker News

esafak • yesterday at 2:58 PM • 1 reply • view on HN

They updated their methodology when the new openai model didn't rate higher than the old one, or something like that.


Replies

mchusma • yesterday at 8:44 PM

I do t think it was a fiasco. I thought the updates were good. They should do more work on benchmarks. There has been good work on testing benchmarks for example to ensure they are accurate, and as issues arise they should change the benchmarks. The point of AA and Arena is trying to understand how to think about models.