alt Hacker NewsIsn’t the point of ARC that you can’t train against it? Or doesn’t it achieve that goal anymore somehow?
Replies
theywillnvrknw • yesterday at 6:42 PM
* that you weren't supposed to be able to
Isn’t the point of ARC that you can’t train against it? Or doesn’t it achieve that goal anymore somehow?
* that you weren't supposed to be able to
How can you make sure of that? AFAIK, these SOTA models run exclusively on their developers hardware. So any test, any benchmark, anything you do, does leak per definition. Considering the nature of us humans and the typical prisoners dilemma, I don't see how they wouldn't focus on improving benchmarks even when it gets a bit... shady?
I tell this as a person who really enjoys AI by the way.