alt Hacker NewsAnything you can verify to be right or wrong can be done by a model. All benchmarks will be saturated
Replies
jdthedisciple • yesterday at 10:44 PM
Yes, but not necessarily under tight budget constraints.
➕ show 1 reply
Anything you can verify to be right or wrong can be done by a model. All benchmarks will be saturated
Yes, but not necessarily under tight budget constraints.
Disagree.
Examples:
- predict a coinflip: easy to verify, hard to learn
- earn $100: easy to verify, hard to learn
- increase paid subscriptions in an A/B test: easy to verify, hard to learn
I won't get into it, but there are many properties beyond verifiability that are needed to saturate a benchmark.