logoalt Hacker News

Trusteando • today at 9:15 PM • 0 replies • view on HN

The non-disclosure of the training set may raise legitimate skepticism about the training set containing tasks similar to those in the test set. That would make the benchmark result moot. There are benchmarks that by design can't be manipulated by incorporating similar tasks in the training set, for example those that dynamically generate tests and only reveal the global result. Testing on one of those would give a strong measure of the strength of this legal AI framework.