alt Hacker News"Opus 5 states the hidden rules before its first action, then plays a byte-identical optimal solution in 5/5 seeds at temperature 1.0. Zero exploration."
I guess there is no way this can happen without benchmark being part of the training data??
Replies
modeless • today at 6:16 PM
They state the puzzle is "Witness-like" which I assume means that it follows the rules from the well-known puzzle game "The Witness" which Opus definitely knows.
stogot • today at 1:15 PM
It may read information about the benchmark, such as on blog post, or Twitter feeds (example OP) without active cheat
What seems to implied is that some of his hold out testing suite includes simple/common tests that are out in the wild, and for those opus went straight to a memorised solution .