I had a similar thought -- rather than fixed benchmarks, you want dynamically-generated tests, specifically designed to exercise newly-exposed corner cases. So the way forward might be antagonistic benchmarks generated by another LLM.