First, there are plenty of auto-verifiable problems like ML benchmarks datasets, benchmarks on hardware / software performance for coding tasks, and lean certificates for math, e.g.

Second, I think RSI will not be used like N -> N+1 where N is the frontier, since doing those experiments at scale is too expensive. My guess is they are doing N -> m_{1,…,99} where m << N, and then taking the insights from that and trying it at successively larger scales.