alt Hacker News“We ran 153 autonomous runs across 18 frontier models on the nanoGPT optimizer speedrun.”
Uh.. okay.. but whats a run… read blog
“We want to measure how well frontier models can conduct research….””we ran 153 autonomous runs on the nanoGPT optimizer speedrun across”
Okay but what is a optimiser run and what connection does it have to being good at research?
“For comparison, Anthropic's internal automated AI R&D evaluation optimizes a model on a CPU node,”
So I should go look what Anthropic was doing to understand?
Why not just explain what it means in their blog..
Replies
So maybe this is a simple way to put it..
They gave 18 frontier models the task of “researching” how to improve a lab-rat nano model’s training. Stopping when it met a quality goal of a target loss rate. During each autonomous research session, the AI repeatedly tried changes, tested them, and used the results to decide what to try next. They repeated the whole research session many times with different seeds to average out variance.
I think that's explained here:
https://www.primeintellect.ai/blog/measuring-autonomous-rese...
Basically they do 8 runs trying to optimize to under 3.28 loss in the fewest training steps possible under time/token constraint. I dunno why 18 * 8 != 153 (it's 144)
Yeah, I'm with you on this, I think this is just what fable/opus-5 slop looks like now...
- "A frozen verify.py accepts the claim" (what does it mean to freeze a python script?)
- "which trains the recipe eight times on fixed seeds it can't touch" (what does it mean to not be able to touch a seed)
- "One other detail is that we gave an estimation of the speedrun noise in program.md that was slightly too large. 62 out of ~100 runs measured it themselves instead of trusting our number" (What does it mean for a "run" to "distrust" a noise measurement)
- "One important disclaimer is that our benchmark has a lot of variance" (Actually this one makes sense, but congratulations for burying the lede that your entire article is bogus.)
- "Almost every model finds the same winning ideas. What separates the best traces is what an experiment leaves behind" (This implies the the graph would show every model finding a plateau in whatever metric the experiment is measuring, but I don't see every model scoring the same in the graph)