The animation in the article shows GPT-5.6 Sol Ultrafast solving a question at 3 secs vs Fable's 27 secs. The "11 hour vs 78 hours" animation implied that they were solved in sequence and not in parallel

> Answering 2,500 independent questions is an embarrassingly parallel workload, all it needs is scale out.

I believe the assumption of the benchmark creators is that it's meant to measure the sequential speed that a single instance of the LLM and piece of hardware can get through all the tasks from start to finish, like running a race. As a rudimentary comparison, sort of like doing prime number calculations as a benchmark of the CPUs in one bare metal server. Of course you'd get a speedup in total number of primes searched if you ran the same software of GIMPS on 8 servers with the same hardware in parallel rather than 1 server.

Of course if you took all the individual questions in humanity's last exam and fed them in parallel into separate queries to Claude that land on separate hardware instances of the claude model you'd get a speed up. Because each question is independent and not related to knowledge/calculations that are performed in any other question it is indeed very open to speed up by breaking it into separately dispatched parallel tasks.

>Answering 2,500 independent questions is an embarrassingly parallel workload

So is calculating the total time required to answer all of the questions.

Amadahl’s law dictates it’s approximately always better (as in, more efficient computationally) to have one super fast thing than many slower things doing the same job in parallel.

i assume they meant serially....

750t/s tells the story. This isn't purely (or even mostly) driven by parallelization.

Presumably when they're talking about benchmarking, they're using a fixed amount of resources, like everyone else does when benchmarking?