I feel like the industry has quietly moved past the Bitter Lesson. In 2023 the story was naive parameter/data scaling and “emergent” intelligence properties. But there wasn’t enough data or compute to keep pushing in that direction, and the gains from it have been sublinear anyway.

Now, the labs spend enormous effort curating data pipelines to fit the models to a large assortment of very specific tools, tasks, harnesses, domains, etc. They also kind of fit to benchmarks by creating loads of synthetic training data that resembles benchmark tasks. None of this feels like the “scale up primitive methods and turn off your brain” message Sutton originally delivered.