More compute/larger datasets during training != larger models. The Bitter Lesson was that just scaling things up beats custom hand-crafted optimizations. Up until 2024, we thought that meant scaling up the parameter count, but then that started to plateau. After o1 was released, we thought it was about scaling up test-time compute. Now, seeing how 30b models can easily outperform 200b models from a few years ago, it seems like what we need to scale is RL, at least for agentic capabilities. It looks like 30b is already enough for good agentic capabilities. Larger models aren't considerably "smarter" (especially since they're mostly MoE anyway, with something like the same 30-50b active parameter range), they just know more (better world knowledge), which lets them make more informed decisions. Maybe we just need to scale up the retrieval layer.