It's hard to say for sure because the closed labs don't share anything about their improvements. However, if you look at DeepSeeks papers, most of the improvements come from:

* Better RL environments that themselves are generated by agents

* Architecture improvements that increase effective capability per parameter or FLOP

If you're curious about the latter I'd look into manifold constrained hyper connections, engram, deepseek compressed attention, kimi delta attention etc. At first these might seem like just speedup tricks, but several of these allow the overall capacity of the model to increase per training FLOP or parameter. This then lets pretraining & the RL environments grow & be more effective.