Right. But those still sound like modest gain territory, or qualitative gains within the same rough performance, rather than the "breakthrough" improvement notion I was responding to.

My naïve impression is that the LLM world will keep delivering these fractional improvements for some years at the cost of simplicity. And sure, ontological support seems quite promising.

But making things radically better or faster for small models in the way that is hypothesised, that feels like it can only come as a result of hardware performance improvements and likely architecture changes.

Because there's no free lunch, right? Speculative drafting for example, noticeably improves performance until acceptance rates drop for reasons that have to do with the particular application, and then it starts hurting you, especially near the limits of the memory bandwidth. Because once it is wasteful it is an extra overhead.

I gather Gemma 4 supports, in principle, dynamic speculative draft lengths, to help with this — where it will stop making bold predictions when the success rate falls. But I'm not sure if any of the inference engines I've tested with support that.

I think small models are miraculous — I still think Gemma 4 12B is astonishing — but I guess what I am saying is that I think maybe technology is moving quickly enough that the developers are done with the low-hanging fruit.

[dead]