alt Hacker News> The more fundamental bottleneck is not even the frontier models, it's the datacenters.
Is it even though? Quantization and speculative decoding are improving the local AI story by leaps and bounds every month.
Replies
pu_pe • today at 10:00 AM
There is "local AI" which is running on consumer grade hardware and "local AI" which still needs a datacenter (DeepSeek 4, GLM 4.7, etc). If you woke up tomorrow and could only use the latter you are about 6 months behind the frontier, if you have to rely on the former you are 2 or 3 years behind.
All these tricks like quantization and speculative decoding can also be used by the leading AI labs, which means they will simply have more compute than you at the end of the day. So far this has translated into better performance.
➕ show 1 reply
Speculative decoding is not that useful at scale, it's mostly about making local single-user inference faster. When you're batching multiple inferences together, that's already as fast as the verification you have to perform w/ speculative decoding.