alt Hacker NewsI've been saying efficiency is the next "frontier" in AI, at least for LLMs that people use daily. Companies have started to really balk at token costs from the major providers, and there's some evidence that the cheaper Chinese models are chipping away at Anthropic/OpenAI dominance from below (as cheaper Chinese products have done in other industries for many years). I continue to think that the vendor that figures out how to efficiently serve a Good Enough model at a much lower price will be the one to win in the end.
DeepSeek is remarkably efficient at caching and their cached token rates are crazy cheap; using it with Reasonix is free real estate, like 97% cached tokens, ends up costing like 30 cents an hour to use DeepSeek V4 Pro. I hadn't dug into MiMo's caching behavior as I haven't used it as heavily as DeepSeek, but this indicates it's close to DeepSeek.
At this point I don't see a reason to use Sonnet, Haiku, or the smaller GPT models, because their API rates are much higher than the best models from MiMo and DeepSeek.
We're still figuring out the upper bounds of capability and I am still finding next generation models are unlocking things I couldn't readily accomplish before and I'm willing to pay more for them (at least, I'll pay the $100 or $200 subscription rates for them, I couldn't justify the token expense for most of my dev work), but we're already at a point where someone building standard CRUD web apps doesn't need the top models and probably doesn't benefit much from using them.
I agree, but maybe for different reasons. I think Karpathy is right. We need models that reason, not models that memorize.
Karpathy calls it a "Cognitive Core", and it's essentially a small model that learns to reason and look up the data it needs as opposed to a giant model that memorizes all the data in the world and tries to process large chunks of it all at once with every thought. I think it will be based on the thing that grokking, the lottery ticket hypothesis, and the universal weight subpspace hypothesis all point to.
Eventually someone will figure out how to build it and the entire economy that we've now built on top of the wacky idea that nothing can possibly ever get more efficient will collapse overnight.
Sometimes I wonder how much Nvidia would pay someone not to release a thing like that, and then I wonder if that's already happened.