alt Hacker NewsThe obnoxious/confusing thing is that models are released individually rather than as a family.
Luna < Terra < Sol < Astra, fine. But does Terra 6.1 beat Sol 6? Does Fable 5.1 beat Opus 5.5? If I build something around this Terra 6.1 release, then when Luna 6.2 is released in a month, should I swap?
Replies
Run your own evals then if you have this much of a need to "correctly" choose. That'll make it clear.
I'll just stick to mostly Sol on low/medium/high and start experimenting with Terra tomorrow onward to see what it does noticeably better than Luna/Deepseek flash.
Any pareto frontier model, especially from the leading labs is guarnateed to be good enough for whatever your expectations are at this point. A quick few day test should be enough for anyone to make their own conclusions.
> Luna < Terra < Sol < Astra, fine. But does Terra 6.1 beat Sol 6? Does Fable 5.1 beat Opus 5.5? If I build something around this Terra 6.1 release, then when Luna 6.2 is released in a month, should I swap?
I understand how it can be confusing, but I don't think that this is exactly a problem.
Luna through Astra offer qualitative descriptions of the model's size. Presumably OpenAI trains its biggest model (Astra-ish, maybe even larger) first, then distills it to smaller sizes.
Verison numbers reflect something about the architecture or training dataset. Again, they keep these details close to their chest, but we should expect that weight-for-weight higher versions should beat lower versions.
This offers us two axes of 'better', but these axes are also not directly comparable. An earlier but bigger model may or may not beat a newer but smaller model. This is where 'spiky' intelligence comes in.
This is also where calls to 'run your own evaluation' really do start to matter. If you care enough about this performance[1], then you also have specific problems that you care about. No generic benchmark can fully answer that question, hence the need for a specific evaluation.
[1] — And you don't have to! A casual user is probably served reasonably well by anything put out in the past six months. I think Microsoft is still serving GPT-4o under some Copilot branding.
I understand the confusion, but isn't this kind of similar to asking does the iPhone 14 Max beat the iPhone 16 Mini? At least with phones there are hard specs we can use to compare, but in a lot of cases the answer will be "it depends"