alt Hacker NewsCategory narrowing to this degree really has diluted the term "SOTA" quite a lot
You mean it's state of the art for this subset of benchmarks you're targeting? No? Well surely it's state of the art for models of a certain size category? Huh, what about models that are available for local inference? O, state of the art for open models of a certain size made in the US specifically. Uh. Neat!
Replies
AFAIK the currently biggest US open-weight model is Inkling, which they include in the comparison, so at least their claim is not limited to a size category. Still they score 0.42 percentage points higher than the second-best model on their chosen benchmark, so it's hardly a meaningful lead.
It's like the Baseball World Series of Frontier models. It's SOTA in the same way Tesla are SOTA EVs - the best you're permitted to buy in the USA.
The US bit got me as well.
I guess it makes sense - there is a segmentation where users may want to run on domestic "frontier LLMs."
I’m SOTA on being exactly who I am.
Yep, you’re right. Quite easy to dilute the term.
No, the term is well defined. It means being on the Pareto curve of performance vs size, such as: https://tinyurl.com/26sw35fw
For sure, but unfortunately, there's a non-zero chance the state department at some point makes the "US" part of this mandatory for a bunch of companies.
State of the AI-Apocalypse.
It means ot was released recently.
It was already meaningless when US labs used it.
It means new.
Playing games like this also makes it very easy to accidentally say things that aren't true, like in the headline of the article...
> Introducing Ripple-65B Preview, a Frontier Open-Weight Model
It's not "frontier", it's not "frontier" even in the category of open-weight models (though usually that phrasing would mean frontier over all, and also open weight)... it's only "frontier" in the small sub-category of US made open-weight models.