logoalt Hacker News

postalcoderyesterday at 5:25 PM2 repliesview on HN

> Sol xhigh is 90% on TB2.1 but 37% on TB4. Is it also "benchmaxxed"?

A model that was released a couple months ago scores 50% higher than SWE-2, a model released today, on an out-of-sample benchmark. Can I say I’ve come out of this more impressed with Sol?

Like you said, TB2 is saturated. Nobody would bat an eyelash at 90%. And yet here comes SWE-2 coming off top rope with an emphatic 92.4%. this is the definition of bench maxxing.


Replies

Lucasoatoyesterday at 10:34 PM

> A model that was released a couple months ago scores 50% higher than SWE-2, a model released today, on an out-of-sample benchmark. Can I say I’ve come out of this more impressed with Sol?

Wait a second, are we taking into account the massive difference in terms of resources of these two companies?

show 1 reply