logoalt Hacker News

npn • today at 12:13 PM • 4 replies • view on HN

4. they are so behind that they can not create effective datasets like other frontier labs do. So instead they do all the tricks the Chinese labs do: using RL with the single purpose of benchmaxxing the AA score.

Look how all Chinese models, or haiku/luna class US models behave, you can see the same pattern: they think too much and only work well with a subset of tasks


Replies

Doohickey-d • today at 12:25 PM

Which is weird, because Google is in the best position for getting training data: they have YouTube (which I guess is why Google's models are good at multimedia), as well as the Google crawler. Which effectively websites cannot opt out of if they want to be Google searchable. So Google's bot probably is the _only_ bot (scraper) that hasn't been blocked from a large portion of the internet.

➕ show 3 replies
owkman • today at 1:52 PM

Seeing how every new benchmark that has come out has Carbon doing well, I'd have expected the benchmaxxing allegations to die out. But they've been ok incredibly persistent. It's jagged like all frontier models. You could argue about it's warts, but benchmaxxing is unfounded. Did it benchmaxx every public and private dataset? Is that what RL is perhaps?

jckrichabdkejdb • today at 2:42 PM

[dead]

unbrice • today at 5:22 PM

[Citation Needed]