alt Hacker NewsI'm terribly sorry, but scaling curves or GTFO. Any random pile of linear algebra works fine-ish at small scales. Very few random piles of linear algebra push the Pareto envelope at large scales.
Replies
Not every one can afford millions to publish a paper
This exact mentality is cancer for peer review/the industry. We all know who you are if you are using 1000+ TPUs, and yes you do get a "buff" to your peer review scores because people know where you work.
Fuck your scaling curves. More research labs need to #yolo and try stuff that doesn't have good scaling behavior proven yet. State Space models have continued to take forever to proliferate despite being objectively good because only the god dang Chinese understand that you actually need to #yolo sometimes like making some of your layer state space layers in Hunyuan-T1.
Do you want to see scaling curves wrt data and param size? I agree that 1.2B and 10B tokens is not representative, but what scale of parameters and dataset sizes would be convincing?