alt Hacker NewsThe issue with distillation is: one lab spends $$$ on bleeding edge R&D and expensive RL runs to improve capabilities, and other labs just yoink the raw reasoning traces and mid-train/post-train on them to get 90% of the way there for a small fraction of the cost.
An even smaller fraction of the cost if they do it by buying AI access at as much of a discount as they can find, including black market resellers, and then reselling that access to paying users again with a proxy. As is common.
This gives ruthless "fast followers" an economic edge over the innovator that's putting in the real work.
The dynamics are very much alike to what patents and copyright law are supposed to prevent. Same type of "we took the products of your work and used them to undercut you". Except there are no laws against distillation - so most of the enforcement happens on model provider level.
Replies
I'm fine if they put preventative measures in place to protect their work. They already do so. I am NOT fine with their mass manipulation of public opinion to fuel an entirely hypocritical viewpoint. Like, any argument here is hypocritical - but they aren't saying what is REALLY HAPPENING ("distillation steals our work and reduces our profits"), and are actually saying words that make other people fight their battle ("national security", etc).
There's an implication that other companies are improving because they're scraping Anthropic, not because they're investing in better architecture, compute efficiency, or their own synthetic data pipelines. I often see Chinese labs' progress dismissed as "they just distilled Anthropic" and I find it hard to reconcile that with all of the interesting research and open-source tooling that they release.
Is there actually that much capability transfer from non-logit-matched distillation, or is Anthropic just another unwilling source of data?