logoalt Hacker News

foohis • yesterday at 10:16 PM • 10 replies • view on HN

Anthropic spent a lot more time editing their pre-training data with higher-quality material. For example, they used actual books for pre-training, because books are generally run through human editors for higher quality. (this is from the copyright lawsuits)

Other companies would use completely raw internet data for pre-training, which is basically trash. Anthropic's secret sauce is that they filtered and edited down their data for only the highest-quality material. You really don't need quadrillions of tokens to pre-train, as that's just random noise. A smaller-but-higher-quality data set is better in every way.

Meanwhile its obvious that companies like Google would use trash data like email and chat logs or Internet forums for pre-training. Just complete trash data.

This is a classic case of "less is more" and "garbage-in-equals-garbage-out"


Replies

hackingonempty • yesterday at 11:20 PM

  Anthropic spent a lot more time editing their pre-training data with higher-quality material. For example, they used actual books for pre-training, because books are generally run through human editors for higher quality. (this is from the copyright lawsuits)
  [ ... ]
  Meanwhile its obvious that companies like Google would use trash data like email and chat logs or Internet forums for pre-training. Just complete trash data.
Google has been scanning every book they can get their hands on for 20 years or more. They have a bunch of patents on non-destructive book scanning gear. Surely they are using that data to train their models.
➕ show 2 replies
impulser_ • yesterday at 10:40 PM

I don't think it pre-training, because surely these labs all have similar data and Google should be very good at ingesting, cleaning, and filtering data.

I honestly think it just their post-training is fantastic and I think it has something to do with them being very opinionated on how they want the model to act that it basically the best at roleplaying a human. Humans are very good at completely and figuring out things that there aren't "trained" on if given basic set of intelligence. So when it working on these long tasks that might not be directly in it training data it still reason through it very well.

They got made fun of for this in the pass because they do a lot of research on this including hiring a philosopher.

This is just my best guess, because you don't see any other labs talk about how they made the model act and Anthropic is basically obsessed with it and talks about it every model release.

I think they just have better "taste" and that taste is their secret sauce.

nl • yesterday at 11:24 PM

This is incorrect.

Everyone uses books as well as internet data.

Anthropics difference is they have a huge data budget for buying RL environments for real world tasks.

mlmonkey • yesterday at 11:58 PM

Microsoft (MSAI) showed that quality >> quantity with their Phi model a couple of years ago. It was a small model, but trained on very high quality data, and did better than much larger models.

amarcheschi • today at 12:39 AM

I can't say what precisely, but other big companies are also doing what they can to get human data of high quality, just look for all the companies that popped like mushrooms that do ai reinforcement tasks, micro1, ethos, mercor ecc

mrheosuper • today at 2:27 AM

Is that reason why Claude talks so like .... a nerd.

olalonde • today at 1:13 AM

Where do you get that information from? Or is it just speculation?

quixoticelixer- • yesterday at 10:30 PM

That doesn't really make sense because as good as Opus 5.5 is it writes awful terrible, disgusting slop.

➕ show 3 replies
UltraSane • yesterday at 11:28 PM

I think by now everyone training trillion parameter models is using almost every book, magazine, research paper, etc they can get.

Topfi • yesterday at 11:01 PM

Purely outside looking in, but yes, I'd argue you are right and higher-quality is a major part of it, both for pre- and post-training. There appears to be a far greater willingness to adopt specifically tailored training sets from third-parties, consistently update and build on them.

From simple frontend work to strategically adding virtual rulers when working with LIDAR scans to extract more accurate data, there are very clear points when Anthropic introduces a new external data set into their mix and while I have seen the same from competitors such as Google, there simply appears to be a greater tendency to stick with these once they have them.

As an example, at release it was quite apparent that Gemini 3 Pro, 3 Flash and Opus 4.5 had a similar, possible even the same frontend-focused training base bought in from a third party. But while Anthropic has had both smaller and larger improvements across each subsequent Opus release, I have not noticed such a consistent improvement with Gemini models over the last year. Maybe Anthropic simply buys updates more often, maybe they create more artificial PRs on top, who knows.

Same with the LIDAR scans, there was a point a few months ago when models started to exhibit a very systematic way of handling visual references in some few leather-crafting tasks, enabling whole new ways of working more efficiently combining existing measurements with images. Step-by-step pixel measurements, compensation for angles and poor captures, etc. Have since seen this in other models, again very much feels like a bought in data set that they are pushing as far as it goes.

Then there is 3D capability with models like Muse Wonder Full. I have not used it myself, if it ain't on API, it doesn't exist, but have seen PRs and do find the model output very parsable, despite it being terrible at any modelling task because these capabilities do not emerge from just training on vast amounts of data, they require high-quality, well tailored, step-by-step reference data of the entire process.

Harness wise, while pure output performance is very consistent across harnesses in my evals (I nowadays use both OpenCode via API and the labs own harness three times each to normalise), Anthropic also appears to be far more willing to use the time between releases to prepare features for the next. Stuff like workflows, proper subagent implementation, advisor, etc. often have found there way into Claude Code either in-between models or were first seen with a model that couldn't fully leverage these. Also feel that explained the good overall launch experience Fable 5 delivered, they simply used the time between announcing Glasswing and launching full access to get the UX/DX polished.

Meanwhile, I am still testing GPT-6.2 Astra and Sol so will refrain from making any more judgement on either till I have all data, but apparently there has been an extensive last minute rewriting of the system prompt to improve performance and "behaviour". Given the gap between GPT-6 Astra and yesterday, along with statements by OpenAI staff alluding that they had this model ready for a while, such things should happen in a far more structured, systematic, well thought out manner, not last second after some negative outside feedback. Heck, from what I know Anthropic, unlike any other lab, does zero early access outside the limited hardening for very exclusive partners like Glasswing. Certainly no early access for regular users, reviewers or the like from what I've heard, so Anthropic does all that prep-work for model releases with less feedback than other labs.