logoalt Hacker News

hackingonempty • yesterday at 11:20 PM • 2 replies • view on HN

  Anthropic spent a lot more time editing their pre-training data with higher-quality material. For example, they used actual books for pre-training, because books are generally run through human editors for higher quality. (this is from the copyright lawsuits)
  [ ... ]
  Meanwhile its obvious that companies like Google would use trash data like email and chat logs or Internet forums for pre-training. Just complete trash data.
Google has been scanning every book they can get their hands on for 20 years or more. They have a bunch of patents on non-destructive book scanning gear. Surely they are using that data to train their models.

Replies

foohis • today at 12:21 AM

won't help if they also include trash internet text in the training set as well.

➕ show 1 reply
wahnfrieden • yesterday at 11:57 PM

Surely they are destroying scanned books for training

➕ show 1 reply