alt Hacker NewsGPT-6 is supposed to be using a much larger base model that just finished pretraining so the "dump another chunk of all written language" approach is still going strong.
Replies
linkregister • last Thursday at 11:34 PM
Modern pretraining also consists of expensive human-led specialized task creation and grading loops. Synthetic generation and distillation from previous models is another input for training. I wonder how much new text contributes beyond keeping knowledge up-to-date.
zamadatix • yesterday at 2:42 AM
The measure is to see if the results scale, not just the rumored attempts at building such a model, as o3 taught us.
Wowfunhappy • yesterday at 11:49 AM
Where are they getting new data from at this point? Didn't they already read the entire internet?
➕ show 2 replies
Where did they get a second copy of all books, written/audio/visual media, and the web?
Couch cushions?