alt Hacker NewsLLM training will eventually transition from real data to synthetic data, same as alphago -> alphazero.
AI companies are also working to integrate training with real-world experience through sensors and robotics, to shrink the gap between human experience and hallucinated LLM experience.
They all have archives of pre-LLM content. There's also archive.org, google books, and pirate ebook archives. I don't know what they're doing to build video and audio archives, but judging from the cost of spinning rust, they're storing significant quantities of that, too.
Some parts of the internet are curated, and even with LLM influence they're still worth training on. I doubt wikipedia or stackexchange or rosettacode will ever cease to be useful at all.
Replies
That (theoretically) solves training, but it doesn’t change the fact that even smart models can’t extract useful information from a dead internet, so you’ll always be stuck with a stale training cutoff. This is already a problem I run into a lot. I search something first. Top results are slop sites, so I switch to a chatbot. Its answers look suspiciously similar to the slop sites I just noped out of. Check the sources. It’s them.
Neither AlphaGo nor AlphaZero were transformers. Why would you expect the same results?
Going further, current LLMs have at least an order of magnitude more computing resources, but completely suck at go. Why would this suddenly change unless we dumped the countless games played by alphago for them to train on?