despite this limitation, there is still some good stuff out there, and with the priority steering, you can focus compute on what you actually want, fast and cheap.

Thats the fun part, the user just went with happy path. Javascript, captchas, cloudflare protected content did not made to the catalogue. This sort of use case exists in LLM training data a lot which makes it easier. The data gathered by the user is not really practically useful cause there are way too many gotchas when it comes to web scraping and building a catalogue (source: I have done scraping for a particular domain data and had to do at least 10+ iterations to get it >90 right)

They don't: "skips the model entirely if the page is empty, parked, or a bot-challenge wall"