alt Hacker NewsOne thing I see on the new gen of ai search bots are they fake chrome and rarely disclose themselves in UA, and not respect robots.txt, and no rate control for smaller sites.
I’m not accusing any one engine discussed in this thread, but hearing a proxy vender like decodo providing search give me a feeling there’s non zero over lap to bad search and residential proxy crawler.
Think the era of respecting stuff like robots.txt is over.
I build a web-fetch tool for my own LLMs that is essentially the same tech as web-scraping except its single requests not high volume.
You can't really get anywhere these days without disguising UA etc.