alt Hacker NewsI get tens of thousands of IPs making 1 or 2 requests against pages on The Consensus (job search, code viewing, etc.). They're not super expensive pages. But I don't want to deal with the bot traffic (mostly because it messes up view counts I use to sell sponsorships). But then if I require an account (even a free one), valid readership is also going to drop with the new friction.
I could build something more complex to discover the patterns, or use Turnstile maybe. I'm not really sure at the moment.
Replies
Bots give all the information required to block or rate limit them including those that try to look like a real browser (some things they just can't hide, even residential proxies via behavior.). Ask the LLM how to change your log format to gather all the fields your web server can see. Share the resulting updated logs with your favorite LLM. Ask it for generalized rules to block the bots whilst permitting real browsers. A tiny percentage of bots will still get through but the load becomes much more manageable. Tell your LLM you also wish to spot fake chrome via CH and bots via ssl curves. No need for JA3. Nginx can expose the important aspects of GREASE.
seems like open web is dead which is sad
i heard hcaptcha is good
I mind a public access website for a government entity and this is exactly the pattern I see. US residential IPs make a couple of requests then never come back.
The bots may have layer 7 signatures that make them "easy" to block, but I'm dealing with COTS software with no ability to make changes to the stack. Even if I could by the time they're blocked the work is done and they're never coming back anyway.
I don't think I'm seeing the LLM bots in the scraping on this particular site. This site's data is highly sought-after by data brokers, legal services advertising, debt collectors, etc. I tend to think I'm seeing homespun scrapers. This site has been up for 20+ years and the scrapers seem to be well-adapted to using the HTML as an ad-hoc API for the data. The difference in the last few years has been the availability of vast numbers of residential proxies to mask the traffic.