alt Hacker News> The web is getting harder and harder to use due to all the anti-bot mechanisms site operators had to put up.
Indeed. I recently set up a bot on my local machine that I can interact with via Telegram. I gave it the ability to retrieve the contents of a URL via Playwright (with modifications). Basic things like getting the Readability view and getting the site as Markdown with links.
I was essentially mimicking the feature OpenAI etc offer where it can do a web search for you and get you the answer you seek. I just didn't want to pay a company for that feature.
It's not autonomous. It only acts when I tell it to. The actual search is via the Kagi API, but the URL contents is via Playwright. I also ensured it can't do lots of requests at a time. There's a mandatory (random) sleep of between 1 and 5 seconds between two requests.
In less than a week, I've been flagged. When I do a Google search (as a human), I have to prove I'm human. Many sites have simply blocked me - I can't buy stuff from various vendors[1], I cannot access several news sites, etc.
The thing is, I didn't really access that many sites with the bot! At most it's the equivalent of someone mildly addicted to the web. Yet they caught me quickly.
Obviously, I disabled the bot's capability to get site content. I wonder how long my IP will be in the blacklist.
I know some people will say "Good riddance" to the outcome and to what I did. I'd love to have a real, serious discussion on whether what I did was immoral. But whatever your thoughts on the matter, do ponder over the fact that the system has made it hard for average Joes like you and me to have this "feature", but if you pay a giant tech company a monthly fee, you can have it.
[1] Thankfully, Amazon didn't block me. In the past I'd avoid them, but now I'm in the dilemma where I don't want to buy stuff from sites that make it difficult for me to buy stuff from them!
Replies
My residential network has been on Google's shit list for a few years now, I don't think it will ever go away. Like others on this thread if I see a Google recaptcha that site won't get my business as I have to exit since the captcha is an infinite loop.
Expect to remain blocked, that's how defense works, some services plenty use will have is bot or known proxy exit node set to true, and some websites use the same defense services, request velocity is just one factor, if you don't have random human like scrolling and jitter with a bot, it's a dead profile walking, metaphorically speaking
I don't think this is an immoral application at all from your description, but would that be clear on the receiving end?
I just did this as well. I also added paywall bypass by default along with inline article chat. The internet is such trash these days.
I made my modifications at the network level and auto redirect a configurable set of sites to go to reader mode/paywall bypass by default with a MITM proxy and a custom CA cert I trust
Just restart your home router to rotate your IP address.
> I know some people will say "Good riddance" to the outcome and to what I did.
I will instead say "skill issue".
Using Playwright results in some very obvious and easy to detect fingerprints. I'd expect better of someone on hacker news.
Mine hasn't been detected as far as I can tell, so I know it's possible to do better.
I think that major gateways like Google and Cloudflare and Akamai are doing things like TLS-fingerprinting and smearing.
I normally use a VPN to access the Internet. It's an ordinary consumer-grade VPN, and I use it for pretty innocuous things like watching the international versions of Netflix or avoiding being geographically-profiled and charged more when I book travel. The VPN itself, I'm sure, is being used for all sorts of naughty stuff: when I look it up in whatismyip, it comes up as VPN, "abuse detected", and "residential proxy detected". I expect to get blocked occasionally on it and just pause the VPN then.
What I found surprising is that pausing the VPN doesn't always clear the CAPTCHAs. The next request will also fail, sometimes. For a while I was wondering if I have a residential proxy on my network, but I went hunting with lsof and bandwidth monitoring and a list of all the open connections through my router, and it seems clear. Yet even with VPN disabled, I'll get CAPTCHAs from Google and Cloudflare.
I suspect they're using browser/TLS fingerprinting and then smearing the results of that across bad IPs. If you connect to them through a known-bad IP, they not only will CAPTCHA you, but they will record a fingerprint of any data they can. Then if you connect again with that fingerprint, regardless of the IP, they block that too.
Ironically, if you are actually malicious, you can get around this pretty easily. Because you'll know not to use a fingerprintable browser, or to rotate fingerprints. So you have one completely separate identity on a given IP (that is not yours), and if it gets blocked, you roll up another completely separate identity on a different IP. But this is too much work for an ordinary consumer who just wants to keep their IP private. So you end up with this excluded middle where the only rational strategies are to either do exactly what the website expects you to do (with no funny business, even if it's innocuous), or to go whole-hog as the aggressor where you're determined to skirt their policies no matter what, and if you're doing that you might as well amortize the effort by extracting everything of value you can from them.