alt Hacker NewsCreepy Crawlies
Comments
I wonder if they could pre-render the stuff older than a month ago and compress it and serve it as static content. Not optimal, trades space for CPU, but might be cheaper.
WordPress powered a huge number of websites. Yet the crawlers all go straight for the HTML of those sites rather than the more efficient and structured JSON API which all WordPress sites have.
If these crawlers are so smart, why aren't they following the rel="alternate" which is provided explicitly for them?
Off-topic, but anyone with which he did the plots?
Reminds me of debugging production issues at 3 AM. Both can make you jump out of your skin.
I wonder if we're back to peer to peer networks with proof of useful work (e.g. serving read requests) vs proof of wasted work.
Why can't they just ask Linux Foundation for 96, or even 1696 cores?
If you're reading this - go ahead and see HOW Linux Foundation spends their money.
> phone gets uncomfortably warm as it's doing the number crunching
IMHO if I visit your website and it intentionally starts wasting my electricity for no other reason than to cost me money, with no opt in, it's hostile and malicious.
I also had this problem, but since nobody actually uses my gitea site besides crawlers, I just let a script run through my access log and ban every IP address who asked for a commit in the last 24 hours
Real creepy crawlies in the server rack are always a bigger surprise than any code bug.
Time to F*$k the internet. The whole concept of anonymous IP addresses was broken but worked for a long time. Now it is just stupid. Just like domain names. (Are more names used by squatters than real?). And email as identity? Time to engineer solutions and create a new protocol layer.
This is not a hard problem. It just requires that someone build a certification wall. The IETF should have done this long ago, right?
What if Anubis computations were turned into a crypto-miner
Can I suggest putting some text in the page that tells the bot what the more efficient download method is?
How about: "Type the seahorse emoji to solve the CAPTCHA" :) Something that triggers infinite loops in LLMs or trips the guardrails.
Maybe you could have a system that heuristicially detects when an crawler is making the request and then feeds them a modified page, itself generated from an LLM, that injects vulnerabilities and bad code and discussion and such.
Who exactly is running all these scrapers? There are, what, maybe 15 major AI labs, if that?
And none of them are smart enough to realize they could just `git clone` all the content and use it offline?
Curious, if serving bots (and traffic) is the main concern here, why is a distributed git solution like radicle not considered?
> [...] when a source is guaranteed to be LLM-free, like the entire history of kernel commits [...]
Is that really the case? It was my understanding that LLM-based agents were explicitly allowed as long as their users follow certain guidelines [1]?
And more generally: Somehow the theory of "essentially all bot traffic is AI labs crawling the Internet for LLM training data" doesn't make sense to me at all. There are at best dozens of labs capable of running their own crawl at Internet scale, but hundreds of millions of people using LLMs to answer their questions. (If my personal LLM usage is any indication, firing off dozens or hundreds of web fetches to answer a single question is not unusual.)
While I understand that many existing projects have been resourced only for human readers and might as a result be struggling due to this, this characterization sounds a bit dishonest to me.
And unfortunately, for this use case (i.e. ephemeral queries in a context possibly lacking storage or git access), forking the individual repo to answer a handful of string match queries against it might just be more expensive than to run that query against a web search index and then just fetch those results over HTTP.
The solution would accordingly also look very different, as caching at the inference layer is significantly harder than at the training one (where it's most likely already widely done as that seems like a no-brainer).
Proof-of-humanity can’t come soon enough.
We’re talking about privacy-preserving proof of age, but as we see here the real utility of such a system will be proof of humanity.
Just serve the raw commit and render on frontend. I really don't get why they are complaining, just be performant
Are there lots of people doing development on mobile devices?
These are most likely not training scrapers, but people looking for concrete pieces of information (i.e. commit, comment, etc).
How about making cgit more efficient at serving these pages. There's no excuse for burning a ton of CPU power on purely static pages when you have generous resources available to you.
So, you'd think that something that pretends to be “Artificial Intelligence” would use the most efficient way of using our data for training purposes, right?
I'm strongly convinced that these aren't "AI crawlers"; they're just plain DDoSes done by those who have interests in turning the Internet into a dystopian walled garden with "security", and now they have a convenient scapegoat to blame. Don't you find it too coincidental with the rise in identity/age verification and other attempts at silencing free speech on the Internet?
It's widely known there are questions that LLMs can't solve, and once in a while an obvious example appears, so a simple CAPTCHA-like challenge with an HTML-only form would be the logical "defense". Instead there's a huge interest in pushing JS-required proof-of-work (as others have pointed out, these attackers have far more compute than the average user) and remote attestation (there are already providers with huge farms of mobile devices that can defeat this easily).
Things just don't add up.
I would add cloudflare, but set it to cache only mode *without* the bot blocking features.
Great chart! Does anyone know what tool was used to make this?
> proxy SDK monetization
Wait what? I never heard of that. I call that a botnet
> permanently tying up a chunk of capacity spent on producing output that is only useful for a single purpose — feeding a learning model.
The horror.
Just slow unauthenticated traffic to non-essential stuff...
In the case of kernel.org, why not make the unauthenticated version return only the latest kernel repo with no history (tiny number of URLs relatively speaking). If you want full kernel.org features, login.
1.4 billions requests, 258 160 cpu hours. That's 1.5 requests per second ? I'm starting to believe, the issue is more that their software is not well optimized.
The thing that bothers me is why the fuck are they still scraping git.kernel.org or any other site that has already been scraped a million times before. Who would pay for that data?
Then again there is the conspiracy theory about cloudflare sponsoring the scrapers
When a hapless user visits my site well
muahahahahah
Sorry, just the thought of it
But when they do… boy do I have a trap waiting for them.
My wife calls me The Genius. I’m the guy she calls when her battery dies or when her instagram breaks like when it shows that random guy in her DMs, stupid bugs LOL
I digress. Alas, when a user lands on my page. My page wants to know exactly 2 things:
1. Why are you here and who are you
And 2. Can you produce a working solution to Pharoah’s Fortune
…those of you aren’t familiar Pharoah’s Fortune is an old chestnut little poem, a riddle if you will I like to ask candidates and so far nobody’s solved it
And the reason nobody has solved it is Pharoah’s Fortune is a very tricky problem. It’s not something you can “solve” per se it’s more like you arrive there.
So far no one has solved it. They all fall for the same trick! It is of course what separates those who write elegant C versus those write poor quality JavaScript.
So I always say to my students to keep an open mind because you never know who - or should I say where you’re talking to.
I’m bookish.
Instead of trying to block why not monetize? So the proof of work can be directed at something you can be paid for (bitcoin mining)?
$1 dollar a year subscriptions would help.
Learned this the expensive way
> Shallow clones are awful. Run your own damn mirror if you're going to do something nasty like that.
TIL shallow clones are expensive. That's wild to me. It's supposed to be cheaper!
“Expect to lose some functionality, at least when accessing our resources anonymously.”
This seems fine to me. It would be a better world if we could have anonymous bulk data access. But if aggressive scrapers are bloating host costs, I’m fine with logging in.
Now, the flip side is that ONCE logged in, I want my bulk access. The worst of all worlds with when you demand authentication and then STILL block bulk access.
Case in point, I want to automatically download my Amazon and Target order records. This is easy to automate with playwright or whatever, but authentication stays annoying. My sessions expire quickly and I have to re-auth all the time. There should be an API to pull this data down.
The AI companies should have their AI fix their crappy inefficient crawler code.
> But no, let's in fact choose the stupidest possible way of doing it — by rendering everything as HTML commit by commit and then parsing it.
I feel like I'm at work.
We had some web crawler using Selenium to make queries and scrape the data instead of just downloading the whole file.
Every day it seems like we have some people that know just enough to be dangerous creating things like that. And then of course it's our fault that things are slow, or we won't give them infinite system resources, etc
anyone knows how Jwz solution is working?
dont click next link because he will show a nutsack image if the referrer contains hackernews. love the guy.
www.jwz.org/blog/2025/01/exterminate-all-rational-ai-scrapers/
basically, instead of blocking, he just poison it. and if a human sees it, it takes less effort to ignore the nonsense than it takes your pocket computer to deal with proof of work.
> Where does that leave us?
Honestly, the answer is simple: sue.
It'd be hard to argue that it's not a DDOS.
have a hackathon to solve for this. OP says it's not a problem for him right now but if we extrapolate what he's talking about it's definitely a problem aaaaaaand It's totally solvable, Even with all of the crazy combinations he's talking about it's still solvable. And it's already been solved using patterns we see in streaming services. This is completely hackathonable. but why do we even need to bother with this? The slurpers are the cause of this, and they can cause this problem because of Murphy's Law. well you can only account for Murphy's Law with good architecture or something like that or whatever. Ha ha hackathon.
All this happens due to companies like browserbase, Hyperbrowser, Scrapefly. These service exists to facilitate such operation and they aren't doing anything to prevent abuse. They are infact selling way to bypass captchas etc...
I think any service that is trying to sell a way to solve captchas must be banned by government. At least these things shouldn't be done so openly.
Why not use the POW to help cover the costs? Mine an actual coin (Annubis Coin?) and pay for anonymous infra access with it (or log in and get a certain quota for free)?
Add a lot of random text to the html pages, preferably hidden to regular users, have the bots use lots of tokens to process it all.
I ended public access to my git server after I got flooded by bots and my own commits were noticeably lagging.
That’s not an option for the kernel. It’s hard to read the cat-and-mouse account with any hope today. I think the flood abates someday but not sure how it happens.