alt Hacker NewsUnfortunately, geofencing is becoming more and more common specifically because of the extra bot traffic.
A shop selling goods in $COUNTRY has very little to gain from allowing arbitrary users from any other country. Allowing bots has always been a tradeoff between the potential benefits and the impact on infrastructure, which is why it's reasonable to identify yourself as a bot for the purpose of rate limiting purposes.
A journalist anonymously using a crawler is indistinguishable from a bad actor trying to collect data for its own purposes, some of which may actually be against the interests of the company in question. In particular, a blog or news publication offering knowledge in exchange for ad revenue stands to disappear because that same knowledge can be compressed in a LLM and offered to the end user directly, completely depriving the source of any revenue.
Replies
> A shop selling goods in $COUNTRY has very little to gain from allowing arbitrary users from any other country.
Eh?
a) Window shopping for the next time the out-of-country visitor is in the country.
b) An out-of-country visitor can contract someone who's in the country to purchase things from the shop and ship those things to them.
c) Queries and requests from out-of-country visitors can cause the shop to notice that setting things up so that they can ship internationally actually would be a money-making thing for them to do.
I do agree that the obscenely antisocial bots run for the benefit of the LLM Manufacturing Industry make it very hard for small-time operators to keep their websites up and running at any reasonable cost. Hopefully the major LLM Manufacturers will fail to secure enough funding over the next few months, their VC-backed primary customers will then have their money faucets shut off, and the attack bots will reduce down to a dull roar because folks have stopped paying "infinity" dollars for them.
> Unfortunately, geofencing is becoming more and more common specifically because of the extra bot traffic.
How is geofencing supposed to help? The crawler operated by technical staff in $foreign_country just switches to compromised smart TVs in your own country and you ultimately end up blocking primarily real humans who don't all know how to change their apparent geolocation.
> A journalist anonymously using a crawler is indistinguishable from a bad actor trying to collect data for its own purposes, some of which may actually be against the interests of the company in question.
There is a major way to distinguish them. If you pass a law requiring them to do something, journalists from major publications have legal departments who then order them to follow the law (and then can't do their journalism), whereas the bad actor is either not interested in following laws or is not in your jurisdiction to be subject to them to begin with.