logoalt Hacker News

peter_d_shermantoday at 2:18 AM0 repliesview on HN

An interesting article, and an interesting problem to have!

The problem, to recap the title, is that "99% of My Website Traffic Is Bots".

How to fight all of those bots, all of those web-scrapers...

If the sole underlying issue is bandwidth for one's website (someone doesn't have enough of it and/or they pay too much for it), then one possible solution to this might be for a single tech company to create an online cache of web pages, accessible to all bots as an alternate route to fetch the desired web pages.

If a tech company (or consortium thereof) decided to do this, they could certainly charge money to each AI / bot utilizing the service...

Which means that it might be an investable idea as a for-profit service...

Now, if the underlying issue is the privacy of individual web pages (or relative privacy, as the case may be!) from the public internet, then perhaps the solution is to simply put those behind user logins and/or a gauntlet of tests (which could also be tied to login!) designed to exclude AI/bot web traffic from human web traffic.

Consider what some BBS'es and the online services of yesteryear used to do (Compuserve, The Source, Prodigy, QuantumLink, AOL, etc.)... basically they'd get a user's mailing address, and then physically snail mail them their login and password to their physical home address...

Oh sure, user registration wasn't as fast back then as it is today... it might take several days to receive your username and password via postal mail -- but as a system operator you were 99.999999999% guaranteed that when you saw such a login on your BBS, that it was an actual live human user...

There may be a market for a service like that, too...

That is, validate that actual users are actual users, and give them some kind of credential that can be checked by an actual web site... find a way to do this at scale...

Anyway, just thinking aloud... a very interesting article, and a very interesting problem to have!