logoalt Hacker News

tuhgdetzhhyesterday at 8:04 PM5 repliesview on HN

I still think it is just a matter of time until scrapers catch up. There are more and more scrapers that spin up an full blown chromium.


Replies

kstrauseryesterday at 8:12 PM

It seems inevitable, but in the mean time, that's vastly more expensive than running curl in a loop. In fact, it may be expensive enough that it cuts bot traffic down to a level I no longer care about defending against. Like GoogleBot had been crawling my stuff for years without breaking the site. If every bot were like that, I wouldn't care.

solid_fuelyesterday at 11:23 PM

Cool, if they're running full blown chromium maybe the next step can be mining bitcoin on any pages served to bots.

hxtkyesterday at 10:04 PM

Even that functions as a sort of proof of work, requiring a commitment of compute resources that is table stakes for individual users but multiplies the cost of making millions of requests.

gruezyesterday at 10:58 PM

AFAIK you can bypass it with curl because there's an explicit whitelist for it, no need for a headful browser.

cantalopesyesterday at 10:56 PM

Well it's a race, just like security. And as long as anubis is in the front, all looks bright