logoalt Hacker News

thisislife2yesterday at 2:37 PM2 repliesview on HN

Only if they had a robots.txt for their site.


Replies

razingedenyesterday at 2:58 PM

I hadn’t even considered that. Don’t know why that comment is greyed out or downvoted.

It’s a static site that hasn’t been updated since 2016—- so it’s .. since been moved to cloudflare r2 where it’s getting a $0.00 bill, and it now has a disallow / directive. I’m not sure if it’s being obeyed because the cf dash still says it’s getting 700-1300 hits a day even with all the anti bot, “cf managed robots” stuff for ai crawlers in there.

The content is so dry and irrelevant I just can’t even fathom 1/100th of that being legitimate human interest but I thought these things just vacuumed up and stole everyone’s content instead of nailing their pages constantly?

gzreadyesterday at 3:02 PM

No, it's still illegal to DDoS sites that don't have robots.txt.

show 1 reply