logoalt Hacker News

Bender • yesterday at 3:21 PM • 2 replies • view on HN

Seems about right. I rotated my logs this morning. Most humans go to access.log and most bots go to botpoop.log. This is the line count:

     2 access.log [1]
    40 botpoop.log [2]
2 is really 1 since a human will grab the CSS file. Most bots do not bother with the style-sheet so that's a 40:1 bots to humans. I could cut that down by blocking data-centers but then I inadvertently block a lot of VPN's which I really don't need to do for a static compressed blog served from ram. The bots just get a TCP Reset but it's still fun to log and study them. The most interesting one I've seen recently is ReadYou which may be a reader but it appears to be much more, possibly acting as a cell phone distributed bot collecting data for a centralized site.

[1] - https://nochan.net/logs/access.log

[2] - https://nochan.net/logs/botpoop.log

Replies

spockz • yesterday at 3:44 PM

Is there some existing mechanism already that counts how often an ip only scrapes the page and not the css and then block those origin IPs if it occurs “too often”?

Unfortunately, the best practice is to make css cacheable so you need to keep long histories.

➕ show 1 reply
Bender • today at 1:29 AM

More realistic numbers now that HN and others have been visiting all day. Line count:

    104 access.log
    366 botpoop.log
Unique IP's:

     54 access.log
    181 botpoop.log