alt Hacker NewsBecause of bots that don't respect ROBOTS.txt .
If you want an AI bot to crawl your website while you pay for that bandwidth then you wont use the tool.
Because of bots that don't respect ROBOTS.txt .
If you want an AI bot to crawl your website while you pay for that bandwidth then you wont use the tool.
If bandwidth cost is a concern the maybe you should reconsider how you publish your site.
Like, what if you actually post something that gains traction, is it going to bankrupt you or something?