alt Hacker NewsThank you for your comments!
We have an index, it's just not very big, yet. We had a major setback last month with a bug (for less than 24h, the crawler didn't respect robots.txt) and had to delete it entirely, and have been slowly rebuilding it.
You're correct the commercial use is only not allowed before the AGPL comes into play.
You should be able to click on the "top up" link (top or bottom) and see an option for a captcha ("click to prove you're a human"). If you don't, reach out via email (don't share your account number) and I'll give you a voucher for a couple of days.
Replies
That’s on me, didn’t see the text explaining it above the captcha.
I’ll give it a shot. Always happy to see some competition in the market. And I love the idea of handing out the source after a year.
Interesting. How much does it cost to build a search index in current year? Also, how do you get around the fact that everyone's blocking bots?
I've been thinking of making a "small web" indexer so I'm curious about that. I'm seeing even tiny websites being behind CloudFlare, Anubis etc. these days. (And everyone complaining they're getting hammered by mysterious distributed HTTP traffic!)