logoalt Hacker News

alex43578 • today at 10:47 AM • 1 reply • view on HN

Internet search isn’t one of those. Breadth, depth, and quality of search results (ie what you want is the first result) matter. That’s only reasonably attainable at scale. Not to mention it’d be a PITA to optimize for a dozen competing crawl standards.


Replies

MatejKafka • today at 2:27 PM

You don't need a dozen crawlers, just a dozen good search engines that have access to an existing index.