It is a naive question but I do wonder how hard it is to scrape the web and build web indexes that are shared via bit torrent and self host a tfidf traditional engine using a distributed effort. How big the web really is?

Is there a lot of additional secret sauce that made google work well in its prime?