alt Hacker NewsThose indexes cost an incredible amount of money to build, maintain and keep improving. It’s worth it for these orgs as these are considered a competitive edge and the “secret sauce” of these orgs.
I find it hard to believe the same amount of investment will be made into a decentralized web index, precisely for the reason that it cannot be monetized.
Replies
Are you familiar with any systems that use collaborative filtering to distribute the work and the benefits among users?
This. Unless in 2026, we suddenly see Costco sell personal quantum computers for sub $2k, with practically unlimited high speed crystal storage, all cooled/powered by unobtainium fusion reactors that are made with 3D printers at your neighborhood coop, then we either must accept private entities that do this kind of indexing at scale or we should ask government to offer it on taxpayer expense.
The keep improving part hasn't done so well in 10 years already? Maybe this year the new force-fed AI answers got a bit useful, but many times the risk of hallucination means you still have to go and read a more credible source.
No they don't. Sure, if you are trying to capture the entire web with hourly latency for billions of users, that takes significant resources. But if your goal is to catalog the subsection of the web you are interested in with weekly latency for yourself and share that catalog with like minded individuals. any body can do that. Don't make gods out of basic web indexing.