logoalt Hacker News

spindump8930 • yesterday at 9:08 PM • 0 replies • view on HN

> That basically boils down to: Count the amount of times the terms of your query appear in the document and return the top x results (which is impossible to do at Google's scale).

It is very possible to do at google's scale. Jeff Dean and co built Map Reduce to help implement algorithms of this type. For example, https://en.wikipedia.org/wiki/Tf%E2%80%93idf is entirely based on counts. Combined with a sparse encoding and distributed lookups, this is how ALL retrieval worked for decades. In fact, any large scale retrieval system today uses a combination of sparse searches followed by a ranking or scoring with dense methods. You can roughly call sparse methods count-based and dense methods neural network based.

Source: the field of information retrieval https://web.stanford.edu/~jurafsky/slp3/