alt Hacker NewsPageRank explained
Comments
I actually built this, and shipped it, in 1996, with no knowledge of page rank, citation analysis, or bibliometrics, for an internal/external search engine for the Envirolink web site. Envirolink was a directory of environmental web sites, so they already had a list of URLs to crawl. The reason it was feasible to build was that it was a fairly constrained list of URLs, it wasn't the entire web.
I didn't really know what I was doing (I was 17), but it was an awesome unpaid summer internship. There were two parts of the search engine - a crawler and the search engine. Both were written in Perl.
PageRank is fascinating, since it is so easy to explain.
Yet, this is not even half the work. It like a third of the way.
Before you could have invented PageRank, you must think in graphs. That is possible in 1996, but not as widespread as today.
After you invented PageRank, you still need to deploy it. Again, possible but challenging as well. Is Python performant enough in 96? Can you afford more than 4MB RAM?
At least Lego will not sue you for using their bricks to build a server rack in 1996.
Here are two excellent videos that explain and visualize the PageRank algorithm:
* [2020-06-17] Spanning Tree - "How Google's PageRank Algorithm Works" (5m16s): https://www.youtube.com/watch?v=meonLcN7LD4
* [2022-05-23] Reducible - "PageRank: A Trillion Dollar Algorithm" (25m25s): https://www.youtube.com/watch?v=JGQe4kiPnrU
Well, I was a child in 1996, so probably not.
Tying relevancy to link frequency was definitely a novel idea at the time, even if it seems "obvious" or simple in retrospect.
Thinking of an algorithm in the abstract is one thing, implementing it at scale is another.
Yes you could have invented PageRank, but could you also have invented MapReduce, BigFiles/Google File System (GFS), Google Web Server, Bigtable, Protobuf? Then spun up fault-tolerant clusters consisting of cheap commodity PC hardware in an era where AWS wasn't even an idea yet? Then invented the concept of Borg to manage this hardware globally?
Also PageRank as a Markov chain
[0] https://math.libretexts.org/Bookshelves/Linear_Algebra/Under...
As, I think, Page points out in the patent, PageRank's idea comes from Science Citation Index. That was an inverted list of scientific references, where you could look up an scientific paper in an expensive set of bound books and find all the papers in which it was later referenced. You can then use this to see who's getting referenced a lot, which is an ego trip in academia. Academic libraries had copies of that index. Now everybody has that kind of info, but when it had to be done by hand, it was hard.
Inverting the huge, sparse matrix of references for PageRank was expensive. Originally, Google did it about once a week. The big breakthrough was when someone (who?) figured out how to do it incrementally at scale.
Popularity is not equal to relevance
Except in the case of advertising
Even if, hypothetically, company knows what user is searching for before user finishes typing a query, it does not mean company is going to deliver it to user
Company is paid by advertisers, not users
Company can deliver something _popular_, call it "relevant", make bank
"Popular" grows audience, good for advertisers
"Relevant" not necessarily good for advertisers, only good for user
Company serves its customers: advertisers
Company does not need PageRank to determine popularity, only search traffic (search query data)
(Of course a result could be both relevant and popular. But a result could also be relevant and _unpopular_. Company is promotes the former to the exclusion of the later. It profits by selling advertising services to advertisers, not information services to www users)
You could also have invented a wheel because it's so obvious.
Yes anybody can be in the right place at the right time.
Let's not forget that PageRank unintentionally spawned the SEO industry.
> Sergey Brin and Larry Page came up with this precise algorithm, i.e., PageRank, which was one of the key algorithms that helped catapult Google into a household name and made them tons of money. Both Sergey and Larry were grad students at Stanford, so their coming up with such an amazing algorithm doesn’t seem surprising.
No, Brin wasn’t a co-inventor of PageRank.
Here's an in-depth explanation with an interactive demo that I made many moons ago:
https://web.archive.org/web/20130728183938/williamcotton.com...
Related, fascinating foundational work
Citation index https://en.wikipedia.org/wiki/Citation_index
Shepard's Citations https://en.wikipedia.org/wiki/Shepard's_Citations
One is reminded of Damien Hirst's famed retort to a critic who said "Well I could have pickled a shark" ... "But you didn't, did you. I did."
Original HN title: "You could have invented PageRank"
Yup. I was there, using the web, back then. I used Altavista dozens of times a day. It wasn't very good.
I never thought of it.
I never thought of the Million Dollar Homepage, either.
It's still there! https://milliondollarhomepage.com/
I remember the first ever HTML CV (résumé) being published.
I am Slashdot user #6030. I used it for ages before I created a user account.
I was already paying for my own personal email address in 1991 when timbl revealed the WWW to the world. I thought it was a gimmick. It'd never catch on. We already had Gopher and Archie and Veronica.
deep sigh
PageRank is really the beginning of popularity as proxy for authentic/truthful/accurate/etc.
Then we move on to upvotes, retweets, likes, listens, views, please-please-please-subscribe-to-my-channel, reviews etc. Thus the continuum of the "social" internet.
We tried really hard to make truth into a capitalist endeavor.
What I always wonder is, how does google knows how to go to abc.com, xyz.com, mypage.com, etc. so it can crawl them?
I could not have invented it. I was one year old.
How do you compute page rank for billion of pages that have cyclic links? Is that not the problem right after the initial idea?
It's giving "Attention is all you need" but in the year 1996
curious, how do you decide on which page to start from? Popular directories?
No mention of RankDex?
All credit to you for writing this yourself!
PageRank, at its core, symbolizes these basic properties.
Step 1: be sponsored by the CIA
it’ll give you an article on “Hotels for Chickens” if you search “Hotels” because the word matches
What if I fricken want fricken "Hotels for Chickens", what about that, huh?
I mean, page rank was useful but it was also a step in the direction of "search give you what it think you, not what you asked for" and I think now we can how far and dubious progress in that direction has been.
[dead]
If I had received funding from DARPA, and NASA, perhaps I could have.
Importantly, PageRank doesn't work today - you need something else. It was one of many possible ranking hacks, and one that worked at the particular time in that particular state of the web where nobody was gaming links because PageRank didn't exist yet. Maybe you could invent a good ranking algorithm for the modern internet, perhaps just the reciprocal of the number of ads on the page, minus its AI detector score, but probably not that.
Unimportantly, it's named after Larry Page, not after the fact that it ranks pages.