logoalt Hacker News

beastman82last Tuesday at 12:45 PM1 replyview on HN

They've been crawling the web since inception


Replies

snowwrestlerlast Tuesday at 6:45 PM

A web crawler reads page content, extracts content and URLs, places them into an index, and then follows links in that index to further build the index and content corpus. Google and others have special crawlers that execute JavaScript to crawl content delivered dynamically.

Crawlers do not use the browser back button or browser history. So the only way Google could observe such problems is by observing live human browsing behavior.

Also, we know from exhibits in the U.S. DOJ trial that Google does use Chrome browsing behavior as a signal in search ranking. It’s not a hypothetical.

show 1 reply