alt Hacker Newsgoogle.com/goto: Google's anti-scraping update
Comments
I did an interview with Google around 20 years ago, where they posed a challenge involving tracking which specific search results people click. It's obvious in hindsight the solution required rewriting all the urls to redirect through their servers. Note this was in the days before they already did so as a matter of course.
I failed to gain traction on the problem, because to me the very idea of doing such a thing was too reprehensible to seriously consider. It broke an unwritten contract between the company and the user's expectation of how websites worked. You expect to be able to do things like right-click a link and copy the authentic URL, or hover to see where it wants to take you. The notion of obfuscating the link beyond easy recognition and polluting it with tracking markers felt misleading and, well, evil. A move that would mainly only benefit Google, and not it's users. I (quite mistakenly) presumed this opinion would be obvious and self-evident to anyone who spent enough time around the early web to understand its norms.
I explored other ways of achieving the goal, but it clearly wasn't the answer the interviewer sought.
I'm more seasoned now, and experienced enough to say with confidence the approach was wrong. This may seem like a small thing, but a series of misteps and chronic failure to adequately advocate for users is what has led us to the toxic waste dump that so much of the Internet has become today.
I'm really glad to have fresh alternatives (like Kagi), and can't wait for the cultural zeitgeist among developers to swing back around to valuing users as human beings and living up to the trust they place in us.
Direct URLs in Google search results have been replaced with redirect URLs in the form of www.google.com/goto?url=<opaque base64 string>.
The base64 data appears to consist of a very basic protobuf structure, containing a long string of bytes in field 2 which presumably identify the URL.
Sometimes, these redirect URLs take a perceivable amount of time to load, which is very irritating.
I was suspicious when they started obfuscating URLs in their own browser, then on their SERPs, and now this...
For many years, I had my filtering proxy rewrite the URLs in the way mentioned in the article.
Almost exactly a year ago, Google stopped working without JS. I stopped using Google.
Now they're upping the game, and as the article (which is a bit of marketing itself) admits, those who have the resources can still blast through these obstacles while those who don't are locked out.
Since the article brings up "AI scrapers", I'll just point it out as being the latest scare-tactic for coercing people to give up the privacy, anonymity, and (browser) freedom of an open interoperable Internet.
While a lot of people are concerned with local model performance, I wonder how feasible is it now to run a local indexed web search? Surely running an old school Google is possible with the beefy AI rigs today. I know the problem will be crawling which would be bottlenecked by the ISP but I use Google to search SO, Wikipedia, programming language docs, Github issues, and AWS docs. I think a feasible workflow would be to build a set of sites of most interest to you and then prioritize those in crawling.
While typing this out I remembered https://en.wikipedia.org/wiki/Google_Search_Appliance which I never personally used but shows feasibility for the idea. I'm pretty sure one of the newly-announced Macbooks is more than up to the task of matching GSA's offering.
I've been using DuckDuckGo for years now, ever since it became noticeable that two different people searching for the same search term would get two different results back from Google. Meaning they were no longer completely reliable: they might show one person a result that they hide from the other person by burying it on page 3 where few people ever look.
DDG's search results have been poorer recently than they used to — I often see completely unrelated results (to the point of my saying "Why in the world did that come back as a search result??!?") starting from page 2. And yet, I still use them, simply because they aren't Google.
When Google stopped paid API search a few months ago, I looked for an alternative for my agents that I felt would be sustainable (one-time setup, then out of my mind). I quickly excluded SERP as I feard Google would pull exactly this type of shenanigans to cut them off.
I somehow found Mojeek and settled on it. I had never heard of them. Unlike Kagi, their business model is ads (so they hold no particular moral high ground). But they have a cheap, working paid API.
What I was astonished by is the quality of the results. For my uses, it's undistinguishable from Google. The conventional wisdom is that web search is a Google-sized problem. How did those obscure Brits pull it off?
So why are we angry about that ? I mean the end result for the users are exactly the same, it matters only for bots.
Google have such a (justified) bad reputation that whatever they do, people assume it’s entishification. I don’t believed it is on that matter.
At this point, we can all just drop SEO [0]. We are writing content for a robot that hides the source of information.
They are allowed to scrape everybody else but get their feelings hurt when they get scraped....oh yea they respect robots.txt. Guess what, scraping everything that is public is legal.
It's sad that instead of searching things other people put up, we're basically asking sam or dario oracle to tell us the truth. The people should be furious. But we've internalized this idea that they are somehow better.
What does this mean?
This text appears on this page https://www.autom.dev/blog/google-search-goto-links
> The real URL is in the Location header on /goto. Request that URL. Do not follow the redirect.
And this text appears on this page https://www.autom.dev/blog/google-goto-url-fix
> Do not follow the redirect. Read Location.
That's what a redirect is, reading the value of the location header and then requesting it. How do you not follow the redirect by reading the location header? Once you've made the request to the /goto url, with GET or HEAD, to get the location header, google knows you're interested in whatever it is putting in the location header and can assume you're going to go there, if you're letting the User Agent (curl or the browser) go there for you or not.
Dont worry in a year or two, Google wont even redirect you to the actual true url, instead everything will be a page with all links rewritten so all http is tunnelled thru them.
Hmm that's actually pretty clever. They can serve each result page slightly different encrypted links and it should be obvious right away if it's a SERP bot (trying to grab a page of links) or a human that just picks a few here or there.
I wonder if this would also work on other sites getting hammered with bots. Allow each anonymous user 1 "real" page load then turn the rest into encrypted links that the web server can decrypt. If a session cookie with reputation exists, stop screwing with the links.
Kind of annoying but it'd allow tracking if the same agent/bot is churning through IPs/User Agents.
Can someone explain why this matters? Not being flippant I just don’t understand why this would be important.
If you were scraping only Google with all the IPs you can get, then this change really slows you down.
If you're trying to fight scrapers on a small site, delay links can only flatten bursts. If bots can only scrape at human speed per IP, they can just scrape 100x as many sites at the same time. Once every bot operator does that, total traffic will return to the original level.
If I hadn't already ditched Google long ago, this would be more than enough to chase me away.
This is such an incredibly annoying and deliberate defect. I want an extension that will let me resolve the true url without visiting the site, or even rewrites the entire page to show the url.
1789154298 | Google's Emissions Climbed 48% Since 2019 due to AI | https://www.gadgetreview.com/googles-emissions-climbed-48-si... | https://news.ycombinator.com/item?id=49663855 | 0 comments
That's it. I'm done with Google search. I just found you can add Kagi to Safari: https://apps.apple.com/us/app/kagi-for-safari/id1622835804
At least they seem still provide results for my searxng instance. I mean sure, they are horrible but duckduckgo just blocks most queries (and I'm the only person using the ip / seraxng instance)...
Next i'll do is to implement tavilly, exa, tinyfish etc. as search engines for searxng. No agents, no mcp, just their search api endpoint.
Brave search (free) or Kagi (paid) are able to replace Google and not feel like I'm missing out.
Brave has its own independent index which is cool.
> Combined with earlier moves like removing &num=100
Removing this made google search horrible to use. I often use command+f to quickly identify relevant search results, but doing it on 10 results at a time is so laborious that I just don't bother using Google search, resulting in less searches and use of other tools instead.
I've never heard of this particular SERP provider, but some marketer is very excited they wrote this blog post right now (500+ upvotes on an seo blog post).
And their fix here - they just resolve all the urls- which I suppose could make the service slightly more expensive? But otherwise isn't that what every similar provider/ crawler etc will do and this change will only hurt users?
I... Don't see it? It's the result page right? I just search some random string on Google and the results are all direct URLs. Do they get resolved via javascript after page load and replaced automatically? Or am I looking at something else?
I had tried now but I don't see the goto. Is it possible that has been deployed only for USA users ? (I work and live in UE). If so: probably a vpn can help you for a while. In the future: I think I'll really go for payed search engine.
Is it a move to sell more of the paid Google Search API calls? If so, that's a sign of distress.
Surely, the cost of serving search results to bots isn't that high
It's funny. Recently I looked into what Google is doing (see at https://blog.miloslavhomer.cz/how-google-sees-your-site/).
It's a lot of work to get the data to build an index. Why would they give it to everyone for free?
> The real URL is in the Location header on /goto. Request that URL. Do not follow the redirect.
What does this mean? Isn’t the location header the redirect? Am I not following the redirect by requesting the location header url?
I really like the idea of turning the search index into a public utility. It is one of those natural monopoly coordination problem things. Just quasi nationalise it for economic efficiency. Ofc google can still sell adds against their own ui (like everyone else). Hopefully this move will move that idea closer to reality.
What are you going to steal from Google?
The Internet has been dead for years and, after the Scrapocalypse, the small living remnants are behind a login wall.
Google can not provide you anything you couldn't find on either your local ZIM archive or the Media you consume.
Can confirm that when using google not logged in. Now when sharing a link from google search, I won't get the actual link. This will certainly help google's tracking.
Google has gone so bad over the past few years. You only get like 8 results per page. I remember there was a time that I wonder how a site get reached if it ranked on the second page, when I can set the number of results to be 50. The censorship is also really bad, and google doesn't even tell you the results are censored, returning totally nonsense results while other search engines work normally.
I've been supporting Brave search which returns 20 results per page and has other features. It used to be not good a few years ago, but now the results are often better than google's.
Is there any market for an open-source internet search engine that is paid for by honest ads?
Been using Brave Search now for a while including its Brave AI and aside of sporadic times I never needed Google (albeit Brave Search is slower to Google, you get used to it)
The udm=web parameter has been doing the job for a while now. It's a shame you have to know about an undocumented flag to get the old behavior.
I use ChatGPT for almost all my searches now. I’m not joking.
Hopefully this also helps against the rampant CTR manipulation that’s made some search results purely a measure of spend.
Of course, this move is user hostile because you don't see what you're being sent to.
This seems good. I'm not sure why I should be upset that Google is preventing abuse of its service.
The enshittification continues.
Nothing good ever comes from businesses desperately trying to protect their moats rather than making their products better so they don't need to.
"Combined with earlier moves like removing &num=100 and tightening BotGuard/SearchGuard, Google is steadily raising the cost of naive SERP scraping."
Another "move" is suing companies like Autom, e.g., SerpApi
Google's Amended Complaint from their suit against SerpApi
https://ia801008.us.archive.org/25/items/gov.uscourts.cand.4...
"30. Copyright holders have authorized Google to implement access controls like SearchGuard for the content they license to Google, and in some cases insisted that Google do so. Googles authorization takes many forms. For example, Google has an agreement with a prominent licensing partner that holds copyrights to millions of works that it licenses Google to use in its Search results. Under the parties agreement, versions of which date back to 2017, Google is not only authorized, it is obligated to use commercially reasonable efforts to safeguard the licensed content against unauthorized third-party access. Other license agreements contain similar obligations. For example, another major content provider requires that Google ensure the content it licenses will not be available for download by third parties, thereby authorizing the implementation of technical access controls."
"31. In other cases, Googles authorization to implement access control measures like SearchGuard is part and parcel of the grant of licenses themselves, as Google and its licensors recognize that the value of the licensed rights would be undermined if others were free to access, take and resell the licensed content without restriction. For example, Google has a licensing agreement with Reddit, under which Reddit licenses Google to use the copyrighted content of both Reddit and its users in Search Services."
"32. Googles licensing partners have also expressly requested that Google prevent unauthorized access to licensed content. For example, when Reddit suspected that scrapers like SerpApi were accessing, taking, and reselling the content that Reddit had licensed to Google, it specifically asked Google to employ technical measures to prevent such unauthorized appropriation."
But this does not account for material that is not covered by the "license with a prominent licensing partner", its license with "another major content provider" or its agreement with Reddit
Google not only uses SearchGuard on SERPs containing links to the content covered by these licenses, it uses SearchGuard on _all_ SERPs
Google needs more than a "goto" update. It needs to update its terms to require _all_ copyright holders for the materials it has indexed and cached to give Google authorisation to use "technological protection measures" to deny access to certain members of the public, e.g., Google's perceived competitors including any Google user who "searches too fast"
If you don’t like this, just don’t use Google folks. There are alternatives, use them.
Bing has done that for years. Hated it.
Most people i see using ChatGPT for any search related work in everyday life.
Maybe Google is observing this?
Note: Article published by Autom.dev which, from a quick read of their homepage, seems like it scrapes Google search results in violation of Google's terms of service and sells those results to customers via an API. That's just my quick read of it, though, so this could be wrong.
> The url parameter uses a custom, Google-specific encoding.
So that will be figured out, making the whole exercise void. Google knows this. What is the real goal here?
Divide et impera. People need to learn. Get together, do things together, create alternatives, use that, maintain position, do not let outsider saboteurs neuter the project. Problem solved.
I use Google maybe 10 times a month.
As much as I am sad that Google died like 15 years ago, I am past the mourning phase. That was when they announced they were shifting from returning websites to "returning answers" and it has been a long slide into shittification
I do enjoy using their free AI. For actual web search I actually like using Yandex. It reminds me of old Google, returning reasonable results and much less "shaping results to please our corpo-political masters". It is surprising to see how much they have stripped from our view - long tail results, actual results for product reviews and not ad spam, no preference for 20 page recipe sites.
There are still illegal streaming sports and movie sites everywhere (who knew) and all other seedy corners of the internet that have been neatly erased by Google. It makes me nostalgic for that brief window of time when the web was truly uncontrolled, when page rank had meaning and you didn't know if your search would return 0 results or 4,000 pages, which you could actually browse.