alt Hacker NewsThe crawlers would have still just hammered their site though, right?
Replies
pavel_lishin • yesterday at 7:52 PM
I think the idea is that they could store the data in BigQuery, and point users of the site there.
➕ show 3 replies
The crawlers would have still just hammered their site though, right?
I think the idea is that they could store the data in BigQuery, and point users of the site there.
Yes, if I wanted to put a nice HTML interface over the query results then I would still end up with the same problem where some combinatoric explosion of query parameters to the `/search` endpoint, most of which are cache misses, leads to many many TBs of network egress.