logoalt Hacker News

1vuio0pswjnm7today at 8:32 PM1 replyview on HN

FWIW, this site is fast, works perfectly with a text-only browser, e.g., links (1.4MB static binary), elinks, or without a (bloated?) browser, e.g., retrieve all the pages, catenate and read as a single HTML file or convert it to, e.g., SQL, CSV or a plain text file

All the /p/ URLs are in the sitemap, all the site's pages can be retrieved over a single TCP connection

https://debloat.dev/sitemap.xml

This, i.e., retrieving all 200 /p/ URLs over a single TCP connection (using HTTP/1.1 pipelining), results in a 1.9MB HTML file comprising all the /p/ pages, including response headers. NB. This isn't "crawling". All URLs are known before the connection is made

No cookies, no Javascript

Other than CSS, no bloat


Replies

etermtoday at 9:02 PM

I about to splutter that 1.9MB is a huge sitemap, but it's 12kB.

Do you mean crawling the entire sitemap and concatenating the results is 1.9MB?