logoalt Hacker News

mlhpdxtoday at 6:15 PM2 repliesview on HN

Static web content and dynamic data?

> The archive currently spans from 2006-10 to 2026-03-16 23:55 UTC, with 47,358,772 items committed.

That’s more than 5 minutes ago by a day or two. No big deal, but a little bit depressing this is still how we do things in 2026.


Replies

voxic11today at 7:29 PM

That is just the archive part, if you just would finish reading the paragraph you would know that updates since 2026-03-16 23:55 UTC are "are fetched every 5 minutes and committed directly as individual Parquet files through an automated live pipeline, so the dataset stays current with the site itself."

So to get all the data you need to grab the archive and all the 5 minute update files.

archive data is here https://huggingface.co/datasets/open-index/hacker-news/tree/...

update files are here (I know that its called "today" but it actually includes all the update files which span multiple days at this point) https://huggingface.co/datasets/open-index/hacker-news/tree/...

show 1 reply
xandriustoday at 7:17 PM

I don't get what you meant with this comment.

show 1 reply