logoalt Hacker News

simonwtoday at 2:41 PM1 replyview on HN

"static datasets with range payloads small enough to fit into a web response" fits a lot of workloads.

I expect that if your overall data is less than a GB this trick will work really well for you.


Replies

nicoritscheltoday at 3:50 PM

Most reporting I've ever worked on is based on live data and users expect updates. Range requests over parquet cubes are a cool party trick, but you outgrow it quickly once you need to start regularly updating the dataset such as to avoid full recompuation.

The next step in this journey is Iceberg (and a proper incremental pipeline), which can also be read directly in the browser via WASM either via DuckDB or without. This is from the same author as the parquet library mentioned in the OP https://github.com/hyparam/icebird

show 1 reply