logoalt Hacker News

remywang • today at 3:25 PM • 1 reply • view on HN

> Most formats split the table into large groups of rows and store the columns separately inside of each group (this is known as a "row group"). This idea goes back to PAX, from Weaving Relations for Cache Performance (VLDB 2001), and it's how Apache Parquet files work today.

PAX’s “row group” has the same size as a disk page, whereas Parquet row groups are much larger so a page stores only data from the same column. I’ve been trying to track down the origin of this change in design but haven’t been able to find the right paper


Replies

nwatson • today at 3:42 PM

Related US Patent US7024414B2 [0] from Addamark (later Sensage) filed 2001 August, "Storage of row-column data". The patent outlines columnar compressed storage similar in many respects to parquet.

[0] https://patents.justia.com/patent/20030028509