alt Hacker NewsI was processing compressed .jsonl files recently (JSON lines format). I found that lzma gave a much better compression than gzip or bzip2, which helps for archival costs, but it's challenging to work with as software support is lacking. I do duckdb processing which supports gzip transparently. There's an extension for bzip2, but not for lzma or bzip3.
I ended up using gzip because it's best supported by the software I use and most likely to have support in software I adopt. But it gave the worst compression results of the options I tried. These bzip3 numbers certainly give me FOMO...
Replies
zstd is the go-to compression format these days. It's even supported in low-level software such as many linux filesystems.
I don't know much about duckdb but it looks like it supports zstd too: https://duckdb.org/docs/lts/data/json/loading_json
For structured logs and json I've found a lot of success with PPM-style schemes.
If your JSON file has many of the same object, you could see ratios in the single digits.
I'd recommend trying openzl for jsonl.
[dead]
For that type of structured data (logs and such), a custom dictionary can be extremely effective. Zstd among others support generating a custom dictionary. You just run zstd --train over the data first, and then feed that in when you run zstd. For example: I found ~10 gigabytes of Usenet headers compress to ~700 MB using Zstd and 1 MB shared dictionary -- and that's with each header individually compressed, so o(1) lookup time.