alt Hacker NewsHow did I miss zstd?
Here are my benchmarks for 2.3 GB of jsonl, on a laptop. Compressed size, compress time, decompress time; using defaults.
gzip 7.3% 21s 9s
bzip2 4.6% 251s 50s
bzip3 3.3% 82s 69s
zstd 6.9% 2s 3s
lzma 4.7% 51s 3sReplies
But zstd is super tunable. Where gzip gives you compression levels from 1 to 9, zstd gives you up to 22 for ultra compression and negative compression levels for ultra fast. The ultra fast options so fast that they are great as a substitute for memcpy if your CPU is already waiting for other things, like DRAM.
zstd has a built-in benchmark mode to compare different compression levels, e.g. `zstd -b1 -e9 [FILE]` to test levels 1 to 9 (try up to 22 if you have enough spare time)
zstd with better compression level would be nice - these numbers are not really comparable since both time and compression level are too different
At what levels? There’s no guarantee that the default compression level is comparable. You have to normalize by time spent compressing.