logoalt Hacker News

jiggawatts • today at 8:15 PM • 1 reply • view on HN

It’s a Swiss Army knife for data!

I just used DuckDB 2.0 last week to compress 7 terabytes of Azure logs (NDJSON) to 2.5% of the original size in Parquet format in under two hours with a medium sized VM!

Those logs were then small enough to run ad-hoc queries in tens of seconds directly from the archive blob storage.

I created a “frozen DuckLake” and sent the signed blob URL to the security analyst.

No need for a server, no need for Splunk!

It’s easy to use, fast, can be plopped anywhere on a network, and it’s free!

If it just had KQL syntax support, it would be perfect…


Replies

nicoritschel • today at 10:19 PM

> If it just had KQL syntax support, it would be perfect…

Good idea for an extension, especially with the new PEG parser.