logoalt Hacker News

mediamanyesterday at 9:34 PM2 repliesview on HN

Clickhouse has a more intentionally built ingestion system. Duckdb has concurrency limits so you can't have a writer and a reader on the same file if they're not the same process (multiple readers is fine).

But that's not too hard to work around. You can either have a single process that owns both writing and reading that file, or you can do a data lake where you post updates as parquet files into object storage, and duckdb handles the catalog. The Quack protocol also basically fixes this (though still in beta).

With Clickhouse, you can of course still have tenant separation, but you have to do it by managing users within Clickhouse that map to users/tenants of your main app, so that you can restrict SQL access by tenant to only their data store. Not a huge deal but I just like the Unix "it's just a file" simplicity of "Tenant A gets to run arbitrary SQL against their separate read-only, no-ATTACH duckdb file".


Replies

jamesblondeyesterday at 9:47 PM

DuckDB's sweet spot is for 'smallish' lakehouses. So, ingestion should not be via duckdb, but rather something like Python/DLTHub for small scale or Spark for large scale or Kafka/Debezium/Flink for streaming data.

show 1 reply
bocyesterday at 11:00 PM

Ducklake supports postgres for the catalog, so you get the postgres concurrency benefits + duckdb engine to read the parquet files in the bucket.