alt Hacker NewsI'm very curious about the convergence of the best in class fast OLAP products (StarRocks, ClickHouse) with Trino. It sounds like everybody is going for decoupled compute/storage, using S3 or similar as the storage layer, and thus forgoing colocated joins (ok I know ClickHouse joins suck)...
So what does this mean for ingestion (and indexing)? Iceberg V3? Paimon? Bespoke ingestion through the DB engine to do the indexing?
Replies
I’ve historically read this as ‘open format compatible’ but ‘native preferred’ - where this opens up market space and dev velocity - but it’ll be interesting to see if native storage differentiation gets dumped entirely. It just seems like ‘fork and optimize for our engine’ would always be tempting enough that you’d want a native play for when you don’t need the decoupling.
Clickhouse joins have been improving almost every month for the last couple of years. Maybe they still suck but a lot less
It's also interesting how Clickhouse / Starrocks can now also act as a query planner and executor on top of non-native formats (ex. Iceberg).
I assume the native formats will always be faster / more optimized but the need for Trino as a separate executor while running either of these databases seems to be close to gone.