logoalt Hacker News

lsureshyesterday at 12:24 PM1 replyview on HN

Feldera co-founder here. Great discussions here.

Some folks pointed out that no one should design a SQL schema like this and I agree. We deal with large enterprise customers, and don't control the schemas that come our way. Trust me, we often ask customers if they have any leeway with changing their SQL and their hands are often tied. We're a query engine, so have to be able to ingest data from existing data sources (warehouse, lakehouse, kafka, etc.), so we have to be able to work with existing schemas.

So what then follows is a big part of the value we add: which is, take your hideous SQL schema and queries, warts and all, run it on Feldera, and you'll get fully incremental execution at low latency and low cost.

700 isn't even the worst number that's come our way. A hyperscale prospect asked about supporting 4000 column schemas. I don't know what's in that table either. :)


Replies

kwilletsyesterday at 8:04 PM

This site is underweighted on OLAP. Columnstores were invented for precisely this use case; nobody in the field wants to normalize everything.

Which brings me to the question, why a rowstore? Are Z-sets hard to manage otherwise?

Another aspect of wide tables is that they tend to have a lot of dependencies, ie different columns come from different aggregations, and the whole table gets held up if one of them is late. IVM seems like a good solution for that problem.