alt Hacker NewsI agree. That's how our data is produced. We constantly generate real time data into CSV. As far as I can tell, I can't append to parquet file.
I agree. That's how our data is produced. We constantly generate real time data into CSV. As far as I can tell, I can't append to parquet file.
Parquet files are already built for append only. Just add a new file.
This is a new paradigm for folks who aren’t in big data — the conventional approach usually involves doing a row INSERT. In big data, appending simply means adding a new file - the database engine will immediately recognize its presence. This is why “select * from ‘*.parquet’” will always operate on the latest dataset.