alt Hacker NewsAnd now I'm curious whether there's a way to actually index external files (make these queries over 600GB faster) and have this index (or many indices) be persistent. I might have missed that when I looked at the docs...
And now I'm curious whether there's a way to actually index external files (make these queries over 600GB faster) and have this index (or many indices) be persistent. I might have missed that when I looked at the docs...
If the data is in Parquet they are already indexed in a sense. No further indexing necessary.
If they are stored in DuckDB’s native format (which I don’t use), it supports some state of the art indices.
https://duckdb.org/docs/stable/sql/indexes.html
I find Parquet plenty fast though.