alt Hacker News> People typically start with Excel and graduate to Pandas somewhere in the GB range. Pandas serves them well into the 10s of GBs range, and then they start hitting memory issues, slow computation, or become frustrated with Pandas’ baroque API.
Assumes that a project moves beyond 10s of GBs. I guess 99.9% of projects that import pandas fall well below this threshold.
Replies
minimaxir • today at 3:20 AM
At my work I had convinced the ML pipeline engineers to switch from pandas to polars for even small ETL pipelines and there were notable performance gain with better CPU/memory utilization.
If a library is performant at large datasets, it is likely performant at small ones too.
➕ show 1 reply
True, but also if Polars and DuckDB offer a similar experience with the ability to scale beyond that range, why not use them (for new projects)?
This isn't a call to arms to rewrite everything in the new shiny, just consider the new shiny for new shiny things