alt Hacker NewsWere you able to process truly large datasets? I worked at a very excel company once but we kept running into row limit issues when scaling workloads and eventually migrated almost everything away from it. What industry were you in at the time?
Replies
coliveira • yesterday at 11:57 PM
The same thing that works for databases also works for excel. You shard the data and run your analysis on the set of shards in excel. It's not elegant, but can solve many problems.
Aerospace. I was interacting with data sets on the order of hundreds of millions to low billions of rows. This particular company had legacy WinForms applications running against SQL Server. Excel can directly connect to the db to run complex, multi-step queries (IIRC it holds the connection open so you can do things in temp table). Hitting “Refresh” in Excel can trigger a lot of work. And then it can do a ton of processing locally. IIRC, there were no row limits on what excel could process locally (so long as the final result stayed under the sheet limit).