alt Hacker NewsAs for the DELETE issue the easy solutions are:
-Delete it batches
-Delay between batches
-Preload the rowids before deleteing with SELECT (Select does not block)
Additionally if data was added sequentially primary to the same table the data is likely stored this way in the file and deleting it in this or in reversed order can be faster (depends on storage medium and other factors).
Row ID preloading is an extremely effective technique—and not just for SQLite. I’ve also used it to great effect on massive Aurora MySQL or Postgres clusters since I could send the SELECT to a replica, and the whole point of deletions was that index memory pressure from the row filtering was putting tons of CPU and buffer cache pressure on the db.
If you’re in a situation where partition pruning or other strategies for getting useless data out of the hot path don’t make sense, this is a killer strategy.