alt Hacker NewsThe problem with using database blobs for load/save is that you usually need a full database client in the application. SQlite advertises that use case, but it is complete overkill. You never need to run any sort of complex SQL query on an image file format for instance. Using XML+ZIP in this day and age is also a strange decision, but at least that way the data is inspectable with unzip, a text editor and an image viewer (assuming they use a standard image format to store the raw pixel data).
Replies
Sqlite is probably overkill, you will probably never have actual relational data in an image format, But what it does bring to the table is a built in b-tree based storage, that is, you don't need to load the entire file into memory to edit it, in a prior age we would use Berkeley db for this. Sqlite in this role(a file format) is probably best thought of as a better superset of the berkleydb style key value store, more than one table per file and additional columns/indexes to keep metadata in.
Nothing wrong with XML, it is well understood, and the tooling is pretty good. But partial loads/edits is one thing it can not do.
> Using XML+ZIP in this day and age is also a strange decision
I agree with you that SQlite is overkill but honestly curious to know why you think xml+zip is strange? what would you use instead?
> You never need to run any sort of complex SQL query on an image file format for instance.
Sure you will. Plenty of features that don't exist, or are implemented badly, because you can't easily do it.
Quick mental translation table: if you think "iterate over every ..." or a `for` loop, that's your SELECT query. If you think about `if` conditions, that's the parts that go after FROM clause.
Think of Lightroom or automation over files. Many semi professionals from wedding photographers to designers want some form of batch automation and organization system over their files.
Several megabytes/gigabytes assets and you want to extract metadata, a preview, running as a batch some filter/compression/, conversion to CMYK, text injection ... fast partial read/write access would be nice. Right now, most reads are performed through indexes because those files are slow to read.
If we take 10k sqlite files and want to retrieve a row, we would be around 3s on SSD, maintaining preemptive indexes become less important for a lot of use cases.
Change management and versioning also becomes quite efficient - sqlite can be configured to not offset bytes, so CVS like Epic Lore can efficiently delta the files and store minimal delta, or the file format itself can keep its edit history. Oh and it's 3x-10x less bytes without compressing the whole thing, so pages are stable through time.
About needing SQLite client, it's real but it's roughly the same size as an XML parser.