alt Hacker NewsI've often wondered why there aren't any (that I'm aware of anyway) Linux hierarchical storage managers (HSM). I realize they are typically used in big iron with things like tape as the base level of storage, but especially when it looked like Optane was going to be a thing it seemed like spinning<->SSD<->Optane<->RAM could be a huge performance opportunity. Though, I guess that's a "layering violation"... Honestly, at this point we've gone almost entirely SSD except for our ZFS backup servers, and that layering violation already has an SSD and RAM cache in it...
Replies
Even the corporate/enterprise multi-level HSMs has had to compete with rapid improvement in disk drive capacity.
Buying a Sony or IBM HSM with 20GB tapes backing a 5TB store (built with 2GB disks) soon looks pretty stupid when drives exceeded 40 and 80GB. (Adjust your calendar or drive scaling to match your expectations.)
Back in the day I worked on a quote for a European weather agency who wanted a CD-ROM archive built with robots. CD-ROM! DVD-R was just coming up in those days. The effective product lifespan would only have been 2 or 3 years until large scale enterprise disk would have superceded it.
Nowadays with multi-TB tapes the minimum viable archivable unit is in the gigabytes. Nobody would be 2million files on a single LTO tape.
HSM is really a product category: do you want write-only archive (think weather data, satellite images, etc). Are some files expirable so you want to actually eventually "delete" files by re-writing tapes with only valid data? What's the allowable time-to-first-byte on retrieval? What kind of redundancy is appropriate? What is the operating cost vs capital cost tradeoff? And, importantly, what database is supporting the metadata and how is it maintained?
Antipatterns:
- grep on a caching filesystem backed with a tape library - monthly retrievals of the same file year after year (even LTO has a limited tape operation lifetime) - management failure/refusal to migrate to newer tech every 4 years - buying 10 years of capacity in advance
You can get a lot done with tar, mt, and bash in a grow-your-own nearline archive with a tape technology of your choice.
It’s just easier done at the application layer. At a previous engagement, in the Before AI era, we originally had a block-level tiering solution that went from RAM to local nvme to NAS on striped flash to S3. It worked okay, but the application-level tiering worked better.
Would have loved a read-through union filesystem but you pretty much had to do your own.
There is bcache and dm-cache. bcache was the basis for bcachefs but I don't know anyone who uses them or dm-cache
https://en.wikipedia.org/wiki/Bcache
https://en.wikipedia.org/wiki/Dm-cache
Sometimes I shoot over 100GB of raw photos and don't finish processing it all in 1 session. It would be nice if that got cached in an SSD rather than having to read the hard drive every time. But it's not a big enough issue for me test one of the caching layers.
Isn't bcache that thing you're looking for?
RAM is always used first for caching, so it's a given.
RAM cache is already layered in front of persistent storage. When it comes to easy hybrid SSD/HDD setups bcachefs was close to deliver, but then things jumped the shark.
There is LVM Cache which uses dm-cache. That allows you to use SSDs to cache HDDs. Either can be RAIDed too. You can choose your file system because it happens at the block layer.
Besides the trivial caching on SSD there's also ways to do write buffering quite easily. I did that once a long time back when SSDs were too expensive for getting a 2TB disk. So probably ~10 years back.
That's exactly why I keep Windows on my NAS. You can configure storage tiering with arbitrary redundancy. You can have different redundancy on SSD and HDD tiers. As far as I am concerned ReFS gives me all the features of ZFS and doesn't eat my RAM. Also BitLocker + Secure Boot aren't pain in the ass there.
When you're running ZFS (or probably some other clever layering violation like bcachefs) on heterogeneous workloads, you will quickly run into the limits of built-in assumptions. At least that's my experience:
- ARC may require manual tuning for size and metadata size to accommodate page cache, in-app caches, file count etc
- L2ARC and SLOGs are very obviously useful for some clear workloads, but have some cliffs you don't expect, such as ram required for L2ARC mapping, unexpected cold/hot/boot behavior
- Reasoning about reliability becomes harder the more tiers you have
Just imagine, to pick a simple example, you're running postgres, thousands of sqlite3 dbs and one huge duckdb (larger-than-ram) on a single such system - it would perhaps manage to adapt to one such workload at a time, but would it balance all three at the same time?
So much as I'd love to see this direction, I can also imagine the pains coming with it.