Even the corporate/enterprise multi-level HSMs has had to compete with rapid improvement in disk drive capacity.
Buying a Sony or IBM HSM with 20GB tapes backing a 5TB store (built with 2GB disks) soon looks pretty stupid when drives exceeded 40 and 80GB. (Adjust your calendar or drive scaling to match your expectations.)
Back in the day I worked on a quote for a European weather agency who wanted a CD-ROM archive built with robots. CD-ROM! DVD-R was just coming up in those days. The effective product lifespan would only have been 2 or 3 years until large scale enterprise disk would have superceded it.
Nowadays with multi-TB tapes the minimum viable archivable unit is in the gigabytes. Nobody would be 2million files on a single LTO tape.
HSM is really a product category: do you want write-only archive (think weather data, satellite images, etc). Are some files expirable so you want to actually eventually "delete" files by re-writing tapes with only valid data? What's the allowable time-to-first-byte on retrieval? What kind of redundancy is appropriate? What is the operating cost vs capital cost tradeoff? And, importantly, what database is supporting the metadata and how is it maintained?
Antipatterns:
- grep on a caching filesystem backed with a tape library - monthly retrievals of the same file year after year (even LTO has a limited tape operation lifetime) - management failure/refusal to migrate to newer tech every 4 years - buying 10 years of capacity in advance
You can get a lot done with tar, mt, and bash in a grow-your-own nearline archive with a tape technology of your choice.
I was lucky to find you comment about how a lot can be done with tar mt and bash, I was doing something along the lines of trying to fit as much data as possible into a single tape without writing there more than it can be fit into one if possible, would you kind internet stranger happen to have a few wise words pointing me towards the solution or maybe a GitHub gist with an example of something along these lines? Thank you very much upfront!