-
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
I recently started looking into using an LTO-7 tape drive that I got handed down, along with a few dozens of pristine LOT-6 tapes, for archiving purposes. I got to play around a bit with SAS HBAs, and was kinda shocked how much of a difference that can make in the user (or shall I say sysadmin?) experience: LTO-6 tapes are spec'd to transfer rates of around 150MB/s, so well within the reach of even the the first SAS gear generation. However, the very first SAS HBA with external SFF-8088 connector I managed to get my hands on (an LSI SAS1068e) topped out at a disappointing 80MiB/s, no matter what I tried in terms of blocking and buffering. Switching to a more modern (but still old) LSI SAS2008-based HBA got me close to the theoretical maximum.
Then there's the (to me, still open) question of how to best use the actual tape storage capactiy... Since my hardware is newer than LTO-5, LTFS (https://github.com/LinearTapeFileSystem/ltfs) is an option for convenient access, especially listing tape contents, but that could make it hard for other people down the line to restore data from the tapes I create.
It's probably safest to assume that tar will always be there, at least wherever there's tape, too. GNU tar also handles multi-volume/-tape archives, which seems like a necessity if you need to back up amounts of data that exceed a single tape's capacity. Then again, if you want to use encryption with actual tar (important for the kind of data I need to archive), your only option seems to be piping the whole archive through something to compress the stream, which will make accessing individual records in the archive opaque to the drive itself... and you can't just dispose of individual keys to make select parts of the archived data go away for good, either.
Also, I would like to conserve as much tape as (conveniently) possible in my archiving adventure. There's "projects" (i.e., top-level directories of directory trees) that consume more than one tape of their own, and then there's smaller projects that you can bin-pack together onto tapes that can fit more than one such project.
I've started implementing a small python wrapper around GNU tar to solve a number of these problems by bin-packing projects into "tape slots" and also keeping track of tape-to-file mappings in a small sqlite database, but a workable solution for the encryption problem(s) is not something I managed to come up with yet... If someone has an idea (or better yet, a complete and free implementation of what I am trying to hack together :)), please be so kind and let me know!
Is there a unix-style streaming tool, like tar/zstd/age, that does forward error correction? I'd love to stick some ECC in that pipeline, data>zstd>age>ecc>tape, cause I'm paranoid about bitrot. I search for such a thing every few months and haven't scratched the itch.
The closest is things like inFECtious, which is more of just a library.
https://github.com/vivint/infectious
