Open-source projects categorized as tiered | Edit details

tiered Open-Source Projects

  • GitHub repo autotier

    A passthrough FUSE filesystem that intelligently moves files between storage tiers based on frequency of use, file age, and tier fullness.

    Project mention: Any news on Qtier or similar reaching QuTS Hero? | reddit.com/r/qnap | 2021-08-09

    Given current pricing, larger SSDs, in terms of about 1TB aren't actually twice the price of 512GB ones, which in turn aren't twice as expensive as the 256GB units, seem to be better buys. At the same time, there are benefits to going SSDs for certain capacities and workloads, while in many cases a correctly functioning tiering system is more beneficial. Qnap's Qtier, however, is using dm-thin/md-adm features that are not in use/available by design in QuTS Hero and as such their regular QTS's feature of Qtier is unavailable on the ZFS-based variant. SSDs can be used for SSD pools or caches (as in ZIL for writes/L2ARC for reads/metadata for metadata and deduplication tables or a mix of those) only, similar to QES. Contrary to QES target audience, many ZFS-powered units are reaching customer base that is more home, SOHO or even light on the budget, so going with "huge" (in terms of disk quantity and overall capacity) and resilient all-flash pools is merely a dream, so the caches are more or less the goto option. As stated earlier, this leads to often either overexpenditure or underutilisation of the disk resources. Still, Qnap seems to have been quite tight-lipped about any possibility of Qtier-like solution being offered in their QuTSHero stack for a long time (presumably either because they don't want to offer it to not canibalise any of the remaining comparative QTS only or QES solution sales, or due to their inablity to come up with a solution). Given that the underlying system is a tuned linux, I wonder why autotier-like (https://github.com/45Drives/autotier) FUSE filesystem has not been prepared as a "replacement"/alternative. Sure, block-level monitoring/tiering would have been even better but the nature of COW/ZFS does not play well with regular block-monitoring tools (one could splice something up but that would require yet another huge table for block counts tracking to combine spinning rust and silicon traps in persistent and ephemeral levels into one global map for storage). However, a FUSE filesystem of autotier deals in file-level analysis, which would greatly increase usefulness of a mixed SSD and HDD pools appliance possibly without incurring much of a performance impact. Qnap certainly has the manpower or funds to prepare a solution that is unique/optimised for their devices that while not as good at moving hot data around as the block-level solution, would be the next best thing without too much of a hustle and it could still joggle around data when required. The question then remains, why haven't Qnap gone through with releasing such a solution, or maybe there is one in the works that Qnap plans to add as a showcase for QuTS Hero v.5? It could greatly improve the QuTS perspective in devices such as TS-673A (wherein one could than get a 3-tier solution with 2x NVMe SSDs for hottest files/data or caches, and the 6 drivebays populated with 3x HDDs for cold and 3x SATA SSDs for warm data layers) or its 8-bay counterpart, or the TS-h973AX (wherein we're getting 3 tiers by design with 2x U.2 for hot sauce or caches + 2x SATA SSDs for warm data + 5x HDDs for cold storage) as example units of somewhat lower pricing in the ZFS-based range/spectrum. This could certainly boost sales of the ZFS units. Or maybe, I've just been deaf and blind to the corporate PR to that degree that I've missed Qnap official unveiling of a future showcase piece plans for later? Or maybe such features have crept silently in some beta releases already and been dropped due to some issues?

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2021-08-09.

tiered related posts


Project Stars
1 autotier 85
Find remote jobs at our new job board 99remotejobs.com. There are 29 new remote jobs listed recently.
Are you hiring? Post a new remote job listing for free.
OPS - Build and Run Open Source Unikernels
Quickly and easily build and deploy open source unikernels in tens of seconds. Deploy in any language to any cloud.