-
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
Seaweed File System
Discontinued SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, cross-DC active-active replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding. [Moved to: https://github.com/seaweedfs/seaweedfs] (by chrislusf)
We use it in anger at Splitgraph [0] to let people upload/download datasets and Postgres table snapshots (instead of storing them directly in S3).
Pros:
* Less platform-dependent. By self-managing it, we can also run deploys to GCP / Azure / Scaleway / other providers without writing a separate adapter for e.g. Azure Blob Storage.
* Python API [1] much more pleasant to use than boto3 (and can speak to normal S3). It doesn't do everything that boto3 does, but it supports everything we need (e.g. pre-signed URLs).
* minio server itself supports a large chunk of S3's functionality (e.g. SELECT API / AssumeRole / bucket versioning)
* Don't pay per request and for egress: this was a big deal since people might want to download large amounts of data from us (or make a bunch of small requests to download/upload a subset of data).
Cons:
* Have to manage own infrastructure. We run it on managed VMs so it's semi-managed, but we still have to provision block storage, set up backup policies etc.
* In a similar vein, scaling and availability all have to be DIY [2]. We haven't run into situations yet where Minio would be the bottleneck, but it might be something to keep in mind.
* Obviously not as seamless: you don't get things like Glacier or integration with other IAM.
[0] https://www.splitgraph.com/
[1] https://github.com/minio/minio-py
[2] https://docs.min.io/docs/distributed-minio-quickstart-guide....
So these guys have a storage system on top of Amazon S3 with a distributed (timestamp based) RW mutex locking, that uses NTP (github.com/beevik/ntp). That seems to be about it.
https://github.com/minio/minio/blob/master/pkg/bucket/object...
https://github.com/minio/minio/blob/master/pkg/dsync/drwmute...
And this is their test:
https://github.com/minio/minio/blob/master/pkg/dsync/dsync_t...
I just browsed quickly but it is littered with Amazon Simple Storage Service hardcoded bits like this:
https://github.com/minio/minio/blob/master/pkg/bucket/object...
There is not a single document that I can find that discusses the MinIO architecture. I guess "MinIO is a simple wrapper around S3 with a homegrown distributed state tech using NTP and it is 'fast!'" does not make for a sexy doc.
The column "trouble with OSS distributed DBs" without Jepsen tests has probably already been written. There should also be one about "competitor's mature product bashing blogs" are HN clickbait.
I am working on SeaweedFS, which supports S3 API for object store, and can also use Cassandra as the metadata db. Cassandra has been performing well for most SeaweedFS users.
The article listed many known Cassandra characteristics and cited as limitations. However, it all depends on use cases. There are no file system that works for all cases, and not all of them needs ACID, CA vs CP, etc. The rest points are not convincing either. They are related to how to design the data structure better.
Actually, SeaweedFS can use many other database/KV stores as the metadata DB. The list includes Redis, Cassandra, HBase, MySql, Postgres, Etcd, ElasticSearch, etc. https://github.com/chrislusf/seaweedfs/wiki/Filer-Stores
I did find one drawback for Cassandra as the metadata store though. One use case is that the customer uploaded a lot of zip files to one folder /tmp, unzip them, and then moved to a final folder. The are about 3000 files per second created and then deleted. Being a LSM structure, the tombstones quickly pile up and the directory listing was slow.
The solution was to use Redis for that /tmp folder, and still use Cassandra for the rest of folders. With Redis B-tree structure, the creation and deletion are cheap.
So it is all depends on use cases.