S3 Is Showing Its Age

• seaweedfs

  37 21,489 9.9 Go

  SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, cross-DC active-active replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding.

  

> When it gets too out of hand, people will paper it over with a new, simpler abstraction layer, and the process starts again, only with a layer of garbage spaghetti underneath.

I'm pretty happy that there are S3 compatible stores that you can host yourself, that aren't insanely complex.

MinIO: https://min.io/

SeaweedFS: https://github.com/seaweedfs/seaweedfs (this one's particularly nice and is permissively licensed, in contrast to everything else)

There was also Zenko, but I don't think they gained a lot of traction for the most part: https://www.zenko.io/

Of course, many will prefer hosted/managed solutions and that's perfectly fine, but at least when you run software yourself, you are more in control over it and for the most part can also make the judgement on how hard it is to operate and keep operational (e.g. similar to what you'd experience when running PostgreSQL/MariaDB/MySQL or trying to run Oracle).

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• Zenko

  5 551 9.6 Gherkin

  Zenko is the open source multi-cloud data controller: own and keep control of your data on any cloud.

  

> When it gets too out of hand, people will paper it over with a new, simpler abstraction layer, and the process starts again, only with a layer of garbage spaghetti underneath.

I'm pretty happy that there are S3 compatible stores that you can host yourself, that aren't insanely complex.

MinIO: https://min.io/

SeaweedFS: https://github.com/seaweedfs/seaweedfs (this one's particularly nice and is permissively licensed, in contrast to everything else)

There was also Zenko, but I don't think they gained a lot of traction for the most part: https://www.zenko.io/

Of course, many will prefer hosted/managed solutions and that's perfectly fine, but at least when you run software yourself, you are more in control over it and for the most part can also make the judgement on how hard it is to operate and keep operational (e.g. similar to what you'd experience when running PostgreSQL/MariaDB/MySQL or trying to run Oracle).

• s4

  6 30 3.2 Go

  super simple storage service + data local compute + shuffle (by nathants)

s3 is not perfect, but close enough.

the ideal design is to use s3 for immutable objects, then build a similar system on ec2 instance stores for mutable/ephemeral data.

the one i use by default is s4[1].

1. https://github.com/nathants/s4

• mirror

  1 1 7.6 Go

  immutable disk mirror so simple that unborkable (by nathants)

i do the same thing!

i use spinning rust in x3 single usb external enclosures[1].

i mirror s3 data, both as an extra backup and to prevent s3 egress billing for random access.

s3 egress is for disaster recovery only.

1. https://github.com/nathants/mirror

• Peergos

  36 1,887 9.4 Java

  A p2p, secure file storage, social network and application protocol

  

AWS SDK is huge yes, but you can implement an S3 client in 300 lines of Java.

https://github.com/Peergos/Peergos/blob/master/src/peergos/s...

• vos

  25 1,881 9.5 V

  Vinix is an effort to write a modern, fast, and useful operating system in the V programming language

  

That's true for a vast majority of devs, but not for everyone. There are people like Jon Blow and projects like https://vlang.io

Suggest a related project