A distributed Posix file system built on top of Redis and S3

• juicefs

  42 9,774 9.7 Go

  JuiceFS is a distributed POSIX file system built on top of Redis and S3.

  

The file format is described in https://github.com/juicedata/juicefs#architecture

We will provide tools to dump the metadata as JSON, then you could recover your files using that.

• awesome-go-storage

  7 4,256 4.1

  A curated list of awesome Go storage projects and libraries

  

This is neat! I am quite a fan of all the go based file systems that are springing up. Question: what are the main compare and contrast points between juice and seaweed fs?

Here is a compendium for those interested:

https://github.com/gostor/awesome-go-storage

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• redisraft

  8 799 7.4 C

  A Redis Module that make it possible to create a consistent Raft cluster from multiple Redis instances.

  

Redis can be persisted with RDB and AOF, can also be replicated to another machine. In the cloud, you don't need to worry about that, hosted Redis are ready to use.

The is an ongoing effort [1] to improve the persistency and availability in general, which is expected to be GA in 2021.

[1] https://github.com/RedisLabs/redisraft

• containers-roadmap

  80 5,137 2.0 Shell

  This is the public roadmap for AWS container services (ECS, ECR, Fargate, and EKS).

  

FUSE is not support by AWS Lambda, so we can't mount JuiceFS in Lambda.

We can have a SDK to access JuiceFS from Lambda, similar to S3 SDK, when you need to use JuiceFS outside of Lambda.

Same to Fargate, we can not mount JuiceFS in Farget because of lacking FUSE permission, people are asking for it[1].

https://github.com/aws/containers-roadmap/issues/412

• s3-benchmark

  4 773 0.0 Go

  Measure Amazon S3's performance from any location.

TTFB in S3 is 20-30ms around the 50th percentile. it can go much higher at p99 [1]. In any case, rotational latency for HDD drives is an order of magnitude lower (typically 2-5ms for a seek operation).

S3 is great for higher throughput workloads where TTFB is amortized across larger downloads (this is why it's very common to use S3 as a "data lake" where larger columnar files are stored, usually at the order of hundreds of MiB).

I think it's an interesting project but perhaps explaining the use cases where this solution is beneficial would go a long way here.

[1] https://github.com/dvassallo/s3-benchmark

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

Suggest a related project