awesome-go-storage VS s3-benchmark

Awesome Go Storage (GitHub)

Any many many more. Check https://github.com/gostor/awesome-go-storage

C is significantly slower at concurrency when implemented naively. It's as fast as languages like Go when implemented using the same techniques, which is not obvious and trivial to use like in a higher level GC'd language. GC actually helps out a ton there, for example look at the complexity of async/await in Rust which requires the notion of pinning.

https://github.com/gostor/awesome-go-storage#database

https://java-source.net/open-source/database-engines

Not a database but honorable mention, LMAX disrupter: https://lmax-exchange.github.io/disruptor/

You may want to start from here: awesome-go-storage and choose what fit your needs

This is neat! I am quite a fan of all the go based file systems that are springing up. Question: what are the main compare and contrast points between juice and seaweed fs?

Here is a compendium for those interested:

https://github.com/gostor/awesome-go-storage

There's another benchmark somewhere showing S3 can max out a 100Gbps instance.

https://github.com/dvassallo/s3-benchmark

Another potential issue is ListBucket rate limiting. If you have lots of small objects, you'll spend most of the time waiting to discover the names than transferring data

TTFB in S3 is 20-30ms around the 50th percentile. it can go much higher at p99 [1]. In any case, rotational latency for HDD drives is an order of magnitude lower (typically 2-5ms for a seek operation).

S3 is great for higher throughput workloads where TTFB is amortized across larger downloads (this is why it's very common to use S3 as a "data lake" where larger columnar files are stored, usually at the order of hundreds of MiB).

I think it's an interesting project but perhaps explaining the use cases where this solution is beneficial would go a long way here.

[1] https://github.com/dvassallo/s3-benchmark