Seaweed File System Alternatives

Reviews and mentions

• SeaweedFS uses skip list on Redis to store millions of files in one directory.
  1 project | reddit.com/r/filesystems | 11 Oct 2021
• Will Cloudflare R2 Win Customers from Amazon S3?
  2 projects | news.ycombinator.com | 6 Oct 2021

  There are simply too many object store vendors, with different storage pricing, egress pricing, network speed, and reliabilities.

  What about using SeaweedFS gateway to remote object store? https://github.com/chrislusf/seaweedfs/wiki/Gateway-to-Remot...

  Basically you can think it is a cache. You operate with normal S3 APIs, and the data will be uploaded to remote storage asynchronously (most vendors have free uploading).

  It's not a magic pill for all users, but it should be very fast, and cut down the API costs, egress costs.

• Cloudflare R2 Storage: Rapid and Reliable Object Storage, Minus the Egress Fees
  2 projects | news.ycombinator.com | 28 Sep 2021

  Most cloud object storage can be a good off-site backup. But accessing your data fast and cheap is not easy.

  That is why SeaweedFS added a gateway to remote object store. https://github.com/chrislusf/seaweedfs/wiki/Gateway-to-Remot... , which asynchronously write local changes to the cloud. If there are enough local capacity, there should be no egress cost.

  Hopefully, this can change the common pattern, to really treat the cloud object store as a backup.

• Clustered filesystem between servers
  1 project | reddit.com/r/selfhosted | 27 Sep 2021

  I’m going to use https://github.com/chrislusf/seaweedfs on my NAS.

• Software projects need heroes? lessons learned from 1100 projects
  1 project | news.ycombinator.com | 21 Sep 2021

  As a main contributor to an open source project (https://github.com/chrislusf/seaweedfs), I can confirm that this finding is so true.

  However, seems this research did not look into Apache projects, which basically maintain a different culture to encourage more contributors, so much as to as the main contributors to refrain from jumping to solve an issue until another person steps in first.

• Mount and cache S3 to local SeaweedFS cluster
  1 project | reddit.com/r/DataHoarder | 30 Aug 2021
• Ask HN: Companies of one, what is your tech stack (2021)?
  27 projects | news.ycombinator.com | 25 Aug 2021

  Hey, I was trying out Seaweed yesterday. I ran into an issue with HTTPS, it seems to be not very well supported at the moment.

  I managed to get it running mostly secure, but replication didn't work because the volumes were calling eachother with HTTP instead of HTTPS. The HTTP request is hardcoded here: https://github.com/chrislusf/seaweedfs/blob/43fd11278ef81185... and probably in a lot of other places.

  I even tried setting the address of the volume to https in the startup script, but then it makes a request to http://https://volume1.example.com and it still fails.

  I also noticed that the master API is still available over HTTP even when HTTPS is enabled. I can make an issue for these things if you want.

  These issues are currently the only showstopper for me. I need to have every endpoint on TLS with peer verification enabled. If you can get it fixed I will gladly continue testing seaweedfs and support you on Patreon :-)

  27 projects | news.ycombinator.com | 25 Aug 2021

  > That would make it impossible to link directly to a single storage server as a single file would also be distributed.

  Check out https://github.com/chrislusf/seaweedfs/ implementation of reed solomon. Small files can still be served from 1 server.

  It's also efficient for small files, which a image store requires.

• Anything similar to StorJ? For self hosted purposes?
  4 projects | reddit.com/r/selfhosted | 17 Aug 2021
• Our own distributed open storage network? A large "Secure" open directory
  2 projects | reddit.com/r/DataHoarder | 17 Aug 2021

  Another notable Mention https://github.com/chrislusf/seaweedfs

• MinIO: A Bare Metal Drop-In for AWS S3
  9 projects | news.ycombinator.com | 10 Aug 2021

  I don't know when this was written, but MinIO does not have a great story (or really any story) around horizontal scalability. Yes, you can set it up in "distributed mode", but that is completely fixed at setup time and requires a certain number of nodes right from the beginning.

  For anyone who wants HA and horizontal elastic scalability, checkout SeaweedFS instead, it is based on the Facebook "Haystack" paper: https://github.com/chrislusf/seaweedfs

  9 projects | news.ycombinator.com | 10 Aug 2021

  I haven't used seaweedfs yet, but it looks better (and small file/object performance should be miles better). W.r.t. to fsync/durability with the seaweedfs API to a volume server you have to turn fsync on via paramter and it is disabled by default. With S3 it is probably also off by default and you can turn it on per bucket: https://github.com/chrislusf/seaweedfs/wiki/Path-Specific-Co... .

  Both should default to fsync on, with the option to turn it off. So not a great choice of defaults. Again, it probably looks good in benchmarks when people naively compare S3 stores. But it just shouldn't eat your data per default.

  9 projects | news.ycombinator.com | 10 Aug 2021

  Well, the author has to be aware of it: https://github.com/chrislusf/seaweedfs/issues/2216

• MicroK8s – Low-ops, minimal Kubernetes, for cloud, clusters, Edge and IoT
  15 projects | news.ycombinator.com | 21 Jul 2021
• IT Pro Tuesday #157 - MS Mac Downloads, Cabling Tip, CSP Cheatsheet & More
  1 project | reddit.com/r/ITProTuesday | 6 Jul 2021

  SeaweedFS is a fast, distributed storage system for blobs, objects, files and data that stores/serves billions of files. Can transparently integrate with the cloud with both fast local access and elastic cloud storage capacity. Blob store has O(1) disk seek, local and cloud tiering. Filer supports cross-cluster active-active replication, Kubernetes, POSIX, S3 API, encryption, Erasure Coding for warm storage, FUSE mount, Hadoop and WebDAV. chrislusf finds "it is much faster than the 'high performance' Minio."