object-storage

Top 19 object-storage Open-Source Projects

  • seaweedfs

    SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, cross-DC active-active replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding.

    Project mention: Billion File Filesystem | news.ycombinator.com | 2024-02-08

    If you want/need to take out the metadata, there's some nice solutions for that https://github.com/seaweedfs/seaweedfs

  • parse-server

    Parse Server for Node.js / Express

    Project mention: The 2024 Web Hosting Report | dev.to | 2024-02-20

    Backend as a Service (BaaS) goes back to early 2010’s with companies like Parse and Firebase. These products integrated everything a backend provides to a webapp in a single, integrated package that makes it easier to get started and enables you to offload some of the devops maintenance work to someone else.

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  • juicefs

    JuiceFS is a distributed POSIX file system built on top of Redis and S3.

    Project mention: South Korea's No.1 Search Engine Chose JuiceFS over Alluxio for AI Storage | dev.to | 2024-01-18

    Support for Kerberos keytab files

  • databend

    𝗗𝗮𝘁𝗮, 𝗔𝗻𝗮𝗹𝘆𝘁𝗶𝗰𝘀 & 𝗔𝗜. Modern alternative to Snowflake. Cost-effective and simple for massive-scale analytics. https://databend.com

    Project mention: Solutions to manage runaway Snowflake costs? | news.ycombinator.com | 2024-01-16

    Databend vs. Snowflake: https://github.com/datafuselabs/databend/issues/13059

  • cubefs

    cloud-native file store

  • lakeFS

    lakeFS - Data version control for your data lake | Git for data

    Project mention: A Step-by-Step Guide to Implementing Data Version Control | dev.to | 2023-09-04

    # Download the LakeFS binary wget https://github.com/treeverse/lakeFS/releases/latest/download/lakefs # Make the binary executable chmod +x lakefs # Initialize LakeFS with S3 as the storage backend ./lakefs init --backend s3 --s3-gateway-endpoint --s3-region --s3-force-path-style --s3-access-key --s3-secret-key

  • storj

    Ongoing Storj v3 development. Decentralized cloud object storage that is affordable, easy to use, private, and secure.

    Project mention: Do you donate your CPU time, storage, or bandwidth from your homelab to any altruistic purpose? | /r/homelab | 2023-10-16

    I did storj.io but was not profitable and the support was worthless. Did join NTP Pool (as I have a stratum 2 GPS NTP) but the power supply died and I haven't been able to get time to fix it.

  • WorkOS

    The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  • S3 Server

    Zenko CloudServer, an open-source Node.js implementation of the Amazon S3 protocol on the front-end and backend storage capabilities to multiple clouds, including Azure and Google.

    Project mention: Show HN: OpenSign – The open source alternative to DocuSign | news.ycombinator.com | 2023-10-28

    > Theoretically they could swap with minio but last time we used it it was not a drop-in replacement yet.

    Depends on whether AGPL v3 works for you or not (or whether you decide to pay them), I guess: https://min.io/pricing

    I've actually been looking for more open alternatives, but haven't found much.

    Zenko CloudServer seemed to be somewhat promising, but doesn't seem to be managed very actively: https://github.com/scality/cloudserver/issues/4986 (their Docker images on DockerHub were last updated 10 months ago, which is what the homepage links to; blog doesn't seem active since 2019, forums don't have much going on, despite some action on GitHub still)

    There was also Garage, but that one is also AGPL v3: https://garagehq.deuxfleurs.fr/

    The closest I got was discovering that SeaweedFS has an S3 compatible mode: https://github.com/seaweedfs/seaweedfs

  • aistore

    AIStore: scalable storage for AI applications

  • Nitrite Database

    NoSQL embedded document store for Java

  • storage

    S3 compatible object storage service that stores metadata in Postgres (by supabase)

    Project mention: What's the simplest solution to sell digital downloads via SvelteKit? | /r/sveltejs | 2023-12-05

    For instance, consider a supabase setup where you place your photos in storage and your orders in a table. Then, have a function run automatically whenever a new row is added to the orders table. That function would be responsible for identifying what the customer bought and delivering it to them.

  • Zenko

    Zenko is the open source multi-cloud data controller: own and keep control of your data on any cloud.

    Project mention: Kubernetes Is Hard | news.ycombinator.com | 2023-03-27

    > Iteration speed and blazing fast automated tests.

    Wholeheartedly agreed!

    It is also nice to have that additional assurance of being able to self-host things (if ever necessary) and not being locked into a singular implementation. For example, that's why managed database offerings generally aren't that risky to use, given that they're built on already established projects (e.g. compatible with MySQL/MariaDB/PostgreSQL).

    > When I discovered minio, I suddenly got much more confident coding against s3.

    MinIO is pretty good, but licensing wise could become problematic if you don't work on something open source but ever want to run it in prod. Not really what this discussion is about, but AGPL is worth mentioning: https://github.com/minio/minio/blob/master/LICENSE

    That said, thankfully S3 is so common that we have alternatives even to MinIO available, like Zenko https://www.zenko.io/ and Garage https://garagehq.deuxfleurs.fr/ both of which are good for both local development as well as hosting in whatever environments necessary.

  • garage

    (Mirror) S3-compatible object store for small self-hosted geo-distributed deployments. Main repo: https://git.deuxfleurs.fr/Deuxfleurs/garage (by deuxfleurs-org)

    Project mention: SeaweedFS fast distributed storage system for blobs, objects, files and datalake | news.ycombinator.com | 2024-02-02

    Take a look at GarageS3, it's a niceoption for "just an S3 server" for self hosting.

    https://garagehq.deuxfleurs.fr/

    I use it for self hosting.

  • phalanx

    Phalanx is a cloud-native distributed search engine that provides endpoints through gRPC and traditional RESTful API.

  • lithops

    A multi-cloud framework for big data analytics and embarrassingly parallel jobs, that provides an universal API for building parallel applications in the cloud ☁️🚀

  • objstore

    Go module providing unified interface and efficient clients to work with various object storage providers until like GCS, S3, Azure, SWIFT, COS and more.

  • busyfs

    A zero-dependency, single `.h/.a/.so/.dylib/.dll` library to interact (read/write) with ANY object storage, e.g., local filesystem, HDFS, S3, OSS, and more.

    Project mention: Show HN: Libobjectfs – zero-dependency, single-artifact library for many storage | news.ycombinator.com | 2023-04-02
  • iceberg-intro-workshop

    Hands-on workshop with Apache Iceberg

    Project mention: FLaNK Stack Weekly for 13 November 2023 | dev.to | 2023-11-13
  • NiFItoKafkaConnect

    NiFi -> Kafka Connect -> HDFS

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2024-02-20.

object-storage related posts

Index

What are some of the best open-source object-storage projects? This list will help you:

Project Stars
1 seaweedfs 20,796
2 parse-server 20,580
3 juicefs 9,675
4 databend 7,078
5 cubefs 4,280
6 lakeFS 4,022
7 storj 2,854
8 S3 Server 1,617
9 aistore 1,060
10 Nitrite Database 797
11 storage 587
12 Zenko 535
13 garage 346
14 phalanx 341
15 lithops 296
16 objstore 80
17 busyfs 11
18 iceberg-intro-workshop 10
19 NiFItoKafkaConnect 3
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com