object-storage

Open-source projects categorized as object-storage
Edit details
Language: + Go + JavaScript + Rust + Shell + Java + TypeScript + Gherkin + Python
Topics: S3 Storage Kubernetes Distributed Systems distributed-storage
The modern identity platform for B2B SaaS
The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
workos.com
Sponsored

Top 20 object-storage Open-Source Projects

  • seaweedfs

    SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, cross-DC active-active replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding.

    • Project mention: DwarFS – The Deduplicating Warp-Speed Advanced Read-Only File System | news.ycombinator.com | 2024-04-11

    Whoops: WebDAV:

    https://news.ycombinator.com/item?id=39417503

    SeaweedFS supports WebDAV. https://github.com/seaweedfs/seaweedfs/wiki/WebDAV

    I'm not able to find if both/restic supports mounting backups as WebDAV, but in theory there's nothing stopping you.

    It's 100% user space (expose a rest service) and supported by a bunch of file-browsers with a bit of a network aware component to it as well.

  • parse-server

    Parse Server for Node.js / Express

    • Project mention: The 2024 Web Hosting Report | dev.to | 2024-02-20

    Backend as a Service (BaaS) goes back to early 2010’s with companies like Parse and Firebase. These products integrated everything a backend provides to a webapp in a single, integrated package that makes it easier to get started and enables you to offload some of the devops maintenance work to someone else.

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo

  • juicefs

    JuiceFS is a distributed POSIX file system built on top of Redis and S3.

    • Project mention: South Korea's No.1 Search Engine Chose JuiceFS over Alluxio for AI Storage | dev.to | 2024-01-18

    Support for Kerberos keytab files

  • databend

    𝗗𝗮𝘁𝗮, 𝗔𝗻𝗮𝗹𝘆𝘁𝗶𝗰𝘀 & 𝗔𝗜. Modern alternative to Snowflake. Cost-effective and simple for massive-scale analytics. https://databend.com

    • Project mention: Solutions to manage runaway Snowflake costs? | news.ycombinator.com | 2024-01-16

    Databend vs. Snowflake: https://github.com/datafuselabs/databend/issues/13059

  • cubefs

    cloud-native file store

  • lakeFS

    lakeFS - Data version control for your data lake | Git for data

    • Project mention: A Step-by-Step Guide to Implementing Data Version Control | dev.to | 2023-09-04

    # Download the LakeFS binary wget https://github.com/treeverse/lakeFS/releases/latest/download/lakefs # Make the binary executable chmod +x lakefs # Initialize LakeFS with S3 as the storage backend ./lakefs init --backend s3 --s3-gateway-endpoint --s3-region --s3-force-path-style --s3-access-key --s3-secret-key

  • storj

    Ongoing Storj v3 development. Decentralized cloud object storage that is affordable, easy to use, private, and secure.

    • Project mention: Do you donate your CPU time, storage, or bandwidth from your homelab to any altruistic purpose? | /r/homelab | 2023-10-16

    I did storj.io but was not profitable and the support was worthless. Did join NTP Pool (as I have a stratum 2 GPS NTP) but the power supply died and I haven't been able to get time to fix it.

  • WorkOS

    The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

    WorkOS logo

  • S3 Server

    Zenko CloudServer, an open-source Node.js implementation of the Amazon S3 protocol on the front-end and backend storage capabilities to multiple clouds, including Azure and Google.

    • Project mention: Show HN: OpenSign – The open source alternative to DocuSign | news.ycombinator.com | 2023-10-28

    > Theoretically they could swap with minio but last time we used it it was not a drop-in replacement yet.

    Depends on whether AGPL v3 works for you or not (or whether you decide to pay them), I guess: https://min.io/pricing

    I've actually been looking for more open alternatives, but haven't found much.

    Zenko CloudServer seemed to be somewhat promising, but doesn't seem to be managed very actively: https://github.com/scality/cloudserver/issues/4986 (their Docker images on DockerHub were last updated 10 months ago, which is what the homepage links to; blog doesn't seem active since 2019, forums don't have much going on, despite some action on GitHub still)

    There was also Garage, but that one is also AGPL v3: https://garagehq.deuxfleurs.fr/

    The closest I got was discovering that SeaweedFS has an S3 compatible mode: https://github.com/seaweedfs/seaweedfs

  • aistore

    AIStore: scalable storage for AI applications

  • Nitrite Database

    NoSQL embedded document store for Java

  • storage

    S3 compatible object storage service that stores metadata in Postgres

    • Project mention: Supabase Storage: now supports the S3 protocol | news.ycombinator.com | 2024-04-18

    [3] https://github.com/supabase/storage/issues/439

  • Zenko

    Zenko is the open source multi-cloud data controller: own and keep control of your data on any cloud.

  • garage

    (Mirror) S3-compatible object store for small self-hosted geo-distributed deployments. Main repo: https://git.deuxfleurs.fr/Deuxfleurs/garage (by deuxfleurs-org)

    • Project mention: SeaweedFS fast distributed storage system for blobs, objects, files and datalake | news.ycombinator.com | 2024-02-02

    Take a look at GarageS3, it's a niceoption for "just an S3 server" for self hosting.

    https://garagehq.deuxfleurs.fr/

    I use it for self hosting.

  • phalanx

    Phalanx is a cloud-native distributed search engine that provides endpoints through gRPC and traditional RESTful API.

  • lithops

    A multi-cloud framework for big data analytics and embarrassingly parallel jobs, that provides an universal API for building parallel applications in the cloud ☁️🚀

  • objstore

    Go module providing unified interface and efficient clients to work with various object storage providers until like GCS, S3, Azure, SWIFT, COS and more.

  • tiny-storage-client

    Tiny node client to request distributed AWS S3 or the OpenStack Swift Object Storage.

    • Project mention: S3 Client against disasters (hacks, fires, catastrophes) | dev.to | 2024-03-29

    Cloud providers are selling "multi-region", which is useful; however, setting up your own "multi-cloud" provider using tiny-storage-client will make your production application more resilient.

  • busyfs

    A zero-dependency, single `.h/.a/.so/.dylib/.dll` library to interact (read/write) with ANY object storage, e.g., local filesystem, HDFS, S3, OSS, and more.

  • iceberg-intro-workshop

    Hands-on workshop with Apache Iceberg

    • Project mention: FLaNK Stack Weekly for 13 November 2023 | dev.to | 2023-11-13

  • NiFItoKafkaConnect

    NiFi -> Kafka Connect -> HDFS

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    SaaSHub logo
NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2024-04-18.

object-storage related posts

Index

What are some of the best open-source object-storage projects? This list will help you:

Project Stars
1 seaweedfs 20,956
2 parse-server 20,600
3 juicefs 9,774
4 databend 7,157
5 cubefs 4,306
6 lakeFS 4,053
7 storj 2,864
8 S3 Server 1,627
9 aistore 1,084
10 Nitrite Database 800
11 storage 600
12 Zenko 539
13 garage 361
14 phalanx 341
15 lithops 304
16 objstore 81
17 tiny-storage-client 15
18 busyfs 11
19 iceberg-intro-workshop 10
20 NiFItoKafkaConnect 3

Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com