Top 11 blob-storage Open-Source Projects

seaweedfs

34 21,076 9.9 Go

SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, cross-DC active-active replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding.

Project mention: DwarFS – The Deduplicating Warp-Speed Advanced Read-Only File System | news.ycombinator.com | 2024-04-11

Whoops: WebDAV:
https://news.ycombinator.com/item?id=39417503
SeaweedFS supports WebDAV. https://github.com/seaweedfs/seaweedfs/wiki/WebDAV
I'm not able to find if both/restic supports mounting backups as WebDAV, but in theory there's nothing stopping you.
It's 100% user space (expose a rest service) and supported by a bunch of file-browsers with a bit of a network aware component to it as well.

roapi

24 3,080 6.9 Rust

Create full-fledged APIs for slowly moving datasets without writing a single line of code.

Project mention: Full-fledged APIs for slowly moving datasets without writing code | news.ycombinator.com | 2023-10-25

InfluxDB

www.influxdata.com featured

Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
Azurite

5 1,702 8.8 TypeScript

A lightweight server clone of Azure Storage that simulates most of the commands supported by it with minimal dependencies
components-contrib

3 528 9.2 Go

Community driven, reusable components for distributed apps

Project mention: How to tune Dapr bulk publish/subscribe for maximum throughput | dev.to | 2024-02-14

Changing this setting did not make the warning completely go away. As Alessandro Segala states in this issue : "This happens because when it's time to renew the locks, at the interval, we "snapshot" the active messages and then renew each one's lock in sequence. If you have a high number for maxActiveMessages, it takes time for the component to renew the locks for each one, and if the app has ACK'd the message in the meanwhile, then the lock renewal fails."

FluentStorage

1 196 7.6 C#

A polycloud .NET cloud storage abstraction layer. Provides Blob storage (AWS S3, GCP, FTP, SFTP, Azure Blob/File/Event Hub/Data Lake) and Messaging (AWS SQS, Azure Queue/ServiceBus). Supports .NET 5+ and .NET Standard 2.0+. Pure C#.

Project mention: Motivated by discussion in a comparison Go post | /r/dotnet | 2023-06-28

For blobs I use abstraction like this.

ReductStore

44 140 9.3 Rust

A time series database for storing and managing large amounts of blob data

Project mention: Time Series Blob Data: ReductStore vs. MongoDB | dev.to | 2024-04-08

In edge computing, managing time series blob data efficiently is critical for performance-sensitive applications. This blog post will compare ReductStore, a specialized time series database for unstructured data, and MongoDB, a widely-used NoSQL database.

storage-abstraction

1 92 8.1 TypeScript

Provides an abstraction layer for interacting with a storage; the storage can be local or in the cloud.

Project mention: Library for Abstracting Object Storage Services of GCP/AWS/Azure | /r/node | 2023-12-07

Found one similar library tweedegolf/storage-abstraction

SaaSHub

www.saashub.com featured

SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
goblob

3 66 6.6 Go

A fast enumeration tool for publicly exposed Azure Storage blobs.
Storage

5 54 8.0 C#

Storage library provides a universal interface for accessing and manipulating data in different cloud blob storage providers (by managedcode)

Project mention: Live coding session | /r/dotnet | 2023-10-13

I'm gearing up for a live coding session on my open-source C# library, https://github.com/managedcode/Storage. During the session, we'll be adding new features and refactoring some existing code. It's a great opportunity to collaborate, learn, and improve the library together.

waihona

5 49 0.0 Rust

Rust crate for performing cloud storage CRUD actions across major cloud providers e.g aws
Upload-AutopilotInfo-To-Blob

1 3 4.6 PowerShell

Upload Autopilot Hardware hash to Azure Blob Storage

Project mention: Mass register devices with Windows Autopilot | /r/Intune | 2023-05-10

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).

blob-storage related posts

Time Series Blob Data: ReductStore vs. MongoDB

4 projects | dev.to | 8 Apr 2024
ReductStore v1.9.0 Released

2 projects | dev.to | 14 Mar 2024
How to Use Reductstore as a Data Sink for Kafka

3 projects | dev.to | 4 Feb 2024
Easy Guide to Integrating Kafka: Practical Solutions for Managing Blob Data

5 projects | dev.to | 22 Jan 2024
How To Implement Data Streaming In PyTorch From A Remote Database

2 projects | dev.to | 15 Jan 2024
From Lab to Live: Implementing Open-Source AI Models for Real-Time Unsupervised Anomaly Detection in Images

4 projects | dev.to | 15 Oct 2023
ReductStore 1.6.0 has been released with new license and client SDK for Rust

3 projects | dev.to | 15 Aug 2023
A note from our sponsor - SaaSHub
www.saashub.com | 3 May 2024

SaaSHub helps you find the best software and product alternatives Learn more →

Index

What are some of the best open-source blob-storage projects? This list will help you:

	Project	Stars
1	seaweedfs	21,076
2	roapi	3,080
3	Azurite	1,702
4	components-contrib	528
5	FluentStorage	196
6	ReductStore	140
7	storage-abstraction	92
8	goblob	66
9	Storage	54
10	waihona	49
11	Upload-AutopilotInfo-To-Blob	3