Top 13 HDFS Open-Source Projects

• seaweedfs

  34 21,123 9.9 Go

  SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, cross-DC active-active replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding.

  

Project mention: DwarFS – The Deduplicating Warp-Speed Advanced Read-Only File System | news.ycombinator.com | 2024-04-11

Whoops: WebDAV:

https://news.ycombinator.com/item?id=39417503

SeaweedFS supports WebDAV. https://github.com/seaweedfs/seaweedfs/wiki/WebDAV

I'm not able to find if both/restic supports mounting backups as WebDAV, but in theory there's nothing stopping you.

It's 100% user space (expose a rest service) and supported by a bunch of file-browsers with a bit of a network aware component to it as well.

• Ceph

  34 13,259 10.0 C++

  Ceph is a distributed object, block, and file storage platform

  

Project mention: First time user sturggles | /r/ceph | 2023-06-24

curl --silent --remote-name --location https://github.com/ceph/ceph/raw/octopus/src/cephadm/cephadmchmod a+x cephadm./cephadm bootstrap --mon-ip 192.168.1.41

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• juicefs

  42 9,824 9.8 Go

  JuiceFS is a distributed POSIX file system built on top of Redis and S3.

  

Project mention: South Korea's No.1 Search Engine Chose JuiceFS over Alluxio for AI Storage | dev.to | 2024-01-18

Support for Kerberos keytab files

• smart_open

  6 3,093 8.3 Python

  Utils for streaming large files (S3, HDFS, gzip, bz2...)

  

• TileDB

  14 1,764 9.7 C++

  The Universal Storage Engine

  

Project mention: Ask HN: Who is hiring? (May 2024) | news.ycombinator.com | 2024-05-01

TileDB, Inc. | Full-Time | REMOTE | USA, Greece/EU | [https://tiledb.com](https://tiledb.com/)

TileDB has recently announced a $34 million Series B fund-raise and is actively hiring for engineers across a range of roles (SRE, backend/distributed systems, database internals, and more). You will have the opportunity to work on innovative technology that creates impact for challenging problems in genomics, geospatial, machine learning, distributed systems, and many other areas.

TileDB Cloud is the modern database, allowing developers and scientists to capture, analyze, and share any data with any tool. We build on a broad foundation of open source, maintaining the TileDB storage engine, libraries for genomics (single-cell and population), geospatial (raster, point clouds, and more), a TileDB visualization engine extending Babylon.js, and much more ([github.com/TileDB-Inc/TileDB](http://github.com/TileDB-Inc/TileDB))

With TileDB, all data — tables, genomics, images, videos, location, time-series — is captured as multi-dimensional arrays. To supercharge this data, TileDB Cloud implements a serverless infrastructure delivering query execution, access control, data and code sharing, and distributed computing at global scale — eliminating cluster management, minimizing TCO, and promoting scientific collaboration and reproducibility.

Website: [https://tiledb.com](https://tiledb.com/) | GitHub: https://github.com/TileDB-Inc/TileDB | Blog: https://tiledb.com/blog

We are actively hiring for several roles including:

- Site Reliability Engineer (k8s, Terraform, automation, Prometheus, CloudWatch, GitOps; Golang, Python)

• hdfs

  3 1,347 3.6 Go

  A native go client for HDFS

• kafka-connect-ui

  1 496 0.0 JavaScript

  Web tool for Kafka Connect |

  

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

• rumble

  1 207 7.8 Java

  ⛈️ RumbleDB 1.21.0 "Hawthorn blossom" 🌳 for Apache Spark | Run queries on your large-scale, messy JSON-like data (JSON, text, CSV, Parquet, ROOT, AVRO, SVM...) | No install required (just a jar to download) | Declarative Machine Learning and more (by RumbleDB)

  

• TileDB-Py

  1 179 9.0 Python

  Python interface to the TileDB storage engine

  

• elbencho

  2 147 7.5 C++

  A distributed storage benchmark for file systems, object stores & block devices with support for GPUs

• apache-spark-docker

  1 40 0.0 VBA

  Dockerizing an Apache Spark Standalone Cluster

• hdfs-rs

  0 33 0.0 Rust

  libhdfs binding and wrapper APIs for Rust

• NiFItoKafkaConnect

  0 3 0.0 Shell

  NiFi -> Kafka Connect -> HDFS

HDFS related posts

• DuckDB + dbt for a serverless event correlation pipeline?

  1 project | /r/dataengineering | 7 Dec 2023

• SeaweedFS

  1 project | /r/devopspro | 9 Jun 2023

• Experience running rook-ceph in production/large clusters

  1 project | /r/kubernetes | 21 Apr 2023

• First Homelab as a 19yr old Software Developer

  2 projects | /r/homelab | 15 Apr 2023

• SeaweedFS is a fast distributed storage system for blobs, objects and files

  1 project | news.ycombinator.com | 3 Apr 2023

• pandas 2.0 and the Arrow revolution (part I)

  1 project | /r/dataengineering | 23 Feb 2023

• SeaweedFS vs JuiceFS

  1 project | dev.to | 16 Feb 2023
• A note from our sponsor - SaaSHub
  www.saashub.com | 4 May 2024

  SaaSHub helps you find the best software and product alternatives Learn more →

Index

Sponsored

SaaSHub - Software Alternatives and Reviews

SaaSHub helps you find the best software and product alternatives

www.saashub.com