Replication

Open-source projects categorized as Replication
Edit details
Language: + Go + C + C++ + Shell + Python + Perl + Jinja + Rust
Topics: Database Postgresql Distributed Systems Cdc MySQL
The modern identity platform for B2B SaaS
The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
workos.com
Sponsored

Top 23 Replication Open-Source Projects

  • seaweedfs

    SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, cross-DC active-active replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding.

    • Project mention: DwarFS – The Deduplicating Warp-Speed Advanced Read-Only File System | news.ycombinator.com | 2024-04-11

    Whoops: WebDAV:

    https://news.ycombinator.com/item?id=39417503

    SeaweedFS supports WebDAV. https://github.com/seaweedfs/seaweedfs/wiki/WebDAV

    I'm not able to find if both/restic supports mounting backups as WebDAV, but in theory there's nothing stopping you.

    It's 100% user space (expose a rest service) and supported by a bunch of file-browsers with a bit of a network aware component to it as well.

  • Ceph

    Ceph is a distributed object, block, and file storage platform

    • Project mention: First time user sturggles | /r/ceph | 2023-06-24

    curl --silent --remote-name --location https://github.com/ceph/ceph/raw/octopus/src/cephadm/cephadmchmod a+x cephadm./cephadm bootstrap --mon-ip 192.168.1.41

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo

  • litestream

    Streaming replication for SQLite.

    • Project mention: Ask HN: SQLite in Production? | news.ycombinator.com | 2024-04-07

    I have not, but I keep meaning to collate everything I've learned into a set of useful defaults just to remind myself what settings I should be enabling and why.

    Regarding Litestream, I learned pretty much all I know from their documentation: https://litestream.io/

  • GameNetworkingResources

    A Curated List of Multiplayer Game Network Programming Resources

    • Project mention: A Curated List of Game Network Programming Resources | news.ycombinator.com | 2023-05-22

  • orchestrator

    MySQL replication topology management and HA

  • GlusterFS

    Gluster Filesystem : Build your distributed storage in minutes

    • Project mention: Tell HN: ZFS silent data corruption bugfix – my research results | news.ycombinator.com | 2023-12-06

    https://github.com/gluster/glusterfs/issues/894

    And apparently apart from modern coreutils using that, it is mostly gentoo users hitting the bugs in lseek.

  • wal-e

    Continuous Archiving for Postgres

    • Project mention: Run PostgreSQL. The Kubernetes Way | news.ycombinator.com | 2023-09-22

    See the GitHub: https://github.com/wal-e/wal-e

    Unmaintained would’ve made more sense to say, but the maintainer choose the words “obsolete” so I took those. :)

    Seems to be obsolete due to a lack of interest and contributions.

  • WorkOS

    The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

    WorkOS logo

  • cloudnative-pg

    CloudNativePG is a comprehensive platform designed to seamlessly manage PostgreSQL databases within Kubernetes environments, covering the entire operational lifecycle from initial deployment to ongoing maintenance

    • Project mention: Homelab: Running Postgres on Kubernetes | news.ycombinator.com | 2024-01-02

    My holiday project was doing another pass at my Homelab Kubernetes cluster, part of which involved switching to a proper operator to manage Postgres. Coincidentally, I setup cloudnative-pg (https://github.com/cloudnative-pg/cloudnative-pg) yesterday.

  • sanoid

    These are policy-driven snapshot management and replication tools which use OpenZFS for underlying next-gen storage. (Btrfs support plans are shelved unless and until btrfs becomes reliable.)

    • Project mention: ZFS took full backup (send/receive) without snapshots but now what to prevent another full transfer? | /r/zfs | 2023-12-10

  • mydumper

    Official MyDumper Project

    • Project mention: Restore or Migrate your MySQL Database to Railway | dev.to | 2023-11-03

    You can create a MySQL backup for Railway.app using the MyDumper tool or by using a backup service like SimpleBackups. Both methods involve generating a dump file that can be used for restoration.

  • gravity-sync

    💫 The easy way to synchronize the DNS configuration of two Pi-hole 5.x instances.

    • Project mention: Setting up a secondary Pi-Hole on my home network | news.ycombinator.com | 2024-02-06

  • marmot

    A distributed SQLite replicator built on top of NATS

    • Project mention: Distributed SQLite: Paradigm shift or hype? | news.ycombinator.com | 2024-04-09

    If you're willing to accept eventual consistency (a big ask, but acceptable in some scenarios) then there are options like marmot [1] that replicate cdc over nats.

    [1]: https://github.com/maxpert/marmot

  • postgresql_cluster

    PostgreSQL High-Availability Cluster (based on "Patroni" and DCS "etcd" or "consul"). Automating with Ansible.

    • Project mention: Automated Deployment of PostgreSQL Ha Clusters Anywhere | news.ycombinator.com | 2023-06-21

    Creating a robust and reliable database cluster is often fraught with challenges.A manual deployment process can be fraught with risks, particularly for those who lack extensive experience. Even minor errors in configuration can drastically affect your database's performance and stability.

    Recognizing this intricacy, many businesses choose to use cloud-based managed databases. These services offer convenience but not without limitations. These drawbacks can include being tied to a specific cloud provider and limited direct access to the database server for adjustments. These constraints can hinder your ability to tune your system and perform thorough performance analyses.

    In response to these challenges, we've developed a project to make this process easier and more efficient. This project uses automation to streamline the deployment of High-Availability PostgreSQL Clusters. By doing so, it helps mitigate the risks involved with manual configuration and speeds up the deployment process.

    Our solution caters to various deployment environments. It allows you to establish your cluster either in your own data center or in the cloud, providing a robust database solution similar to managed databases, but with you retaining full control.

    Explore the project here: https://github.com/vitabaks/postgresql_cluster

  • wal2json

    JSON output plugin for changeset extraction

    • Project mention: On the performance of REPLICA IDENTITY FULL in Postgres | dev.to | 2023-06-07

    To illustrate the above, let’s look at an example, using wal2json for convenience. An update event looks something like this:

  • librg

    🚀 Making multi-player gamedev simpler since 2017

  • mycelite

    Mycelite is a SQLite extension that allows you to synchronize changes from one instance of SQLite to another.

    • Project mention: LiteFS Cloud: Distributed SQLite with Managed Backups | news.ycombinator.com | 2023-07-05

  • skyplane

    🔥 Blazing fast bulk data transfers between any cloud 🔥

    • Project mention: Moving cloud storage | /r/seedboxes | 2023-05-19

  • lizardfs

    LizardFS is an Open Source Distributed File System licensed under GPLv3.

  • NuRaft

    C++ implementation of Raft core logic as a replication library

    • Project mention: ClickHouse Keeper: A ZooKeeper alternative written in C++ | news.ycombinator.com | 2023-09-27

    That's true - C++ libraries are typically bug-ridden and require exhaustive efforts to clean up.

    But the latest bugs found by ClickHouse continuous integration system in the related library were fixed about a year ago:

    https://github.com/eBay/NuRaft/pull/373

  • pglogical

    Logical Replication extension for PostgreSQL 15, 14, 13, 12, 11, 10, 9.6, 9.5, 9.4 (Postgres), providing much faster replication than Slony, Bucardo or Londiste, as well as cross-version upgrades.

  • osync

    A robust two way (bidirectional) file sync script based on rsync with fault tolerance, POSIX ACL support, time control and near realtime sync

  • zrepl

    One-stop ZFS backup & replication solution

    • Project mention: Zrepl – ZFS replication | /r/patient_hackernews | 2023-08-25

  • gitlab-mirrors

    A set of scripts adding the ability of managing remote mirrors to GitLab.

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    SaaSHub logo
NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).

Replication related posts

Index

What are some of the best open-source Replication projects? This list will help you:

Project Stars
1 seaweedfs 21,013
2 Ceph 13,197
3 litestream 9,964
4 GameNetworkingResources 6,758
5 orchestrator 5,473
6 GlusterFS 4,478
7 wal-e 3,423
8 cloudnative-pg 3,342
9 sanoid 2,872
10 mydumper 2,347
11 gravity-sync 2,133
12 marmot 1,621
13 postgresql_cluster 1,258
14 wal2json 1,236
15 librg 1,216
16 mycelite 1,038
17 skyplane 966
18 lizardfs 945
19 NuRaft 939
20 pglogical 935
21 osync 909
22 zrepl 895
23 gitlab-mirrors 807

Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com