Top 7 Go Deduplication Projects

restic

357 23,706 9.7 Go

Fast, secure, efficient backup program

Project mention: Building a Managed Service Provider Business With Open Source | dev.to | 2024-04-04

Restic - GitHub

kopia

224 6,270 9.6 Go

Cross-platform backup tool for Windows, macOS & Linux with fast, incremental backups, client-side end-to-end encryption, compression and data deduplication. CLI and GUI included.

Project mention: DwarFS – The Deduplicating Warp-Speed Advanced Read-Only File System | news.ycombinator.com | 2024-04-11

I think Kopia would be great for your use case
https://kopia.io/
It has a great system to snapshot files but only store data if it's changed. I use it in an environment where I can't use something like zfs to snapshot data because I don't have the ability to make decisions about what filesystem we're using. It's been amazing, love it so much!

WorkOS

workos.com sponsored

The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
alertmanager

13 6,270 9.2 Go

Prometheus Alertmanager
autorestic

12 1,085 7.3 Go

Config driven, easy backup cli for restic.

Project mention: Duplicity | news.ycombinator.com | 2024-01-24

I really like restic, and am personally happy to use it via the command line. It's very fast and efficient! However, I do wish there was better tooling / wrappers around it. For example, Pika Backup is a popular UI for Borg of which no equivalent exists for Restic. I'd love to be able to set something simple up on my partner's Macbook.
For my own purposes, I've been using a script I found on Github[0] for a while, but it only really supports Backblaze B2 AFAIK.[1]
I've been meaning to try autorestic[2] and resticprofile[3] as they are potentially more flexible than the script I'm currently using, and prestic[4] looks intriguing for my partner's use, but seems to have very few users. And the fact that there are so many competing tools makes it difficult to land on one.
[0] https://github.com/erikw/restic-automatic-backup-scheduler
[1] https://github.com/erikw/restic-automatic-backup-scheduler/i...
[2] https://github.com/cupcakearmy/autorestic
[3] https://github.com/creativeprojects/resticprofile
[4] https://github.com/ducalex/prestic

simpleflow

3 44 2.2 Go

Generic simple workflows and concurrency patterns
stringdedup

2 15 3.7 Go

String deduplication package for Go
fsdd

3 0 1.9 Go

deduplicate data via hardlinks, analyze drive status, prepare & optimize - filesystems (app/lib/api)
InfluxDB

www.influxdata.com sponsored

Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).