Top 9 data-version-control Open-Source Projects

dolt

93 16,971 10.0 Go

Dolt – Git for Data

Project mention: A MySQL compatible database engine written in pure Go | news.ycombinator.com | 2024-04-09

Hi, this is my project :)
For us this package is most important as the query engine that powers Dolt:
https://github.com/dolthub/dolt
We aren't the original authors but have contributed the vast majority of its code at this point. Here's the origin story if you're interested:
https://www.dolthub.com/blog/2020-05-04-adopting-go-mysql-se...

dvc

109 13,116 9.7 Python

🦉 ML Experiments and Data Management with Git

Project mention: My Favorite DevTools to Build AI/ML Applications! | dev.to | 2024-04-23

Collaboration and version control are crucial in AI/ML development projects due to the iterative nature of model development and the need for reproducibility. GitHub is the leading platform for source code management, allowing teams to collaborate on code, track issues, and manage project milestones. DVC (Data Version Control) complements Git by handling large data files, data sets, and machine learning models that Git can't manage effectively, enabling version control for the data and model files used in AI projects.

InfluxDB

www.influxdata.com sponsored

Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
deeplake

13 7,708 9.8 Python

Database for AI. Store Vectors, Images, Texts, Videos, etc. Use with LLMs/LangChain. Store, query, version, & visualize any AI data. Stream data in real-time to PyTorch/TensorFlow. https://activeloop.ai

Project mention: FLaNK AI Weekly 25 March 2025 | dev.to | 2024-03-25

lakeFS

48 4,066 9.8 Go

lakeFS - Data version control for your data lake | Git for data

Project mention: A Step-by-Step Guide to Implementing Data Version Control | dev.to | 2023-09-04

# Download the LakeFS binary wget https://github.com/treeverse/lakeFS/releases/latest/download/lakefs # Make the binary executable chmod +x lakefs # Initialize LakeFS with S3 as the storage backend ./lakefs init --backend s3 --s3-gateway-endpoint --s3-region --s3-force-path-style --s3-access-key --s3-secret-key

quilt

2 1,313 9.5 Jupyter Notebook

Quilt is a data mesh for connecting people with actionable data
sgr

22 326 5.4 Python

sgr (command line client for Splitgraph) and the splitgraph Python library

Project mention: Show HN: Loofi – Our AI-Powered SQL Query Builder | news.ycombinator.com | 2023-05-21

awesome-data-temporality

17 96 10.0

A curated list to help you manage temporal data across many modalities 🚀.

Project mention: FLaNK Stack Weekly for 14 Aug 2023 | dev.to | 2023-08-14

WorkOS

workos.com sponsored

The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
gittargets

1 81 4.7 R

Data version control for reproducible analysis pipelines in R with {targets}.

Project mention: Feedback needed: building Git for data that commits only diffs (for storage efficiency on large repositories), even without full checkouts of the datasets | /r/datascience | 2023-05-27

This is was attempted in an R package called gittargets

ZnTrack

2 41 8.0 Python

Create, visualize, run & benchmark DVC pipelines in Python & Jupyter notebooks.

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).