fides
dvc
fides | dvc | |
---|---|---|
3 | 118 | |
359 | 13,840 | |
1.1% | 1.4% | |
9.8 | 9.4 | |
3 days ago | 5 days ago | |
Python | Python | |
Apache License 2.0 | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
fides
- Fides: The Privacy Engineering and Compliance Framework
-
What data governance tool are you folks using?
I’ve also been impressed with the approach of Fides, an open source privacy management framework that ties into ci/cd, though I haven’t used it myself yet. The thing about it that stood out was Fideslang, their language and taxonomy for representing data privacy primitives.
-
Privacy-as-Code: Preventing Facebook’s $5B violation using Fides Open-Source
Fides is built to solve for problems like this. In its current release, you can already draft a policy in YAML using fideslang and enforce that policy to ensure engineers across a team can’t accidentally or intentionally misuse data in a way that deviates from the promises a business or application makes to its users.
dvc
- Data Version Control
-
S3 as a Git remote and LFS server
I haven't heard of dvc, so I had to google it, which took me to: https://dvc.org/
But I'm still confused as to what is dvc is after a cursory glance at their homepage.
-
serverless-registry: A Docker registry backed by Workers and R2
I’m self-hosting gitea just for their private docker registry. LFS is actually slow for heavy deep learning workflow with millions of small files. I’m using DVC [1] instead.
[1]: https://dvc.org
- GitOps ML Experiments, data versioning, model registry
-
25 Open Source AI Tools to Cut Your Development Time in Half
Implementing version control for machine learning projects entails managing both code and the datasets, ML models, performance metrics, and other development-related artifacts. Its purpose is to bring the best practices from software engineering, like version control and reproducibility, to the world of data science and machine learning. DVC enables data scientists and ML engineers to track changes to data and models like Git does for code, making it able to run on top of any Git repository. It enables the management of model experiments.
-
Essential Deep Learning Checklist: Best Practices Unveiled
Tool: Consider using Data Version Control (DVC) to manage your datasets, models, and their respective versions. DVC integrates with Git, allowing you to handle large data files and model binaries without cluttering your repository. It also makes it easy to version your training datasets and models, ensuring you can always match a model back to its exact training environment.
-
10 Open Source Tools for Building MLOps Pipelines
As Git helps you with code versions and the ability to roll back to previous versions for code repositories, DVC has built-in support for tracking your data and model. This helps machine learning teams reproduce the experiments run by your fellows and facilitates collaboration. DVC is based on the principles of Git and is easy to learn since the commands are similar to those of Git. Other benefits of using DVC include:
-
A step-by-step guide to building an MLOps pipeline
The meta-data and model artifacts from experiment tracking can contain large amounts of data, such as the training model files, data files, metrics and logs, visualizations, configuration files, checkpoints, etc. In cases where the experiment tool doesn't support data storage, an alternative option is to track the training and validation data versions per experiment. They use remote data storage systems such as S3 buckets, MINIO, Google Cloud Storage, etc., or data versioning tools like data version control (DVC) or Git LFS (Large File Storage) to version and persist the data. These options facilitate collaboration but have artifact-model traceability, storage costs, and data privacy implications.
-
AI Strategy Guide: How to Scale AI Across Your Business
Level 1 of MLOps is when you've put each lifecycle stage and their intefaces in an automated pipeline. The pipeline could be a python or bash script, or it could be a directed acyclic graph run by some orchestration framework like Airflow, dagster or one of the cloud-provider offerings. AI- or data-specific platforms like MLflow, ClearML and dvc also feature pipeline capabilities.
-
My Favorite DevTools to Build AI/ML Applications!
Collaboration and version control are crucial in AI/ML development projects due to the iterative nature of model development and the need for reproducibility. GitHub is the leading platform for source code management, allowing teams to collaborate on code, track issues, and manage project milestones. DVC (Data Version Control) complements Git by handling large data files, data sets, and machine learning models that Git can't manage effectively, enabling version control for the data and model files used in AI projects.
What are some alternatives?
awesome-machine-unlearning - Awesome Machine Unlearning (A Survey of Machine Unlearning)
MLflow - Open source platform for the machine learning lifecycle
differential-privacy-library - Diffprivlib: The IBM Differential Privacy Library
lakeFS - lakeFS - Data version control for your data lake | Git for data
fiftyone - Refine high-quality datasets and visual AI models
Activeloop Hub - Data Lake for Deep Learning. Build, manage, query, version, & visualize datasets. Stream data real-time to PyTorch/TensorFlow. https://activeloop.ai [Moved to: https://github.com/activeloopai/deeplake]
fideslang - Open-source description language for privacy to declare data types and data behaviors in your tech stack in order to simplify data privacy globally. Supports GDPR, CCPA, LGPD and ISO 19944.
delta - An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
datahub - The Metadata Platform for your Data Stack
ploomber - The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️
pandas-datareader - Extract data from a wide range of Internet sources into a pandas DataFrame.
aim - Aim 💫 — An easy-to-use & supercharged open-source experiment tracker.