MLflow
alibi-detect
MLflow | alibi-detect | |
---|---|---|
56 | 9 | |
17,284 | 2,085 | |
1.3% | 1.6% | |
9.9 | 7.6 | |
6 days ago | 11 days ago | |
Python | Python | |
Apache License 2.0 | GNU General Public License v3.0 or later |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
MLflow
-
Observations on MLOps–A Fragmented Mosaic of Mismatched Expectations
How can this be? The current state of practice in AI/ML work requires adaptivity, which is uncommon in classical computational fields. There are myriad tools that capture the work across the many instances of the AI/ML lifecycle. The idea that any one tool could sufficiently capture the dynamic work is unrealistic. Take, for example, an experiment tracking tool like W&B or MLFlow; some form of experiment tracking is necessary in typical model training lifecycles. Such a tool requires some notion of a dataset. However, a tool focusing on experiment tracking is orthogonal to the needs of analyzing model performance at the data sample level, which is critical to understanding the failure modes of models. The way one does this depends on the type of data and the AI/ML task at hand. In other words, MLOps is inherently an intricate mosaic, as the capabilities and best practices of AI/ML work evolve.
-
My Favorite DevTools to Build AI/ML Applications!
MLflow is an open-source platform for managing the end-to-end machine learning lifecycle. It includes features for experiment tracking, model versioning, and deployment, enabling developers to track and compare experiments, package models into reproducible runs, and manage model deployment across multiple environments.
-
Exploring Open-Source Alternatives to Landing AI for Robust MLOps
Platforms such as MLflow monitor the development stages of machine learning models. In parallel, Data Version Control (DVC) brings version control system-like functions to the realm of data sets and models.
-
cascade alternatives - clearml and MLflow
3 projects | 1 Nov 2023
-
EL5: Difference between OpenLLM, LangChain, MLFlow
MLFlow - http://mlflow.org
- Explain me how websites like Dall-E, chatgpt, thispersondoesntexit process the user data so quickly
- [D] What licensed software do you use for machine learning experimentation tracking?
-
Exploring MLOps Tools and Frameworks: Enhancing Machine Learning Operations
MLflow:
-
Options for configuration of python libraries - Stack Overflow
In search for a tool that needs comparable configuration I looked into mlflow and found this. https://github.com/mlflow/mlflow/blob/master/mlflow/environment_variables.py There they define a class _EnvironmentVariable and create many objects out of it, for any variable they need. The get method of this class is in principle a decorated os.getenv. Maybe that is something I can take as orientation.
-
[D] Is there a tool to keep track of my ML experiments?
I have been using DVC and MLflow since then DVC had only data tracking and MLflow only model tracking. I can say both are awesome now and maybe the only factor I would like to mention is that IMO, MLflow is a bit harder to learn while DVC is just a git practically.
alibi-detect
-
Exploring Open-Source Alternatives to Landing AI for Robust MLOps
Numerous tools exist for detecting anomalies in time series data, but Alibi Detect stood out to me, particularly for its capabilities and its compatibility with both TensorFlow and PyTorch backends.
- Looking for recommendations to monitor / detect data drifts over time
-
[D] Distributions to represent an Image Dataset
That is, to see whether a test image belongs in the distribution of the training images and to provide a routine for special cases. After a bit of reading Ive found that this is related to the field of drift detection in which I tried out alibi-detect . Whereby the training images are trained by an autoencoder and any subsequent drift will be flagged by the AE.
-
[D] Which statistical test would you use to detect drift in a dataset of images?
Wasserstein distance is not very suitable for drift detection on most problems given that the sample complexity (and estimation error) scales with O(n^(-1/d)) with n the number of instances (100k-10m in your case) and d the feature dimension (192 in your case). More interesting will be to use for instance a detector based on the maximum mean discrepancy (MMD) with estimation error of O(n^(-1/2)). Notice the absence of the feature dimension here. You can find scalable implementations in Alibi Detect (disclosure: I am a contributor): MMD docs, image example. We just added the KeOps backend for the MMD detector to scale and speed up the drift detector further, so if you install from master, you can leverage this backend and easily scale the detector to 1mn instances on e.g. 1 RTX2080Ti GPU. Check this example for more info.
-
Ask HN: Who is hiring? (January 2022)
Seldon | Multiple positions | London/Cambridge UK | Onsite/Remote | Full time | seldon.io
At Seldon we are building industry leading solutions for deploying, monitoring, and explaining machine learning models. We are an open-core company with several successful open source projects like:
* https://github.com/SeldonIO/seldon-core
* https://github.com/SeldonIO/mlserver
* https://github.com/SeldonIO/alibi
* https://github.com/SeldonIO/alibi-detect
* https://github.com/SeldonIO/tempo
We are hiring for a range of positions, including software engineers(go, k8s), ml engineers (python, go), frontend engineers (js), UX designer, and product managers. All open positions can be found at https://www.seldon.io/careers/
- What Machine Learning model monitoring tools can you recommend?
- Ask HN: Who is hiring? (December 2021)
-
[D] How do you deal with covariate shift and concept drift in production?
I work in this area and also contribute to outlier/drift detection library https://github.com/SeldonIO/alibi-detect. To tackle this type of problem, I would strongly encourage following a more principled, fundamentally (statistically) sound approach. So for instance measuring metrics such as the KL-divergence (or many other f-divergences) will not be that informative since it has a lot of undesirable properties for the problem at hand (in order to be informative requires already overlapping distributions P and Q, it is asymmetric, not a real distance metric, will not scale well with data dimensionality etc). So you should probably look at Integral Probability Metrics (IPMs) such as the Maximum Mean Discrepancy (MMD) instead which have much nicer behaviour to monitor drift. I highly recommend the Interpretable Comparison of Distributions and Models NeurIPS workshop talks for more in-depth background.
-
[D] Is this a reasonable assumption in machine learning?
All of the above functionality and more can be easily used under a simple API in https://github.com/SeldonIO/alibi-detect.
What are some alternatives?
clearml - ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution
pytorch-widedeep - A flexible package for multimodal-deep-learning to combine tabular data with text and images using Wide and Deep models in Pytorch
Sacred - Sacred is a tool to help you configure, organize, log and reproduce experiments developed at IDSIA.
cleanlab - The standard data-centric AI package for data quality and machine learning with messy, real-world data and labels.
zenml - ZenML 🙏: Build portable, production-ready MLOps pipelines. https://zenml.io.
pyod - A Comprehensive and Scalable Python Library for Outlier Detection (Anomaly Detection)
guildai - Experiment tracking, ML developer tools
seldon-core - An MLOps framework to package, deploy, monitor and manage thousands of production machine learning models
dvc - 🦉 ML Experiments and Data Management with Git
river - 🌊 Online machine learning in Python
tensorflow - An Open Source Machine Learning Framework for Everyone
Anomaly_Detection_Tuto - Anomaly detection tutorial on univariate time series with an auto-encoder