evidently VS alibi-detect

Hi everyone, I’m one of the creators of Evidently, an open-source (Apache 2.0) tool for production ML monitoring. We’ve just launched a free open course on ML observability that I wanted to share with the community.

Evidently itself is an open-source ML monitoring tool with 3m+ downloads so it's fairly popular https://github.com/evidentlyai/evidently. The course will show it but also other OSS tools like Mlflow and Grafana.

Disclaimer: I am one of the people working on Evidently.

Repo:https://github.com/evidentlyai/evidently/tree/main/examples/integrations/postgres_grafana_batch_monitoring

Thank you for your answer. I'm trying it today and the the other libraries mentioned + https://github.com/evidentlyai/evidently

evidently: https://github.com/evidentlyai/evidently

Evidently.

Consideration Number #5: For model observability look to Evidently.ai, Arize.ai, Arthur.ai, Fiddler.ai, Valohai.com, or whylabs.ai.

Hi HN, we are Evidently AI http://evidentlyai.com. We're building monitoring for machine learning models in production. The tool is open source and available on GitHub: https://github.com/evidentlyai/evidently. You can use it locally in a Jupyter notebook or in a Bash shell. There’s a video showing how it works in Jupyter here: https://www.youtube.com/watch?v=NPtTKYxm524.

Machine learning models can stop working as expected, often for non-obvious reasons. If this happens to a marketing personalization model, you might spam your customers by mistake. If this happens to credit scoring models, you might face legal and reputational risks. And so on. To catch issues with the model, it is not enough to just look at service metrics like latency. You have to track data quality, data drift (did the inputs change too much?), underperforming segments (does the model fail only for users in a certain region?), model metrics (accuracy, ROC AUC, mean error, etc.), etc.

Emeli and I have been friends for many years. We first met when we both worked at Yandex (the company behind CatBoost and ClickHouse). We worked on creating ML systems for large enterprises. We then co-founded a startup focused on ML for manufacturing. Overall we've worked on more than 50 real-world ML projects, from e-commerce recommendations to steel production optimization. We faced the monitoring problem on our own when we put models in production and had to create and build custom dashboards. Emeli is also an ML instructor on Coursera (co-author of the most popular ML course in Russian) and a number of offline courses. She knows first-hand how many data scientists try to repeatedly implement the same things over and over. There is no reason why everyone should have to build their own version of something like drift detection.

We spent a couple of months talking to ML teams from different industries. We learned that there are no good, standard solutions for model monitoring. Some quoted us horror stories about broken models left unnoticed which led to $100K+ in losses. Others showed us home-grown dashboards and complained they are hard to maintain. Some said they simply have a recurring task to look at the logs once per month, and often catch the issues late. It is surprising how often models are not monitored until the first failure. We spoke to many teams who said that only after the first breakdown they started to think about monitoring. Some never do, and failures go undetected.

If you want to calculate a couple of performance metrics on top of your data, it is easy to do ad hoc. But if you want to have stable visibility into different models, you need to consider edge cases, choose the right statistical tests and implement them, design visuals, define thresholds for alerts etc. That is a harder problem that combines statistics and engineering. Beyond that, monitoring often involves sharing the results with different teams: from domain experts to developers. In practice, data scientists often end up sharing screenshots of their plots and sending files here and there. Building a maintainable software system that supports these workflows is a project in itself, and machine learning teams usually do not have time or resources for it.

Since there is no standard open-source solution, we decided to build one. We want to automate as much as possible to help people focus on the modeling work that matters, not boilerplate code.

Our main tool is an open-source Python library that generates interactive reports on ML model performance. To get it, you need to provide the model logs (input features, prediction, and ground truth if available) and reference data (usually from training). Then you choose the report type and we generate a set of dashboards. We have pre-built several reports to detect things like data drift, prediction drift, visualize performance metrics, and help understand where the model makes errors. We can display these in a Jupyter notebook or HTML. We can also generate a JSON profile instead of a report. You can then integrate this output with any external tool (like Grafana) and build a workflow you want to trigger retraining or alerts.

Under the hood, we perform the needed calculations (e.g. Kolmogorov Smirnov or Chi-Squared test to detect drift) and generate multiple interactive tables and plots (using Plotly on the backend). Right now it works with tabular data only. In the future, we plan to add more data types, reports and make it easier to customize metrics. Our goal is to make it dead easy to understand all aspects of model performance and monitor them.

We differ from other approaches in a couple of ways. There are end-to-end ML platforms on the market that include monitoring features. These work for teams who are ready to trade flexibility in order to have an all-in-one tool. But most teams we spoke to have custom needs and prefer to build their own platform from open components. We want to create a tool that does one thing well and is easy to integrate with whatever stack you use. There are also some proprietary ML monitoring solutions on the market, but we believe that tools like these should be open, transparent, and available for self-hosting. That is why we are building it as open source.

We launched under Apache 2.0 license so that everyone can use the tool. For now, our focus is to get adoption for the open-source project. We don’t plan to charge individual users or small teams. We believe that the open-source project should remain open and be highly valuable. Later on, we plan to make money by providing a hosted cloud version for teams that do not want to run it themselves. We're also considering an open-core business model where we charge for features that large companies care about like single sign-on, security and audits.

If you work in tech companies, you might think that many ML infra problems are already solved. But in more traditional industries like manufacturing, retail, finance, etc., ML is just hitting adoption. Their ML needs and environment are often very different due to legacy IT systems, regulations, and types of use cases they work with. Now that many move from ML proof-of-concept projects to production, they will need the tools to help run the models reliably.

We are super excited to share this early release, and we’d love if you could give it a try: https://github.com/evidentlyai/evidently. If you run models in production - let us know how you monitor them and if anything is missing. If you need some help to test the tool - happy to chat! We want to build this open-source project together with the community, and it is very important for us to hear your thoughts and feedback.

Numerous tools exist for detecting anomalies in time series data, but Alibi Detect stood out to me, particularly for its capabilities and its compatibility with both TensorFlow and PyTorch backends.

That is, to see whether a test image belongs in the distribution of the training images and to provide a routine for special cases. After a bit of reading Ive found that this is related to the field of drift detection in which I tried out alibi-detect . Whereby the training images are trained by an autoencoder and any subsequent drift will be flagged by the AE.

Wasserstein distance is not very suitable for drift detection on most problems given that the sample complexity (and estimation error) scales with O(n^(-1/d)) with n the number of instances (100k-10m in your case) and d the feature dimension (192 in your case). More interesting will be to use for instance a detector based on the maximum mean discrepancy (MMD) with estimation error of O(n^(-1/2)). Notice the absence of the feature dimension here. You can find scalable implementations in Alibi Detect (disclosure: I am a contributor): MMD docs, image example. We just added the KeOps backend for the MMD detector to scale and speed up the drift detector further, so if you install from master, you can leverage this backend and easily scale the detector to 1mn instances on e.g. 1 RTX2080Ti GPU. Check this example for more info.

Seldon | Multiple positions | London/Cambridge UK | Onsite/Remote | Full time | seldon.io

At Seldon we are building industry leading solutions for deploying, monitoring, and explaining machine learning models. We are an open-core company with several successful open source projects like:

* https://github.com/SeldonIO/seldon-core

* https://github.com/SeldonIO/mlserver

* https://github.com/SeldonIO/alibi

* https://github.com/SeldonIO/alibi-detect

* https://github.com/SeldonIO/tempo

We are hiring for a range of positions, including software engineers(go, k8s), ml engineers (python, go), frontend engineers (js), UX designer, and product managers. All open positions can be found at https://www.seldon.io/careers/

I work in this area and also contribute to outlier/drift detection library https://github.com/SeldonIO/alibi-detect. To tackle this type of problem, I would strongly encourage following a more principled, fundamentally (statistically) sound approach. So for instance measuring metrics such as the KL-divergence (or many other f-divergences) will not be that informative since it has a lot of undesirable properties for the problem at hand (in order to be informative requires already overlapping distributions P and Q, it is asymmetric, not a real distance metric, will not scale well with data dimensionality etc). So you should probably look at Integral Probability Metrics (IPMs) such as the Maximum Mean Discrepancy (MMD) instead which have much nicer behaviour to monitor drift. I highly recommend the Interpretable Comparison of Distributions and Models NeurIPS workshop talks for more in-depth background.

All of the above functionality and more can be easily used under a simple API in https://github.com/SeldonIO/alibi-detect.