Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality. Learn more →
Top 8 Python model-monitoring Projects
-
deepchecks
Deepchecks: Tests for Continuous Validation of ML Models & Data. Deepchecks is a holistic open-source solution for all of your AI & ML validation needs, enabling to thoroughly test your data and models from research to production.
Project mention: Detect, Defend, Prevail: Payments Fraud Detection using ML & Deepchecks | dev.to | 2024-01-13Also if you have any confusion related to it. You can directly go to their discussion section in github :
-
In order to try to solve this issue, NannyML was created. NannyML is an open-source Python library designed in order to make it easy to monitor drift in the distributions of our model input variables and estimate our model performance (even without labels!) thanks to the Confidence-Based Performance Estimation algorithm they developed. But first of all, why do models need to be monitored and why their performance might vary over time?
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
Project mention: Show HN: You don't need to adopt new tools for LLM observability | news.ycombinator.com | 2024-02-14
So why should it be different when the app you're building happened to be using LLMs?
So today we're open-sourcing OpenLLMetry-JS. It's an open protocol and SDK, based on OpenTelemetry, that provides traces and metrics for LLM JS/TS applications and can be connected to any of the 15+ tools that already support OpenTelemetry. Here's the repo: https://github.com/traceloop/openllmetry-js
A few months ago we launched the python flavor here (https://news.ycombinator.com/item?id=37843907) and we've now built a compatible one for Node.js.
Would love to hear your thoughts and opinions!
Check it out -
Docs: https://www.traceloop.com/docs/openllmetry/getting-started-t...
Github:
-
whitebox
[Not Actively Maintained] Whitebox is an open source E2E ML monitoring platform with edge capabilities that plays nicely with kubernetes (by squaredev-io)
-
inferencedb
🚀 Stream inferences of real-time ML models in production to any data lake (Experimental)
-
Project mention: Call for open source devs to build the future of healthcare! | /r/github | 2023-06-30
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
Python model-monitoring related posts
- evidently: Evaluate and monitor ML models from validation to production
- [P] InferenceDB - Makes it easy to store predictions of real-time ML models in S3
- InferenceDB: Stream inferences of real-time ML models to S3 using Kafka
- InferenceDB: Stream inferences of real-time ML models in production to any data lake 🚀
- InferenceDB – Stream ML inferences to S3 or any data lake with CRDs
- InferenceDB – Stream predictions from KServe to S3 or any data lakes
- InferenceDB – Stream predictions of real-time ML models to data lakes
-
A note from our sponsor - InfluxDB
www.influxdata.com | 30 Mar 2024
Index
What are some of the best open-source model-monitoring projects in Python? This list will help you:
Project | Stars | |
---|---|---|
1 | deepchecks | 3,295 |
2 | nannyml | 1,735 |
3 | openllmetry | 1,089 |
4 | whitebox | 180 |
5 | inferencedb | 77 |
6 | cyclops | 65 |
7 | bitbeast | 30 |
8 | reco-model-monitoring | 2 |