river
alibi-detect
Our great sponsors
river | alibi-detect | |
---|---|---|
17 | 9 | |
4,754 | 2,074 | |
2.3% | 1.9% | |
9.2 | 7.6 | |
4 days ago | about 1 month ago | |
Python | Python | |
BSD 3-clause "New" or "Revised" License | GNU General Public License v3.0 or later |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
river
-
🔍Underrated Open Source Projects You Should Know About 🧠
River is a Python library for online machine learning. Online machine learning can dynamically adapt to new patterns in the data, or when the data itself is generated as a function of time, e.g., stock price prediction, content personalization.
- Ask HN: What Underrated Open Source Project Deserves More Recognition?
-
Unexpected Expected Thriller: A Tale of Coding Curiosity
Today, I'm going to take you on a thrilling coding adventure inspired by a LinkedIn code snippet, where I tangled with FastAPI, River, Watchdog, and Tenacity. Ready? Buckle up!
-
Elevate Your Python Skills: Machine Learning Packages That Transformed My Journey as ML Engineer
Complimentary: river and skorch
-
What are your favorite tools or components in the Kafka ecosystem?
River - https://github.com/online-ml/river (Online machine learning, best used with Bytewax for Kafka integration)
-
Show HN: Want something better than k-means? Try BanditPAM
Hey, great work. Do you think this algorithm would be amenable to be done online? I'm the author of River (https://riverml.xyz) where we're looking for good online clustering algorithms.
-
Python's “Disappointing” Superpowers
If you don't know Rust, but know Python, you can install Python libraries written in Rust with pip. Like, pip install polars or pip install robyn. In this case you follow the two bottom links. But then you don't write your own libraries and stuff so.. I guess that's not what you want.
But, if you want to learn Rust, you probably wouldn't start out with pyo3. You first install Rust with https://rustup.rs/ and then check out the official book, and the book rust by example, that you can find here https://www.rust-lang.org/learn - and maybe write some code on the Rust playground https://play.rust-lang.org/ - then, you use pyo3 to build Python libraries in Rust, and then use maturin https://www.maturin.rs/ to build and publish them to Pypi.
But if you still prefer to begin with Rust by writing Python libraries (it's a valid strategy if you are very comfortable with working with multiple stacks), the Maturin link has a tutorial that setups a program that is half written in python, half written in Rust, https://www.maturin.rs/tutorial.html (well the pyo3 link I sent also has one too. You should refer to the documentation of both, because you will use the two together)
After learning Rust, the next step is looking for libraries that you could leverage to make Python programs ultra fast. Here https://github.com/rayon-rs/rayon is an obvious choice, see some examples from the Rust cookbook https://rust-lang-nursery.github.io/rust-cookbook/concurrenc... - when you create a parallel iterator, it will distribute the processing to many threads (by default, one per core). The rust cookbook, by the way, is a nice reference to see the most used crates (Rust libraries) in the Rust ecosystem.
Anyway there are some posts about pyo3 on the web, like this blog post https://boring-guy.sh/posts/river-rust/ (note: it uses an outdated version of pyo3, and doesn't seem to use maturin which is a newer tool). This post was written by the developers of https://github.com/online-ml/river - another Python library written in Rust
- [D] Is it possible to update random forest parameters with new data instead of retraining on all data?
-
If ChatGPT that could browse to the internet, what would you ask it to do?
Oh they definitely can be incrementally updated, there is just added complexity. Online learning has been used with more classical machine learning methods in real-time analytics for a while now. River is a library that handles that.
-
[D] Good online learning-to-rank models
We have both bandits and FTRL implemented in River (https://riverml.xyz) if that helps.
alibi-detect
-
Exploring Open-Source Alternatives to Landing AI for Robust MLOps
Numerous tools exist for detecting anomalies in time series data, but Alibi Detect stood out to me, particularly for its capabilities and its compatibility with both TensorFlow and PyTorch backends.
- Looking for recommendations to monitor / detect data drifts over time
-
[D] Distributions to represent an Image Dataset
That is, to see whether a test image belongs in the distribution of the training images and to provide a routine for special cases. After a bit of reading Ive found that this is related to the field of drift detection in which I tried out alibi-detect . Whereby the training images are trained by an autoencoder and any subsequent drift will be flagged by the AE.
-
[D] Which statistical test would you use to detect drift in a dataset of images?
Wasserstein distance is not very suitable for drift detection on most problems given that the sample complexity (and estimation error) scales with O(n^(-1/d)) with n the number of instances (100k-10m in your case) and d the feature dimension (192 in your case). More interesting will be to use for instance a detector based on the maximum mean discrepancy (MMD) with estimation error of O(n^(-1/2)). Notice the absence of the feature dimension here. You can find scalable implementations in Alibi Detect (disclosure: I am a contributor): MMD docs, image example. We just added the KeOps backend for the MMD detector to scale and speed up the drift detector further, so if you install from master, you can leverage this backend and easily scale the detector to 1mn instances on e.g. 1 RTX2080Ti GPU. Check this example for more info.
-
Ask HN: Who is hiring? (January 2022)
Seldon | Multiple positions | London/Cambridge UK | Onsite/Remote | Full time | seldon.io
At Seldon we are building industry leading solutions for deploying, monitoring, and explaining machine learning models. We are an open-core company with several successful open source projects like:
* https://github.com/SeldonIO/seldon-core
* https://github.com/SeldonIO/mlserver
* https://github.com/SeldonIO/alibi
* https://github.com/SeldonIO/alibi-detect
* https://github.com/SeldonIO/tempo
We are hiring for a range of positions, including software engineers(go, k8s), ml engineers (python, go), frontend engineers (js), UX designer, and product managers. All open positions can be found at https://www.seldon.io/careers/
- What Machine Learning model monitoring tools can you recommend?
- Ask HN: Who is hiring? (December 2021)
-
[D] How do you deal with covariate shift and concept drift in production?
I work in this area and also contribute to outlier/drift detection library https://github.com/SeldonIO/alibi-detect. To tackle this type of problem, I would strongly encourage following a more principled, fundamentally (statistically) sound approach. So for instance measuring metrics such as the KL-divergence (or many other f-divergences) will not be that informative since it has a lot of undesirable properties for the problem at hand (in order to be informative requires already overlapping distributions P and Q, it is asymmetric, not a real distance metric, will not scale well with data dimensionality etc). So you should probably look at Integral Probability Metrics (IPMs) such as the Maximum Mean Discrepancy (MMD) instead which have much nicer behaviour to monitor drift. I highly recommend the Interpretable Comparison of Distributions and Models NeurIPS workshop talks for more in-depth background.
-
[D] Is this a reasonable assumption in machine learning?
All of the above functionality and more can be easily used under a simple API in https://github.com/SeldonIO/alibi-detect.
What are some alternatives?
python-tidal - Python API for TIDAL music streaming service
pytorch-widedeep - A flexible package for multimodal-deep-learning to combine tabular data with text and images using Wide and Deep models in Pytorch
wayfire - A modular and extensible wayland compositor
cleanlab - The standard data-centric AI package for data quality and machine learning with messy, real-world data and labels.
PySyft - Perform data science on data that remains in someone else's server
pyod - A Comprehensive and Scalable Python Library for Outlier Detection (Anomaly Detection)
edl - Inofficial Qualcomm Firehose / Sahara / Streaming / Diag Tools :)
seldon-core - An MLOps framework to package, deploy, monitor and manage thousands of production machine learning models
makinage - Stream Processing Made Easy
Anomaly_Detection_Tuto - Anomaly detection tutorial on univariate time series with an auto-encoder
frameworks - Sample code and build environments for MPC frameworks
conductor - Conductor is a microservices orchestration engine.