linfa
db-benchmark
linfa | db-benchmark | |
---|---|---|
14 | 91 | |
3,426 | 320 | |
2.5% | 0.0% | |
6.3 | 0.0 | |
9 days ago | 11 months ago | |
Rust | R | |
Apache License 2.0 | Mozilla Public License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
linfa
- Why is Rust not more popular in ML and secure edge computing?
-
Polars vs ndarray performance
I've been playing with data analytics and ml in rust for the last couple of weeks. A typical ML job requires transforming some data to feed the ml model to the then train the model. For ML I've been using linfa (https://github.com/rust-ml/linfa) which is surprisingly nice. I've been experimenting with ndarray and polars for data transformation (linfa uses ndarray) - from a UX standpoint. I'm pretty surprised by polars' performance (https://h2oai.github.io/db-benchmark/), which sits on top of arrow2, and it's definitely a great candidate for OLAP tasks. But I couldn't find any comparison between ndarray and polars, has anyone had any meaningful experience with the two or/and can point me to a benchmark comparison?
-
Ask HN: What is the job market like, for niche languages (Nim, crystal)?
The most comprehensive current view of the Rust machine learning ecosystem at the moment is probably at https://www.arewelearningyet.com/ (I sometimes help maintain this site)
Rust has a weird mix at the moment, and not one that's likely to significantly change within the next 12 months, at least. Certain tools are genuinely best-in-class, especially around simple operations on insane amounts of data. Rust kills it in that space due to its native speed and focus on concurrency.
There's also growing projects like Linfa [1]. that while not at the level of scikit-learn, have significantly increased their coverage on common data science/classical ML problems in the past couple years, along with improved tooling. The space does have a few pure-Rust projects coming down the pipeline around autodifferentiation, GPU compute, etc. that are likely to yield some really valuable results in deep learning, but that aren't quite available and will take some time to pick up some traction even once they're released. At the same time, areas like data visualization are unlikely to reach parity with something like matplotlib/pyplot in the near future.
Python is the de-facto standard, and will be for some time, but Rust's ability to build accessible high-level APIs on top of performant, language-native libraries is attracting some attention and I wouldn't be surprised to start seeing ingress in the certain areas over the next few years, where instead of the Python/C++ combination, it's just Rust all the way down.
[1] https://github.com/rust-ml/linfa
-
Is RUST aiming to build an ecosystem on scientific computing?
take a look at https://github.com/rust-ml/linfa for machine learning related crates
-
What is a FOSS which is needed but doesn't exist yet/needs contributers?
Check out smartcore and linfa. At work I was badly in need of an NMF function similar to MATLAB's one these days but not enough time to write one myself. If you're good at math and machine learning, this sounds like a task you could try tackling.
- Any role that Rust could have in the Data world (Big Data, Data Science, Machine learning, etc.)?
-
How far along is the ML ecosystem with Rust?
For other algorithms, there is not yet a single library to rule them all (linfa might become that at some point) but searching for the algorithm you need on crate.io is likely to give you some results (obligatory plug to Friedrich, my gaussian process implementation).
- Linfa: A Rust machine learning framework
-
AII4DEVS #10: Diverse knowledge is the key to grow the next generation of ML practitioners into AI engineers.
To all folks in love with Rust programming language, **linfa** is a promising library to check out: a complete porting of the well known scikit-learn library, which enables common preprocessing tasks and classical ML algorithms such as clustering, linear learners, logistic regression, and decision trees as well as support vector machines and Bayesian algorithms such as Naive Bayes. We all know that Python has the 98% of the machine learning languages market share, but if I looked to something else, a super-fast Rust implementation would be my first stop.
-
Linfa has a website now!
for a start I will implement the TryFrom for Dataset under a feature flag. But to be really useful some of the algorithms have to start using something like DatasetBase here Records are currently bounded by an associated type for the element type, we would have to relax that too. Just read your blogpost on polars ๐
db-benchmark
- Database-Like Ops Benchmark
-
Polars
Real-world performance is complicated since data science covers a lot of use cases.
If you're just reading a small CSV to do analysis on it, then there will be no human-perceptible difference between Polars and Pandas. If you're reading a larger CSV with 100k rows, there still won't be much of a perceptible difference.
Per this (old) benchmark, there are differences once you get into 500MB+ territory: https://h2oai.github.io/db-benchmark/
-
DuckDB performance improvements with the latest release
I do think it was important for duckdb to put out a new version of the results as the earlier version of that benchmark [1] went dormant with a very old version of duckdb with very bad performance, especially against polars.
[1] https://h2oai.github.io/db-benchmark/
-
Show HN: SimSIMD vs. SciPy: How AVX-512 and SVE make SIMD cleaner and ML faster
https://news.ycombinator.com/item?id=33270638 :
> Apache Ballista and Polars do Apache Arrow and SIMD.
> The Polars homepage links to the "Database-like ops benchmark" of {Polars, data.table, DataFrames.jl, ClickHouse, cuDF, spark, (py)datatable, dplyr, pandas, dask, Arrow, DuckDB, Modin,} but not yet PostgresML? https://h2oai.github.io/db-benchmark/ *
LLM -> Vector database: https://en.wikipedia.org/wiki/Vector_database
/? inurl:awesome site:github.com "vector database"
-
Pandas vs. Julia โ cheat sheet and comparison
I agree with your conclusion but want to add that switching from Julia may not make sense either.
According to these benchmarks: https://h2oai.github.io/db-benchmark/, DF.jl is the fastest library for some things, data.table for others, polars for others. Which is fastest depends on the query and whether it takes advantage of the features/properties of each.
For what it's worth, data.table is my favourite to use and I believe it has the nicest ergonomics of the three I spoke about.
-
Any faster Python alternatives?
Same. Numba does wonders for me in most scenarios. Yesterday I've discovered pola-rs and looks like I will add it to the stack. It's API is similar to pandas. Have a look at the benchmarks of cuDF, spark, dask, pandas compared to it: Benchmarks
-
Pandas 2.0 (with pyarrow) vs Pandas 1.3 - Performance comparison
The syntax has similarities with dplyr in terms of the way you chain operations, and itโs around an order of magnitude faster than pandas and dplyr (thereโs a nice benchmark here). Itโs also more memory-efficient and can handle larger-than-memory datasets via streaming if needed.
-
Pandas v2.0 Released
If interested in benchmarks comparing different dataframe implementations, here is one:
https://h2oai.github.io/db-benchmark/
- Database-like ops benchmark
-
Python "programmers" when I show them how much faster their naive code runs when translated to C++ (this is a joke, I love python)
Bad examples. Both numpy and pandas are notoriously un-optimized packages, losing handily to pretty much all their competitors (R, Julia, kdb+, vaex, polars). See https://h2oai.github.io/db-benchmark/ for a partial comparison.
What are some alternatives?
smartcore - A comprehensive library for machine learning and numerical computing. The library provides a set of tools for linear algebra, numerical computing, optimization, and enables a generic, powerful yet still efficient approach to machine learning.
polars - Dataframes powered by a multithreaded, vectorized query engine, written in Rust
Awesome-Rust-MachineLearning - This repository is a list of machine learning libraries written in Rust. It's a compilation of GitHub repositories, blogs, books, movies, discussions, papers, etc. ๐ฆ
datafusion - Apache DataFusion SQL Query Engine
rust-ndarray - ndarray: an N-dimensional array with array views, multidimensional slicing, and efficient operations
Apache Arrow - Apache Arrow is a multi-language toolbox for accelerated data interchange and in-memory processing
rusty-machine - Machine Learning library for Rust
databend - ๐๐ฎ๐๐ฎ, ๐๐ป๐ฎ๐น๐๐๐ถ๐ฐ๐ & ๐๐. Modern alternative to Snowflake. Cost-effective and simple for massive-scale analytics. https://databend.com
Enzyme - High-performance automatic differentiation of LLVM and MLIR.
sktime - A unified framework for machine learning with time series
tract - Tiny, no-nonsense, self-contained, Tensorflow and ONNX inference
DataFramesMeta.jl - Metaprogramming tools for DataFrames