CheeseShop
rayon
Our great sponsors
CheeseShop | rayon | |
---|---|---|
2 | 67 | |
1 | 10,242 | |
- | 2.9% | |
3.8 | 9.0 | |
7 months ago | 5 days ago | |
Rust | Rust | |
MIT License | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
CheeseShop
-
Apache Spark UDFs in Rust
By comparison, PyO3 handles virtually all that boilerplate, so your Rust functions can accept and return many native Rust types and everything just works (for example). Or maybe I'm missing some fundamental difference with how JVM data are handled versus Python.
-
PyO3: Rust Bindings for the Python Interpreter
At work, I'm using PyO3 for a project that churns through a lot of data (step 1) and does some pattern mining (step 2). This is the second generation of the project and is on-demand compared with the large, batch project in Spark that it is replacing. The Rust+Python project has really good performance, and using Rust for the core logic is such a joy compared with Scala or Python that a lot of other pieces are written in.
Learning PyO3, I cobbled together a sample project[0] to demonstrate how some functionality works. It's a little outdated (uses PyO3 0.11.0 compared with the current 0.13.1) and doesn't show everything, but I think it's reasonably clear.
One thing I noticed is that passing very large data from Rust and into Python's memory space is a bit of a challenge. I haven't quite grokked who owns what when and how memory gets correctly dropped, but I think the issues I've had are with the amount of RAM used at any moment and not with any memory leaks.
[0] https://github.com/aeshirey/CheeseShop
rayon
- Rayon: Data-race free parallelization of sequential computations in Rust
- Too Dangerous for C++
-
Which application/problem would you choose for presenting Rust to newcomers in 1h30min?
Do some operations with .iter() then later use rayon to parallelize. So you can show how easy is to add a dependency and how easy is to parallelize.
-
What Are The Rust Crates You Use In Almost Every Project That They Are Practically An Extension of The Standard Library?
rayon: Async CPU runtime for parallelism.
-
Moving from Typescript and Langchain to Rust and Loops
In the quest for more efficient solutions, the ONNX runtime emerged as a beacon of performance. The decision to transition from Typescript to Rust was an unconventional yet pivotal one. Driven by Rust's robust parallel processing capabilities using Rayon and seamless integration with ONNX through the ort crate, Repo-Query unlocked a realm of unparalleled efficiency. The result? A transformation from sluggish processing to, I have to say it, blazing-fast performance.
-
AreWeMegafactoryYet? I just breached simulating 1M buildings @ 60 fps (If I'm not recording, Ryzen 7 1700X 8 Core)
With a lot of rayon, blood, sweat and tears I finally managed to simulate a million buildings at 60fps :) Feel free to AMA, game is Combine And Conquer
-
The Rust I Wanted Had No Future
(see https://github.com/rayon-rs/rayon/tree/master/src/iter/plumbing)
-
Parallel event iterator?
I did some very basic testing with this crate : https://crates.io/crates/rayon and it seems to work :
-
General Recommendations: Should I Use Tree-sitter as the AST for the LSP I am developing?
Sequentially, generating tree-sitter AST for each file and querying for the links of each file takes around 2.3 seconds. However, I randomly remembered this crate rayon, and I decided to test it. It ended up improving the performance (just by changing 2 lines of code) to 200-300ms by parallelizing the iterators and tree-sitter queries. MAJOR.
-
python to rust migration
Now if you really want to use Rust, you can rewrite only the part that are slowing down your consumer. It's easy by using Py03 and maturin. Maybe also rayon to parallelize.
What are some alternatives?
ffi-overhead - comparing the c ffi (foreign function interface) overhead on various programming languages
crossbeam - Tools for concurrent programming in Rust
whatlang-pyo3 - Python Binding for Rust WhatLang, a language detection library
tokio - A runtime for writing reliable asynchronous applications with Rust. Provides I/O, networking, scheduling, timers, ...
dtparse - Fast datetime parser for Python written in Rust
RxRust - The Reactive Extensions for the Rust Programming Language
rust-numpy - PyO3-based Rust bindings of the NumPy C-API
pythran - Ahead of Time compiler for numeric kernels
tokio-rayon - Mix async code with CPU-heavy thread pools using Tokio + Rayon
py2many - Transpiler of Python to many other languages
sqlx - 🧰 The Rust SQL Toolkit. An async, pure Rust SQL crate featuring compile-time checked queries without a DSL. Supports PostgreSQL, MySQL, and SQLite.