Our great sponsors
-
maturin
Build and publish crates with pyo3, cffi and uniffi bindings as well as rust binaries as python packages
-
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
-
-
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
-
-
-
-
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
With PyO3, I built the library to parse datetimes 10x faster than `datetime.strptime` in just a few lines of code: https://github.com/gukoff/dtparse
It just calls the Rust's chrono library that does the parsing and wraps the result in a Python object. You can do it for any Rust library, it's very, very easy!
The only slightly complicated part is the distribution. You need to use https://github.com/PyO3/maturin or https://github.com/PyO3/setuptools-rust, and of course, you need to have Rust installed on the wheel-building machine.
Feel free to use this repo as a reference if you want to build a similar thing. The code is commented, and there's a working GitHub action that builds the wheels for all platforms and uploads them to PyPi: https://github.com/gukoff/dtparse/tree/master/.github/workfl...
This is awesome, thanks for sharing! I think this should be added to the PyO3 examples list :)
https://github.com/PyO3/pyo3#examples
I've been playing with PyO3 for prototyping, and wrapped some Rust code to see if it's faster than Python. The experience was very much like using Boost Python (whcih these days has alternative with https://github.com/pybind/pybind11). It's _really_ easy to wrap code for Python, and it has nice APIs to ensure GIL is held. Being Rust, I'm much more confident I won't suffer from memory unsafety issues which my C++ at the time did.
Now I'm starting to use it as part of Python memory profiler I'm working on (https://pythonspeed.com/fil), in this case to call in to the low-level Python C API which it automatically wraps. At this point it's more like C, except with the benefit of having high-level APIs (for GIL holding, but also object conversion) available when I need it.
Highly recommend trying it out.
Related: RustPython - A Python interpreter written in Rust.
https://github.com/RustPython/RustPython
At work, I'm using PyO3 for a project that churns through a lot of data (step 1) and does some pattern mining (step 2). This is the second generation of the project and is on-demand compared with the large, batch project in Spark that it is replacing. The Rust+Python project has really good performance, and using Rust for the core logic is such a joy compared with Scala or Python that a lot of other pieces are written in.
Learning PyO3, I cobbled together a sample project[0] to demonstrate how some functionality works. It's a little outdated (uses PyO3 0.11.0 compared with the current 0.13.1) and doesn't show everything, but I think it's reasonably clear.
One thing I noticed is that passing very large data from Rust and into Python's memory space is a bit of a challenge. I haven't quite grokked who owns what when and how memory gets correctly dropped, but I think the issues I've had are with the amount of RAM used at any moment and not with any memory leaks.
[0] https://github.com/aeshirey/CheeseShop
https://github.com/PyO3/rust-numpy
Between pyodide, pyo3, rust-cpython, and rustpython, I think Pyo3 is the best way to drop in rust in a python project for a speed up, if that is your goal. Some of the demos show using python from rust, but to me the biggest feature is without a doubt compiling rust code to native python modules. I'm using it to speed up image manipulation backed by numpy arrays.
There’s a setuptools rust [0] extension package that can be used to hook the compilation of the rust into the wheel building or install from source. Maturin [1] seems to be regarded as the new and improved solution for this, but I found that it’s angled toward the using python from rust.
There’s also the rust numpy [2] package by the same org which is fantastic in that it lets you pass a numpy matrix to a native method written in rust and convert it to the rust equivalent data structure, perform whatever transformation you want (in parallel using rayon [3]), and return the array. When building for release, I was seeing speed ups of 100x over numpy on the most matrix mathable function imaginable, and numpy is no joke.
I think there is a lot of potential for these two ecosystems together. If there’s not a python package for something, there’s probably a rust crate.
If anyone is interested the python package that I'm building with some rust backend, its called pyrogis [4] for making custom image manipulations through numpy arrays.
https://github.com/PyO3/setuptools-rust
https://github.com/pierogis/pierogis
https://github.com/rayon-rs/rayon
Huggingface Tokenizers (https://github.com/huggingface/tokenizers), which are now used by default in their Transformers Python library, use pyO3 and became popular due to the pitch that it encoded text an order of magnitude faster with zero config changes.
It lives up to that claim. (I had issues with return object typing when going between Python/Rust at first but those are more consistent now)
There is another way to speed up python:
Write code in python and transpile to another language (could be rust) and then import it back into python
https://github.com/adsharma/py2many/tree/main/tests/expected
Figuring out a mapping between a subset of a compiled language and a subset of statically typed python should be possible.
The hard part is mapping standard library. I suspect something like nim might have an advantage there.
[3] https://github.com/paugier/nbabel
[1] https://github.com/serge-sans-paille/pythran
I'm interested in running Python inside wasmtime. I think PyO3 would be useful. We could build a small Rust wasm binary that exports an "execute_python_script" function. This would finally be a way to run Python in a strong sandbox with memory [0] and CPU [1] restrictions. (In 1999, I asked Guido for sandboxing support in Python, but he refused.)
[0] https://github.com/bytecodealliance/wasmtime/issues/2273
[1] https://github.com/bytecodealliance/wasmtime/issues/2274
ciso8601 is blazingly fast, and also its wall time is very stable. By all means, use ciso8601 if the format allows :)
On my machine, ciso8601 always runs in 240ns, and the Rust lib median time is 1250ns.
You can run a benchcmark too! Just call pytest, and it will generate an .svg report: https://github.com/gukoff/dtparse/blob/master/tests/test_per...