numexpr
poly-match
numexpr | poly-match | |
---|---|---|
4 | 6 | |
2,143 | 31 | |
0.7% | - | |
8.2 | 2.3 | |
about 1 month ago | 25 days ago | |
Python | Python | |
MIT License | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
numexpr
-
Making Python 100x faster with less than 100 lines of Rust
You can just slap numexpr on top of it to compile this line on the fly.
https://github.com/pydata/numexpr
- Extending Python with Rust
-
[D] How to avoid CPU bottlenecking in PyTorch - training slowed by augmentations and data loading?
Are you doing any costly chained NumPy operations in your preprocessing? E.g. max(abs(large_ary)), this produces multiple copies of your data, https://github.com/pydata/numexpr can greatly reduce time spent with such operations
-
Selection in pandas using query
What is not entirely obvious here is that under the hood you can install a nice library called numexpr (docs, src) that exists to make calculations with large NumPy (and pandas) objects potentially much faster. When you use query or eval, this expression is passed into numexpr and optimized using its bag of tricks. Expected performance improvement can be between .95x and up to 20x, with average performance around 3-4x for typical use cases. You can read details in the docs, but essentially numexpr takes vectorized operations and makes them work in chunks that optimize for cache and CPU branch prediction. If your arrays are really large, your cache will not be hit as often. If you break your large arrays into very small pieces, your CPU won’t be as efficient.
poly-match
-
Improving Interoperability Between Rust and C++
Not my experience at all. At work we rewrote a small bit of hotspot python in Rust with no issues. This was what we primarily followed: https://ohadravid.github.io/posts/2023-03-rusty-python/
-
How to convince my boss that Rust is usable
Take at look at this example, it still uses Python as an interface to Rust code. Maybe you can do something similar to still achieve performance improvements without changing the entire codebase.
-
GDScript is fine
People are probably downvoting because it's needlessly hyperbolic and argumentative. Nobody is saying that python isn't faster to iterate with, but they're arguing that it would take months to get negligable performance gains in a lower level language, meanwhile here is a recent post from a company that increased the execution of they're python code by 100x with less than 100 lines of Rust. They also claim that nobody cares if something runs a few milliseconds faster, when we're talking about game dev, where games are frequently judged on how many milliseconds it takes to run game logic between frames.
-
Making Python 100x faster with less than 100 lines of Rust
Semi Vectorized code:
https://github.com/ohadravid/poly-match/blob/main/poly_match...
Expecting Python engineers unable to read defacto standard numpy code but meanwhile expect everyone can read Rust.....
Not to mention that the semi-vectorized code is still suboptimal. Too many for loops despite the author clearly know they can all be vectorized.
For example instead the author can just write something like:
np.argmin(
-
Blog Post: Making Python 100x faster with less than 100 lines of Rust
The article links to a full implementation, so you should be able to test this.
What are some alternatives?
pytorch-lightning - Build high-performance AI models with PyTorch Lightning (organized PyTorch). Deploy models with Lightning Apps (organized Python to build end-to-end ML systems). [Moved to: https://github.com/Lightning-AI/lightning]
jnumpy - Writing Python C extensions in Julia within 5 minutes.
pygfx - A python render engine running on wgpu.
gopy - gopy generates a CPython extension module from a go package.
greptimedb - An open-source, cloud-native, distributed time-series database with PromQL/SQL/Python supported. Available on GreptimeCloud.
StaticCompiler.jl - Compiles Julia code to a standalone library (experimental)
truffleruby - A high performance implementation of the Ruby programming language, built on GraalVM.
jsmpeg - MPEG1 Video Decoder in JavaScript
birthday-book-app - Rust in Anger: high-performance web applications
ruff - An extremely fast Python linter and code formatter, written in Rust.
PythonCall.jl - Python and Julia in harmony.