numexpr
gopy
numexpr | gopy | |
---|---|---|
4 | 5 | |
2,143 | 1,868 | |
0.7% | 1.4% | |
8.2 | 6.7 | |
about 1 month ago | 9 days ago | |
Python | Go | |
MIT License | BSD 3-clause "New" or "Revised" License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
numexpr
-
Making Python 100x faster with less than 100 lines of Rust
You can just slap numexpr on top of it to compile this line on the fly.
https://github.com/pydata/numexpr
- Extending Python with Rust
-
[D] How to avoid CPU bottlenecking in PyTorch - training slowed by augmentations and data loading?
Are you doing any costly chained NumPy operations in your preprocessing? E.g. max(abs(large_ary)), this produces multiple copies of your data, https://github.com/pydata/numexpr can greatly reduce time spent with such operations
-
Selection in pandas using query
What is not entirely obvious here is that under the hood you can install a nice library called numexpr (docs, src) that exists to make calculations with large NumPy (and pandas) objects potentially much faster. When you use query or eval, this expression is passed into numexpr and optimized using its bag of tricks. Expected performance improvement can be between .95x and up to 20x, with average performance around 3-4x for typical use cases. You can read details in the docs, but essentially numexpr takes vectorized operations and makes them work in chunks that optimize for cache and CPU branch prediction. If your arrays are really large, your cache will not be hit as often. If you break your large arrays into very small pieces, your CPU wonβt be as efficient.
gopy
-
Making Python 100x faster with less than 100 lines of Rust
I've used gopy[0] recently to access a go library in Python. It surprisingly Just Worked, but I was disappointed by some performance issues, like converting lists to slices.
[0] https://github.com/go-python/gopy
-
Golang vs python for AI
the heavy lifting is done in native libraries and you get to experiment fast using an easy language. the combo is quite hard to beat. Now there is a missed opportunity to write such libraries in Go, but as I read here and there Go is hard to integrate well as a library. There is gopy but it's light years away from PyO3 for instance, I don't think it'll ever gain traction, but who knows.
- Is the statement true, that Python and its ecosystem lacks speed for mission-critical large-scale applications?
-
I went about learning Rust
> So if you learn Go, you'll never be able to use it to interoperate with e.g. your Python program to speed it up.
Never done it myself, but:
https://www.ardanlabs.com/blog/2020/07/extending-python-with...
https://github.com/go-python/gopy
-
Rust or C/C++ to learn as a secondary language?
Check out gopy for an easy way to extend your Python code with Go.
What are some alternatives?
pytorch-lightning - Build high-performance AI models with PyTorch Lightning (organized PyTorch). Deploy models with Lightning Apps (organized Python to build end-to-end ML systems). [Moved to: https://github.com/Lightning-AI/lightning]
PySCIPOpt - Python interface for the SCIP Optimization Suite
pygfx - A python render engine running on wgpu.
Pulumi - Pulumi - Infrastructure as Code in any programming language. Build infrastructure intuitively on any cloud using familiar languages π
greptimedb - An open-source, cloud-native, distributed time-series database with PromQL/SQL/Python supported. Available on GreptimeCloud.
prisma-engines - π Engine components of Prisma ORM
jnumpy - Writing Python C extensions in Julia within 5 minutes.
poly-match - Source for the "Making Python 100x faster with less than 100 lines of Rust" blog post
jsmpeg - MPEG1 Video Decoder in JavaScript
cpy3 - Go bindings to the CPython-3 API
PythonCall.jl - Python and Julia in harmony.