numexpr VS poly-match

You can just slap numexpr on top of it to compile this line on the fly.

https://github.com/pydata/numexpr

Are you doing any costly chained NumPy operations in your preprocessing? E.g. max(abs(large_ary)), this produces multiple copies of your data, https://github.com/pydata/numexpr can greatly reduce time spent with such operations

What is not entirely obvious here is that under the hood you can install a nice library called numexpr (docs, src) that exists to make calculations with large NumPy (and pandas) objects potentially much faster. When you use query or eval, this expression is passed into numexpr and optimized using its bag of tricks. Expected performance improvement can be between .95x and up to 20x, with average performance around 3-4x for typical use cases. You can read details in the docs, but essentially numexpr takes vectorized operations and makes them work in chunks that optimize for cache and CPU branch prediction. If your arrays are really large, your cache will not be hit as often. If you break your large arrays into very small pieces, your CPU won’t be as efficient.

Not my experience at all. At work we rewrote a small bit of hotspot python in Rust with no issues. This was what we primarily followed: https://ohadravid.github.io/posts/2023-03-rusty-python/

Take at look at this example, it still uses Python as an interface to Rust code. Maybe you can do something similar to still achieve performance improvements without changing the entire codebase.

People are probably downvoting because it's needlessly hyperbolic and argumentative. Nobody is saying that python isn't faster to iterate with, but they're arguing that it would take months to get negligable performance gains in a lower level language, meanwhile here is a recent post from a company that increased the execution of they're python code by 100x with less than 100 lines of Rust. They also claim that nobody cares if something runs a few milliseconds faster, when we're talking about game dev, where games are frequently judged on how many milliseconds it takes to run game logic between frames.

Semi Vectorized code:

https://github.com/ohadravid/poly-match/blob/main/poly_match...

Expecting Python engineers unable to read defacto standard numpy code but meanwhile expect everyone can read Rust.....

Not to mention that the semi-vectorized code is still suboptimal. Too many for loops despite the author clearly know they can all be vectorized.

For example instead the author can just write something like:

   np.argmin(

The article links to a full implementation, so you should be able to test this.