Our great sponsors
-
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
-
-
-
Apache Arrow
Apache Arrow is a multi-language toolbox for accelerated data interchange and in-memory processing
-
-
-
-
-
scalene
Scalene: a high-performance, high-precision CPU, GPU, and memory profiler for Python with AI-powered optimization proposals
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
Julia 1.9 is fast. And you can use https://github.com/Suzhou-Tongyuan/jnumpy to write python extension in Julia now. So I think after 1.9 release julia would be much more usable.
Semi Vectorized code:
https://github.com/ohadravid/poly-match/blob/main/poly_match...
Expecting Python engineers unable to read defacto standard numpy code but meanwhile expect everyone can read Rust.....
Not to mention that the semi-vectorized code is still suboptimal. Too many for loops despite the author clearly know they can all be vectorized.
For example instead the author can just write something like:
np.argmin(
>Today, there is a Python package for everything.
The same could be said about CPAN and NPM. Yet Perl is basically dead and JavaScript isn't used for any machine learning tasks as far as I'm aware. WebAssembly did help bring a niche array of audio and video codecs to the ecosystem[1][2], something I'm yet to see from Python.
I don't use Python, but with what little exposure I've had to it at work, its overall sluggish performance and need to set up a dozen virtualenvs -- only to dockerize everything in cursed ways when deploying -- makes me wonder how or why people bother with it at all beyond some 5-line script. Then again, Perl used to be THE glue language in the past and mod_perl was as big as FastAPI, and Perl users would also point out how CPAN was unparalleled in breadth and depth. I wonder if Python will follow a similar route as Perl. One can hope :-)
You can have your cake and eat it with the likes of
* PythonCall.jl - https://github.com/cjdoris/PythonCall.jl
* NodeCall.jl - https://github.com/sunoru/NodeCall.j
* RCall.jl - https://github.com/JuliaInterop/RCall.jl
I tend to use Julia for most things and then just dip into another language’s ecosystem if I can’t find something to do the job and it’s too complex to build myself
You can have your cake and eat it with the likes of
* PythonCall.jl - https://github.com/cjdoris/PythonCall.jl
* NodeCall.jl - https://github.com/sunoru/NodeCall.j
* RCall.jl - https://github.com/JuliaInterop/RCall.jl
I tend to use Julia for most things and then just dip into another language’s ecosystem if I can’t find something to do the job and it’s too complex to build myself
You can have your cake and eat it with the likes of
* PythonCall.jl - https://github.com/cjdoris/PythonCall.jl
* NodeCall.jl - https://github.com/sunoru/NodeCall.j
* RCall.jl - https://github.com/JuliaInterop/RCall.jl
I tend to use Julia for most things and then just dip into another language’s ecosystem if I can’t find something to do the job and it’s too complex to build myself
I wonder why GraalVM is not more often used for these speed critical cases: https://www.graalvm.org/python/
Is the problem the Oracle involvement? (Same for ruby https://www.graalvm.org/ruby/)
One of the others understated pros of rewriting some parts in Rust, it's that you can parallelize easily with Rayon[0]
I've used gopy[0] recently to access a go library in Python. It surprisingly Just Worked, but I was disappointed by some performance issues, like converting lists to slices.
Apache Arrow (https://arrow.apache.org/) is built exactly around this idea: it's a library for managing the in-memory representation of large datasets.
in the domain of Python tooling made faster with Rust, check out https://github.com/charliermarsh/ruff which is 10-100x faster than pylint etc
You can just slap numexpr on top of it to compile this line on the fly.
LineProfiler is the best tool to learn how to write performant Python and code optimization.
https://github.com/pyutils/line_profiler
You can literally see the hot spot of your code, then you can grind different algorithms or change the whole architecture to make it faster.
For example replace short for loops to list comprehensions, vectorize all numpy operations (only vectorize partially do not help the issue), using 'not any()' instead or 'all()' for boolean, etc.
Doing this for like 2 weeks, basically you can automatically recognized most bad code patterns in a glance.
Very nice. Thanks for sharing.
One of Julia's Achilles heels is standalone, ahead-of-time compilation. Technically this is already possible [1], [2], but there are quite a few limitations when doing this (e.g. "Hello world" is 150 MB [7]) and it's not an easy or natural process.
The immature AoT capabilities are a huge pain to deal with when writing large code packages or even when trying to make command line applications. Things have to be recompiled each time the Julia runtime is shut down. The current strategy in the community to get around this seems to be "keep the REPL alive as long as possible" [3][4][5][6], but this isn't a viable option for all use cases.
Until Julia has better AoT compilation support, it's going to be very difficult to develop large scale programs with it. Version 1.9 has better support for caching compiled code, but I really wish there were better options for AoT compiling small, static, standalone executables and libraries.
You should take a look at Scalene - it's even better.