Extending Python with Rust

This page summarizes the projects mentioned and recommended in the original post on news.ycombinator.com

Our great sponsors
  • Onboard AI - ChatGPT with full context of any GitHub repo.
  • InfluxDB - Power Real-Time Data Analytics at Scale
  • WorkOS - The modern API for authentication & user identity.
  • Numba

    NumPy aware dynamic Python compiler using LLVM

    This blog article was missing the first step of using Numba on numpy https://numba.pydata.org/

  • python-qubit-setup

    All scripts for controlling the instruments and acquiring data in our qubit setup.

    A long time (10 years) ago I wrote some custom C++ code to perform relatively simple arithmetic operations on very large arrays, which was considerably slower in numpy for some reason [1]. As I said that was long ago so maybe Numpy has improved by now, but back then there was a significant penalty when doing these operations directly in Python using Numpy, even though they were only a handful of things (e.g. calculate asin(x)+bcos(y) on some large array), so I didn't expect much overhead from going back and forth between F.

    1: https://github.com/adewes/python-qubit-setup/blob/master/lib...

  • Onboard AI

    ChatGPT with full context of any GitHub repo. Onboard AI learns any GitHub repo in minutes and lets you chat with it to locate functionality, understand different parts, and generate new code. Use it for free at app.getonboardai.com.

  • greptimedb

    An open-source, cloud-native, distributed time-series database with PromQL/SQL/Python supported.

    This is truly a fantastic combination -- implement the logic in Rust and use it in Python. GreptimeDB also implements a similar functionality that allows writing Python script to do post-process of SQL query results, with the help of RustPython and Arrow. Maybe this combination can bring a sweet point between performance and efficiency.

    docs: https://docs.greptime.com/user-guide/coprocessor-and-scripti...

    code: https://github.com/GreptimeTeam/greptimedb/tree/develop/src/...

  • julia

    The Julia Programming Language

    There are some nice tools for 3D PDEs which connect to DiffEq like GridAP (https://docs.sciml.ai/Gridap/stable/) and Ferrite (https://docs.sciml.ai/Ferrite/stable/). PDE tooling is where focus has been moving to as things evolve.

    As for JIT, just today there was a PR that was merged that makes Julia cache and reuse binaries of packages (https://github.com/JuliaLang/julia/pull/47184). It won't be out until the next release of Julia, but it's a pretty major improvement to not JIT fully inferred package calls.

  • NumPy

    The fundamental package for scientific computing with Python.

    Definitely. However, the equivalent for the array multiplication in numpy looks like this:

    https://github.com/numpy/numpy/blob/22e683d84f2584a6f9a57b2c... + https://github.com/numpy/numpy/blob/22e683d84f2584a6f9a57b2c...

    To know what they do, you need the source for INPLACE_GIVE_UP_IF_NEEDED (https://github.com/numpy/numpy/blob/b222eb66c79b8eccba39f46f...) and PyArray_GenericInplaceBinaryFunction (I don't even know where that's coming from, it's not defined in Numpy, maybe it's part of the Python interface?).

    In the end, both are unreadable in their own way. I personally prefer the Rust version above to the macrofied C version that's in Numpy but that's a matter of taste. I'd also trust the safe Rust implementations more than the C implementations because of the memory management guarantees Rust provides, though I suppose for simple operations like multiplication it'll be easier to make the program safe enough in C.

  • graphics_wgpu

    I made my own, using WGPU and this basic graphics engine: https://github.com/David-OConnor/graphics_wgpu

    Note that there's currently no docs or template/examples, and I'm rapidly breaking the API.

    The surface plots are just meshes of a grid divided into triangles.

    So, not a plotting lib at all; a flexible 3d and UI lib. Could probably made usable by others with a basic example of how you interact with the render and UI.

  • progenitor

    rule-based growth of hexagonal cells (by martinxyz)

    Part of that is the ndarray crate, which IMO is too generic often, making syntax complicated. It's not going to get pythonic but the nalgebra crate is a bit nicer IMO, example:

    https://github.com/martinxyz/progenitor/blob/85260/crates/pr...

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  • numexpr

    Fast numerical array expression evaluator for Python, NumPy, Pandas, PyTables and more

  • vswhere

    Locate Visual Studio 2017 and newer installations

    Finding where & how to use an installed VS instance (or selecting one) in automated tooling is solved by the criminally unknown, MIT licensed, MS supported, redistributable, vswhere tool: https://github.com/microsoft/vswhere

  • pygfx

    A python render engine running on wgpu.

    Rather than using matplotlib, you could try either pygfx (https://github.com/pygfx/pygfx) or fastplotlib (https://github.com/kushalkolar/fastplotlib) to make higher performance graphics using Python.

    However, it won't solve your problem of Python not being fast enough doing the calculations.

  • fastplotlib

    A fast plotting library

    Rather than using matplotlib, you could try either pygfx (https://github.com/pygfx/pygfx) or fastplotlib (https://github.com/kushalkolar/fastplotlib) to make higher performance graphics using Python.

    However, it won't solve your problem of Python not being fast enough doing the calculations.

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts