Ingest, store, & analyze all types of time series data in a fully-managed, purpose-built database. Keep data forever with low-cost storage and superior data compression. Learn more →
Top 23 Rust Python Projects
-
Project mention: Any job processing framework like Spark but in Rust? | reddit.com/r/dataengineering | 2023-03-23
For data frames built on Apache Arrow and: https://github.com/pola-rs/polars/
-
I mean considering this is their pylint compatibility I doubt they would start implementing random requests https://github.com/charliermarsh/ruff/issues/970
-
SonarQube
Static code analysis for 29 languages.. Your projects are multi-language. So is SonarQube analysis. Find Bugs, Vulnerabilities, Security Hotspots, and Code Smells so you can release quality code every time. Get started analyzing your projects today for free.
-
So as a first step you'll want to profile your program to figure out where it's slow, and hopefully that'll also tell you why it's slow. I'm the (biased) author of the Sciagraph profiler which is designed for this sort of application (https://sciagraph.com) but you can also try py-spy, which isn't as well designed for data processing/analysis applications (e.g. it won't visualize parallelism at all) but can still be informative (https://github.com/benfred/py-spy). Both are written in Rust ;)
-
I use Qt but I use PyO3 to set up a "Python/PyQt/PySide frontend on top of Rust backend" architecture for the QWidget APIs similar to the "QML frontend on top of C++ backend" that they officially promote for Qt Quick.
-
Project mention: Bridging Async and Sync Rust Code - A lesson learned while working with Tokio | reddit.com/r/rust | 2023-03-10
Problem comes when you want to do this inside an async context since we couldn't block an async task. https://users.rust-lang.org/t/sync-function-invoking-async/43364/6 You might need to do it in another runtime/thread. It is not recommended to do this, but sometimes it is unavoidable while implementing a third-party trait. https://github.com/apache/arrow-datafusion/issues/3777 However, I believe this isn't a problem particular to tokio, or any specific runtime.
-
Project mention: Show HN: ILLA is an Open-source alternative to Retool | news.ycombinator.com | 2022-11-19
Not OP but Authentication is easy, authorization is a cross-cutting concern that often requires custom code. E.g., there are people and teams, both of which can have different kinds of access to something (read/write). Sometimes teams have sub-teams. Do the sub-teams have access to the parent teams' resources and/or vice versa? Also what kind of sharing are you going to support? Do people have to have an account to view stuff shared to them or can you just send a link? There are some efforts to make custom DSLs for describing authorization policies, to avoid cross-cutting code[1].
Computed fields require different treatment at every level of the stack. This isn't inherently hard, but it is an extra feature these low-code/no-code platforms need. Where things get difficult is inn migrations. It's common for a field that is computed at the beginning to become customizable, or for the computation to change. When that happens, what should the value be for old columns? Computed fields also often pull data from multiple other tables, which may require some combination of custom queries and database optimization.
-
postgresml
PostgresML is an end-to-end machine learning system. It enables you to train models and make online predictions using only SQL, without your data ever leaving your favorite database.
Project mention: How to store hugging face model in postgreSQL | reddit.com/r/LanguageTechnology | 2023-02-05I'd encourage you to do inference outside of PostgreSQL (use TF serving and make requests against it, or do batch inference), but if you're determined to do so, they have an extension that integrates with the transformers library and allows for calling models directly from SQL.
-
InfluxDB
Access the most powerful time series database as a service. Ingest, store, & analyze all types of time series data in a fully-managed, purpose-built database. Keep data forever with low-cost storage and superior data compression.
-
erg – A Python-compatible statically typed language
-
maturin
Build and publish crates with pyo3, rust-cpython and cffi bindings as well as rust binaries as python packages
Lastly if you're willing to introduce Rust, I'd consider a gradual approach using native libraries built in rust with PYO3. Check the maturin guide that helps you to streamline the build process of native libraries : https://github.com/PyO3/maturin . From there you could try to find hotspots in your python app and replace those with a native implementation.
-
Project mention: Announcing Scale: A WebAssembly runtime for running Go functions in any other language | reddit.com/r/golang | 2023-03-14
Also I discovered https://github.com/extism/extism recently but I don’t quite understand how their API works.
-
Project mention: WebAssembly: Adding Python Support to WASM Language Runtimes | news.ycombinator.com | 2023-01-31
PyOdide isn't currently supported outside of browsers, though that might change.
Either way, I couldn't figure out how to do the above sequence of steps with any of the available Python WASM runtimes - they're all very under-documented at the moment, sadly. I tried all three of these:
- https://github.com/wasmerio/wasmer-python
-
I've never used pyo3, just cpython, but the latter at least let me do things like:
-
rerun
Log images, point clouds, etc, and visualize them effortlessly. Built in Rust using egui (by rerun-io)
Project mention: Leveraging Rust and the GPU to render user interfaces at 120 FPS | news.ycombinator.com | 2023-03-09Egui seems well suited to render even tens of thousands pointcloud with wgpu backend.
-
lance
Modern columnar data format for ML implemented in Rust. Convert from parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, with more integrations coming..
Project mention: Any job processing framework like Spark but in Rust? | reddit.com/r/dataengineering | 2023-03-23For Feature Stores check out: https://github.com/eto-ai/lance
-
-
I was trying https://github.com/sfu-db/connector-x and hacking around with this https://github.com/spitz-dan-l/postgres-binary-parser but it turned out that a COPY to csv using asyncpg and then converting to parquet was the fastest.
-
Av1an
Cross-platform command-line AV1 / VP9 / HEVC / H264 encoding framework with per scene quality encoding
That might be because it doesn't work.
-
If you absolutely have to use only basic Python for setting up a data lake, the only thing that comes close is delta-rs for delta lake, another open-source lakehouse format. It is still very new, but it does work.
-
Project mention: Robyn: a fast and extensible async Python web server with a Rust runtime | reddit.com/r/rust | 2023-01-16
Recently I found this ASGI compatible server written in rust, but i haven’t tried it yet https://github.com/emmett-framework/granian
-
-
-
-
Project mention: Any job processing framework like Spark but in Rust? | reddit.com/r/dataengineering | 2023-03-23
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
NOTE:
The open source projects on this list are ordered by number of github stars.
The number of mentions indicates repo mentiontions in the last 12 Months or
since we started tracking (Dec 2020).
The latest post mention was on 2023-03-23.
Rust Python related posts
- web service framework and OpenAPI spec
- Any job processing framework like Spark but in Rust?
- Create/write Iceberg with Python?
- no good UI toolkit/framework
- Is Rust faster than Python out of the box
- I use dwm btw
- An example for OpenCL 3.0?
-
A note from our sponsor - InfluxDB
www.influxdata.com | 25 Mar 2023
Index
What are some of the best open-source Python projects in Rust? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | polars | 15,640 |
| 2 | ruff | 11,045 |
| 3 | py-spy | 9,882 |
| 4 | PyO3 | 7,981 |
| 5 | arrow-datafusion | 3,365 |
| 6 | oso | 2,955 |
| 7 | postgresml | 2,395 |
| 8 | erg | 2,145 |
| 9 | maturin | 1,890 |
| 10 | extism | 1,783 |
| 11 | wasmer-python | 1,736 |
| 12 | rust-cpython | 1,710 |
| 13 | rerun | 1,570 |
| 14 | lance | 1,311 |
| 15 | svix-webhooks | 1,263 |
| 16 | connector-x | 1,164 |
| 17 | Av1an | 1,027 |
| 18 | delta-rs | 956 |
| 19 | granian | 755 |
| 20 | autopy | 717 |
| 21 | astro | 677 |
| 22 | chromepass | 657 |
| 23 | arrow-ballista | 644 |
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com