arrow2
DISCONTINUED
PyO3
Our great sponsors
arrow2 | PyO3 | |
---|---|---|
25 | 146 | |
1,071 | 10,791 | |
- | 4.4% | |
0.0 | 9.8 | |
about 1 month ago | 2 days ago | |
Rust | Rust | |
Apache License 2.0 | GNU General Public License v3.0 or later |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
arrow2
-
Polars: Company Formation Announcement
One of the interesting components of Polars that I've been watching is the use of the Apache Arrow memory format, which is a standard layout for data in memory that enables processing (querying, iterating, calculating, etc) in a language agnostic way, in particular without having to copy/convert it into the local object format first. This enables cross-language data access by mmaping or transferring a single buffer, with zero [de]serialization overhead.
For some history, there's has been a bit of contention between the official arrow-rs implementation and the arrow2 implementation created by the polars team which includes some extra features that they find important. I think the current status is that everyone agrees that having two crates that implement the same standard is not ideal, and they are working to port any necessary features to the arrow-rs crate and plan on eventually switching to it and deprecating arrow2. But that's not easy.
-
Data Engineering with Rust
https://github.com/jorgecarleitao/arrow2 https://github.com/apache/arrow-datafusion https://github.com/apache/arrow-ballista https://github.com/pola-rs/polars https://github.com/duckdb/duckdb
-
Polars[Query Engine/ DataFrame] 0.28.0 released :)
Currently datafusion and polars aren't directly operable iirc because they use different underlying arrows implementations, but there seems to be work being done on that here https://github.com/jorgecarleitao/arrow2/issues/1429
-
Rust is showing a lot of promise in the DataFrame / tabular data space
[arrow2](https://github.com/jorgecarleitao/arrow2) and [parquet2](https://github.com/jorgecarleitao/parquet2) are great foundational libraries for and DataFrame libs in Rust.
-
Matano - Open source security lake built with Arrow2 + Rust
[1] https://github.com/jorgecarleitao/arrow2
-
Polars 0.23.0 released
In lockstep with arrow2's 0.13 release, we have published polars 0.23.0.
-
::lending-iterator — Lending/streaming Iterators on Stable Rust (and a pinch of HKT)
This is so freaking life-saving! - we have been using StreamingIterator and FallibleStreamingIterator in libraries (arrow2 and parquet2) and the existing landscape is quite confusing for new users!
-
Polars 0.22 is released!
In lockstep with a new release of arrow2: https://github.com/jorgecarleitao/arrow2/releases/tag/v0.12.0
- Arrow2 0.12.0 released - including almost complete support for Parquet
- Anda para aqui alguém a brincar com Rust (linguagem)?
PyO3
- Polars – A bird's eye view of Polars
-
In Rust for Python: A Match from Heaven
This story unfolds as a captivating journey where the agile Flounder, representing the Python programming language, navigates the vast seas of coding under the wise guidance of Sebastian, symbolizing Rust. Central to their adventure are three powerful tridents: cargo, PyO3, and maturin.
- Segunda linguagem
-
Calling Rust from Python
I would not recommend FFI + ctypes. Maintaining the bindings is tedious and error-prone. Also, Rust FFI/unsafe can be tricky even for experienced Rust devs.
Instead PyO3 [1] lets you "write a native Python module in Rust", and it works great. A much better choice IMO.
-
Python 3.12
Same w/ Rust and Python, this is really neat because now each thread could have a GIL without doing exactly what you said. The pyO3 commit to allow subinterpreters was merged 21 days ago, so this might "just work" today: https://github.com/PyO3/pyo3/pull/3446
-
Removing Garbage Collection from the Rust Language (2013)
I expected someone to write a rust-based scripting language which tightly integrated with rust itself.
In reality, it seems like the python developers and toolchain are embracing rust enough to reduce the benefits to a new alternative.
-
Bytewax: Stream processing library built using Python and Rust
Hey HN! I am one of the people working on Bytewax. Bytewax came out of our experience working with ML infrastructure at GitHub. We wanted to use Python because we could move fast, the team was very fluent in it, and the rest of our tooling was Python-native already. We didn't want to introduce JVM-based solutions into our stack because of the lack of experience and the friction we had trying to get Python-centric tooling working with existing solutions like Flink.
In our research, we found Timely Dataflow (https://timelydataflow.github.io/timely-dataflow/, https://news.ycombinator.com/item?id=24837031) and the Naiad project (https://www.microsoft.com/en-us/research/project/naiad/) as well as PyO3 (https://github.com/PyO3/pyo3) and we thought we found a match made in heaven :). Bytewax leverages both of these projects and builds on them to provide a clean API (at least we think so) and table stakes features like connectors, state recovery, and cloud-native scaling. It has been really cool to learn about the dataflow computation model, Rust, and how to wrangle the GIL with Rust and Python :P.
Would love to get your feedback :).
`pip install bytewax` to get started. We have a page of guides (https://www.bytewax.io/guides) with ready-to-run examples.
-
Tell HN: Rust Is the Superglue
You can practice your Rust skills by writing performant and/or gluey extensions for higher-level language such as NodeJS (checkout napi-rs) and Python or complementing JS in the browser if you target Webassembly.
For instance, checkout Llama-node https://github.com/Atome-FE/llama-node for an involved Rust-based NodeJS extension. Python has PyO3, a Rust-Python extension toolset: https://github.com/PyO3/pyo3.
They can help you leverage your Rust for writing cool new stuff.
-
Writing Python Like Rust
(2020).
Things have arguably become even nicer (although slightly more divergent between the two) since then: Python's `Optional[T]` can now be written as `T | None`, and the core container types can now be annotated directly (e.g. `List[T]` becomes `list[T]`).
Combined via pyO3[1], Python and Rust are a real joy to write together.
- 🚀 GoRules Zen Engine: Rules Engine for Node.js
What are some alternatives?
rust-cpython - Rust <-> Python bindings
pybind11 - Seamless operability between C++11 and Python
RustPython - A Python Interpreter written in Rust
milksnake - A setuptools/wheel/cffi extension to embed a binary data in wheels
bincode - A binary encoder / decoder implementation in Rust.
polars - Dataframes powered by a multithreaded, vectorized query engine, written in Rust
uniffi-rs - a multi-language bindings generator for rust
wasmtime - A fast and secure runtime for WebAssembly
egui - egui: an easy-to-use immediate mode GUI in Rust that runs on both web and native
py2many - Transpiler of Python to many other languages
rust-numpy - PyO3-based Rust bindings of the NumPy C-API
napi-rs - A framework for building compiled Node.js add-ons in Rust via Node-API