r-polars
datafusion-ballista
| r-polars | datafusion-ballista | |
|---|---|---|
| 5 | 12 | |
| 389 | 1,288 | |
| 1.8% | 4.6% | |
| 9.8 | 8.2 | |
| 7 days ago | 5 days ago | |
| R | Rust | |
| GNU General Public License v3.0 or later | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
r-polars
- Polars R Package
- Polars
-
Transitioning from R to Python
I'm an R/python user and just wanted to let you know that polars also exist in R here
-
Pandas 2.0 Released
I am not sure. The R support of polars is entirely picked up by the R community and @sorhawell in particular. You can get certainly more information on that repo: https://github.com/pola-rs/r-polars
-
Why Polars uses less memory than Pandas
Recently there is also a work in progress implementation of Polars rust bindings to R: https://github.com/pola-rs/r-polars
datafusion-ballista
-
Polars
Not super on topic because this is all immature and not integrated with one another yet, but there is a scaled-out rust data-frames-on-arrow implementation called ballista that could maybe? form the backend of a polars scale out approach: https://github.com/apache/arrow-ballista
-
Rust vs. Go in 2023
> Is Rust's compile-time GC about something other than performance somehow?
AFAIK, memory safety and language features as RAII is also available in C++, for instance. About the reasons for slow compilation, take a look at https://www.reddit.com/r/rust/comments/xna9mb/why_are_rust_p...
Not having a GC is also about not having a runtime as you mention (e.g. nice for creating Python extensions and embedded systems programming) and also more runtime deterministic performance: on that, if I'm not mistaken that was the reason for Discourse switching to Rust and also, e.g.: "the choice of Rust as the main execution language avoids the overhead of GC pauses and results in deterministic processing times" https://github.com/apache/arrow-ballista/blob/main/README.md
- Ballista (Rust) vs Apache Spark. A Tale of Woe.
-
Evolution and Trends of Data Engineering 2022/23
Ballista (Arrow-Rust), which is largely inspired by Apache Spark, there are some interesting differences.
-
Data Engineering with Rust
https://github.com/jorgecarleitao/arrow2 https://github.com/apache/arrow-datafusion https://github.com/apache/arrow-ballista https://github.com/pola-rs/polars https://github.com/duckdb/duckdb
- Any job processing framework like Spark but in Rust?
-
Is Apache Arrow DataFusion and Ballista the future of big data engineering/science?
Source: https://github.com/apache/arrow-ballista
-
Pure Python Distributed SQL Engine
Can you explain how this might differ from something like https://github.com/apache/arrow-ballista
I've seen several variants of "next-gen" spark, but nowhere have I really seen the different tradeoffs/advantages/disadvantages between them.
- Scala or Rust? which one will rule in future?
-
Welcome to Comprehensive Rust
Rust has amazing integration with Python through PyO3 [1] so see it like a safe alternative for high performance calculations. The ecosystem itself is starting to come together exciting projects like Polars [2] (Pandas alternative), nalgebra [3], Datafusion [4] and Ballista [5]
[1] https://github.com/PyO3/pyo3
[2] https://github.com/pola-rs/polars/
[3] https://docs.rs/nalgebra/latest/nalgebra/
[4] https://github.com/apache/arrow-datafusion
[5] https://github.com/apache/arrow-ballista
What are some alternatives?
polars - Dataframes powered by a multithreaded, vectorized query engine, written in Rust
duckdb - DuckDB is an in-process SQL OLAP Database Management System
lance - Modern columnar data format for ML and LLMs implemented in Rust. Convert from parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, with more integrations coming..
seafowl - Analytical database for data-driven Web applications 🪶
connector-x - Fastest library to load data from DB to DataFrames in Rust and Python
opteryx - 🦖 A SQL-on-everything Query Engine you can execute over multiple databases and file formats. Query your data, where it lives.
sqlglot - Python SQL Parser and Transpiler
datafusion - Apache DataFusion SQL Query Engine
comprehensive-rust - This is the Rust course used by the Android team at Google. It provides you the material to quickly teach Rust.
ballista - Distributed compute platform implemented in Rust, and powered by Apache Arrow.
arrow2 - Transmute-free Rust library to work with the Arrow format
self-limiters - Async distributed rate limiters for Python