Apache Arrow 4.0.0 Release

• Apache Arrow

  75 13,480 10.0 C++

  Apache Arrow is a multi-language toolbox for accelerated data interchange and in-memory processing

  

Great to see Ballista in arrow https://github.com/apache/arrow/pull/9723

• arrow-datafusion

  55 4,924 9.9 Rust

  Apache DataFusion SQL Query Engine

  

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• arrow-rs

  16 2,176 9.8 Rust

  Official Rust implementation of Apache Arrow

  

• PyCall.jl

  28 1,438 6.1 Julia

  Package to call Python functions from the Julia language

  

I actually don't know very much about how it works (which is a testament to the usability of the libraries!)

I use PyCall to use Python from Julia, and PyJulia to do the reverse. PyCall & PyJulia have functionality to easily share arrays. PyCall is pretty seamless, PyJulia is a bit more work but still solid.

To share a DataFrame, I convert it to Arrow, get a bytearray representation of that, use PyCall/PyJulia to access the array from a different process, and reinterpret it as Arrow data within that process.

PyCall: https://github.com/JuliaPy/PyCall.jl

PyJulia: https://pyjulia.readthedocs.io/en/latest/

Suggest a related project