Apache Arrow Alternatives
Similar projects and alternatives to Apache Arrow
-
-
-
Scout APM
Scout APM: A developer's best friend. Try free for 14-days. Scout APM uses tracing logic that ties bottlenecks to source code so you know the exact line of code causing performance issues and can get back to building a great product faster.
-
h5py
HDF5 for Python -- The h5py package is a Pythonic interface to the HDF5 binary data format.
-
Apache Spark
Apache Spark - A unified analytics engine for large-scale data processing
-
-
-
-
-
-
-
per4m
Profiling and tracing information for Python using viztracer and perf, the GIL exposed.
-
perspective
A data visualization and analytics component, especially well-suited for large and/or streaming datasets.
-
viztracer
VizTracer is a low-overhead logging/debugging/profiling tool that can trace and visualize your python code execution.
-
arrow-datafusion
Apache Arrow DataFusion and Ballista query engines
-
docker
These are the official Dockerfiles for https://github.com/orgs/tripl-ai/packages (by tripl-ai)
-
Airflow
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
-
-
-
-
-
NOTE:
The number of mentions on this list indicates mentions on common posts plus user suggested alternatives.
Hence, a higher number means a better Apache Arrow alternative or higher similarity.
Reviews and mentions
Posts with mentions or reviews of Apache Arrow.
We have used some of these posts to build our list of alternatives
and similar projects. The last one was on 2021-12-04.
-
Any role that Rust could have in the Data world (Big Data, Data Science, Machine learning, etc.)?
Yes https://arrow.apache.org/
-
pigeon-rs: Open source email automation written in Rust
Connectorx is using arrow2 data format for fetching from a database. This data format is optimized for columnar data [1]:
-
Introducing tidypolars - a Python data frame package for R tidyverse users
I think having a basic understanding of pandas, given how broadly it's used, is beneficial. That being said, polars seems to be matching or beating data.table in performance, so I think it'd be very worth it to take it up. Wes McKinney, creator of pandas, has been quite vocal about architecture flaws of pandas -- which is why he's been working on the Arrow project. polars is based on Arrow, so in principle it's kinda like pandas 2.0 (adopting the changes that Wes proposed).
So the question is really - how is polars so fast? Polars is packed by Apache Arrow, which is a columnar memory format that is designed specifically for performance.
- Comparing SQLite, DuckDB and Arrow
-
The Data Engineer Roadmap 🗺
Apache Arrow
-
C++ Jobs - Q4 2021
Technologies: Apache Arrow, Flatbuffers, C++ Actor Framework, Linux, Docker, Kubernetes
-
How to use Spark and Pandas to prepare big data
Pandas user-defined function (UDF) is built on top of Apache Arrow. Pandas UDF improves data performance by allowing developers to scale their workloads and leverage Panda’s APIs in Apache Spark. Pandas UDF works with Pandas APIs inside the function, and works with Apache Arrow to exchange data.
-
Announcing arrow-odbc
arrow-odbc allows you to iterate over an ODBC data source as sequence of Apache Arrow record batches.
-
CuVec: Unifying Python/C++/CUDA memory
IIRC Apache Arrow [1] promised similar goal and it seems covers CUDA as well [2]. I wonder how these relates in the big picture. This one seems much simpler than arrow, which is probably a good thing in terms of the differentiation?
-
Recommendation for a Database for analysis
What you need for your use case is a column-oriented store. I recommend explore bcolz or apache arrow for a column file-based systems. These are very fast, support memory mapping, uses compression and SSD speed (and even CPU architecture, in case of arrow) optimally almost out of the box, and has good interfaces to Numpy and Pandas (in case you are using Python for final data consumption and analysis). The columnar structure makes it easy to add or delete a column easily (or even dynamically). If you need a more scalable (albeit at the cost of speed) solution, you can devise a schema over a regular columnar db or an nosql db - see arctic from Man group for an example.
-
Apache Arrow 4.0.0 Release
Come on, it's on the top of the front page https://arrow.apache.org/
Great to see Ballista in arrow https://github.com/apache/arrow/pull/9723
-
Introducing Cube Store: High concurrency and sub-second latency for any database
Cube Store is based on well-known open-source building blocks. Together with Parquet, Cube Store uses Apache Arrow for its efficient in-memory data structures and DataFusion as a query execution framework.
-
Using MongoDB with Pandas, NumPy, and Arrow
Originally, we intended to build a new API that worked exclusively with Pandas, however Pandas did not provide a stable C-API that we could use. Meanwhile, we sort of fell in love with Apache Arrow. The Apache Arrow project has a set of standards to address long standing inefficiencies in the processing and transport of large datasets in high-performance applications. Conversion of arrow tables to varies formats was simple and fast. Since Arrow is a language independent standard, our Arrow integration will make it easier for developers to move data from Mongodb into a wide variety of OLAP systems.
Stats
Basic Apache Arrow repo stats
26
8,759
9.9
3 days ago
apache/arrow is an open source project licensed under Apache License 2.0 which is an OSI approved license.
Popular Comparisons
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com
Are you hiring? Post a new remote job listing for free.