|about 2 months ago||6 days ago|
|Mozilla Public License 2.0||Apache License 2.0|
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
Will Rust-based data frame library Polars dethrone Pandas? We evaluate on 1M+ Stack Overflow questions
9 projects | reddit.com/r/rust | 25 May 2022
We didn't conduct our own benchmarks for this post, but in this comparison from ~1 year ago, Polars emerged as the fastest https://h2oai.github.io/db-benchmark/
Amazon Redshift Re-Invented
1 project | news.ycombinator.com | 24 May 2022
DuckDB's is in constant development but doesn't yet have native cross-version export/import feature (since its developers claim DuckDB hasn't reached maturity to stabilise its on-disk formats just yet).
I also keep an eye on https://h2oai.github.io/db-benchmark/ Pola.rs and DataFusion sound the most exciting.
It also remains to be seen how DataBrick's delta.io develops (might come in handy for much much larger datasets).
C++ for data analysis
2 projects | reddit.com/r/cpp | 18 May 2022
Fast Lane to Learning R
4 projects | news.ycombinator.com | 14 May 2022
I strongly recommend data.table R. Tidyverse is an improvement on base R, no question. Data.table has less intuitive syntax and can be harder to learn, but is lightning fast and memory efficient. If you're working with more than 1M rows, you should be using data.table.
Here are some benchmarks: https://h2oai.github.io/db-benchmark/
Friendlier SQL with DuckDB
8 projects | news.ycombinator.com | 12 May 2022
Hi, good to hear that you guys care about testing. One thing apart from the Github issues that led me to believe it might not be super stable yet was the benchmark results on https://h2oai.github.io/db-benchmark/ which make it look like it couldn't handle the 50GB case due to a out of memory error. I see that the benchmark and the used versions are about a year old so maybe things changed a lot since then. Can you chime in regarding the current story of running bigger DBs like 1TB on a machine with just 32GB or so RAM? Especially regardung data mutations and DDL queries. Thanks!
I used a new dataframe library (polars) to wrangle 300M prices and discover some of the most expensive hospitals in America. Code/notebook in article
2 projects | reddit.com/r/Python | 9 May 2022
Per these benchmarks it appears Polars is an order of magnitude more performant and it's lazy and Rust is just kinda sexy.
Benchmarking for loops vs apply and others
2 projects | reddit.com/r/rstats | 1 May 2022
This is a much more comprehensive set of benchmarks: https://h2oai.github.io/db-benchmark/
Why is R viewed badly over here? Also, as a college student, should I prioritize Python instead?
1 project | reddit.com/r/datascience | 29 Apr 2022
Its not like pandas is faster than tidyverse either on all the bechmarks, and data.table is faster than both. https://h2oai.github.io/db-benchmark/
Resources for data cleaning
2 projects | reddit.com/r/rstats | 13 Apr 2022
Language isn't really important here; what's important is tooling, and R definitely has the tooling. I would look at this benchmark reference for database-like operations, and you'll see that data.table (a very fast and memory-efficient R package) consistently ranks as one of the fastest tools out there that can also support a wide range of memory loads.
The fastest tool for querying large JSON files is written in Python (benchmark)
16 projects | news.ycombinator.com | 12 Apr 2022
Any project ideas that need a BIG Optimization effort
1 project | reddit.com/r/simd | 26 May 2022
How to use Spark and Pandas to prepare big data
3 projects | dev.to | 10 May 2022
Pandas user-defined function (UDF) is built on top of Apache Arrow. Pandas UDF improves data performance by allowing developers to scale their workloads and leverage Panda’s APIs in Apache Spark. Pandas UDF works with Pandas APIs inside the function, and works with Apache Arrow to exchange data.
Spice.ai v0.6-alpha is now available!
1 project | reddit.com/r/spiceai | 21 Apr 2022
Building upon the Apache Arrow support in v0.6-alpha, Spice.ai now includes new Apache Arrow data processor and Apache Arrow Flight data connector components! Together, these create a high-performance bulk-data transport directly into the Spice.ai ML engine. Coupled with big data systems from the Apache Arrow ecosystem like Hive, Drill, Spark, Snowflake, and BigQuery, it's now easier than ever to combine big data with Spice.ai.
Arrowdantic 0.1.0 released
3 projects | reddit.com/r/Python | 16 Apr 2022
Arrowdantic is a small Python library backed by a mature Rust implementation of Apache Arrow that can interoperate with * Parquet * Apache Arrow and * ODBC (databases).
Introducing Spice.xyz - Data and AI infrastructure for web3
1 project | reddit.com/r/ethfinance | 14 Apr 2022
🔥 Some cool things for eth/finance. We have per-block pool reserve data for Uniswap and Sushiswap and a Python SDK which lets you get data into Pandas, NumPy in 4 lines of code so you can use all the Python ecosystem of finance libraries you are used to. It uses Apache Arrow as the transport, so much faster than JSON. Here's an example Kaggle notebook: https://www.kaggle.com/code/spiceluke/spice-xyz-ethereum-blocks
C++ Jobs - Q2 2022
4 projects | reddit.com/r/cpp | 3 Apr 2022
Technologies: Apache Arrow, Flatbuffers, C++ Actor Framework, Linux, Docker, Kubernetes, Wireguard
What are the differences between feather and parquet?
1 project | reddit.com/r/codehunter | 1 Apr 2022
Both are columnar (disk-)storage formats for use in data analysis systems. Both are integrated within Apache Arrow (pyarrow package for python) and aredesigned to correspond with Arrow as a columnar in-memory analytics layer.
Intro to Apache Arrow
1 project | dev.to | 27 Feb 2022
More information about Apache Arrow can be found at https://arrow.apache.org/ Leave a comment if you have any questions or feedback.
Apache Arrow Feature Parity Timeline?
2 projects | reddit.com/r/rust | 21 Feb 2022
Apache Arrow Feature Matrix
Apache Arrow Flight SQL: Accelerating Database Access
5 projects | news.ycombinator.com | 16 Feb 2022
I'm not tuned into Arrow all that much. I've some of the about and stuff, but the code examples (to my eye) look really complex and complicated. 
Could someone point me to a more glossy "arrow flight sql for dummies" examples? What I'm gleaning from this (or am I wrong?) is you could use a JDBC driver + arrow jdbc client and write... SQL? Or is it something a lot different?
Is this the sort of thing where you could just add a plugin to postgres and be arrowified or something?
What are some alternatives?
h5py - HDF5 for Python -- The h5py package is a Pythonic interface to the HDF5 binary data format.
Apache Spark - Apache Spark - A unified analytics engine for large-scale data processing
polars - Fast multi-threaded DataFrame library in Rust | Python | Node.js
arrow-datafusion - Apache Arrow DataFusion SQL Query Engine
duckdb_and_r - My thoughts and examples on DuckDB and R
Airflow - Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
ta-lib - Python wrapper for TA-Lib (http://ta-lib.org/).
arquero - Query processing and transformation of array-backed data tables.
Apache HBase - Apache HBase
ClickHouse - ClickHouse® is a free analytics DBMS for big data
spark-rapids - Spark RAPIDS plugin - accelerate Apache Spark with GPUs