ibis
Apache Impala
Our great sponsors
ibis | Apache Impala | |
---|---|---|
23 | 1 | |
4,074 | 1,079 | |
7.9% | 1.5% | |
10.0 | 9.7 | |
6 days ago | 3 days ago | |
Python | C++ | |
Apache License 2.0 | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
ibis
-
Show HN: Hashquery, a Python library for defining reusable analysis
I really don't understand the appeal of dbt vs a proper programming language. The templating approach leads to massive spaghetti. I look forward to trying out something like Ibis [0]
0: https://ibis-project.org/
-
This Week In Python
ibis – portable Python dataframe library
- Ibis: The portable Python dataframe library
- FLaNK Stack 26 February 2024
-
Quarto
The main benefit is that you get a Python (or R, Julia or Rust) interpreter. So you can evaluate code. A good example of the value of this is the Ibis docs which use Quarto: https://ibis-project.org/
-
Polars – A bird's eye view of Polars
Ive found polars quite intuitive, though for python, I lean more towards [ibis](https://ibis-project.org/). The interface is nearly identical, but ibis has the benefit if building sql queries before pulling any actual data (like dbplyr) — whereas polars requires the data to be in-memory (at least for rdb’s, though correct me if Im wrong)
this to me seems like a good argument for only using ibis, but Im happy to be convinced otherwise
- Ibis – Universal Interface for Data Wrangling
-
Vanna.ai: Chat with your SQL database
Please add Ibis Birdbrain https://ibis-project.github.io/ibis-birdbrain/ to the list. Birdbrain is an AI-powered data bot, built on Ibis and Marvin, supporting more than 18 database backends.
See https://github.com/ibis-project/ibis and https://ibis-project.org for more details.
- Ibis
Apache Impala
-
Word-Aligned Bloom Filters
> whether this would really work out in most workloads
> just because it keeps the cache-lines hotter and less likely to be evicted.
Okay, so keeping cache for a bloom filter problem is real - but the real force evicting memory out of the cache line is the next row-group you read + all the other stuff you have to do when you implement this in a database product.
So the two things I work with, Apache Hive and Apache Impala switched to a blocked bloom filter at different points in time.
Hive BloomKFilter - https://github.com/apache/hive/blob/master/storage-api/src/j...
Impala/Kudu one - https://github.com/apache/impala/blob/master/be/src/kudu/uti...
The C++ one also has an AVX specialization, while the Java one relies on the JVM to do it (not always) - https://github.com/apache/impala/blob/master/be/src/kudu/uti...
We ran a lot of trivial benchmarks and several benchmarks where the shuffle-join (not sort-merge, this is just a partitioned hash join) generates a bloom filter (a semijoin) before sending rows out and the 1-cache line version won out when the bloom filter went slightly over the 1 Million + 5% rate [1].
The regular bloom filter went from (38ns -> 108ns for 1k -> 1m items), while the BloomK stuck at (27ns) despite making room for a million times more items in the bloom. The bloom-1 (which is the 64bit version) underperformed on accuracy (was ~2x faster at 16ns per op, but worse at filtering out items).
[1] - https://github.com/prasanthj/bloomfilter/tree/master/benchma...
What are some alternatives?
snowflake-connector-python - Snowflake Connector for Python
seed_rl - SEED RL: Scalable and Efficient Deep-RL with Accelerated Central Inference. Implements IMPALA and R2D2 algorithms in TF2 with SEED's architecture.
PySpark-Boilerplate - A boilerplate for writing PySpark Jobs
machin - Reinforcement learning library(framework) designed for PyTorch, implements DQN, DDPG, A2C, PPO, SAC, MADDPG, A3C, APEX, IMPALA ...
pangres - SQL upsert using pandas DataFrames for PostgreSQL, SQlite and MySQL with extra features
bloomfilter - BloomFilter implementation in Java that uses Murmur3 for fast hashing
sqlite_scanner - DuckDB extension to read and write to SQLite databases
Apache Hive - Apache Hive
katacoda
simdjson - Parsing gigabytes of JSON per second : used by Facebook/Meta Velox, the Node.js runtime, ClickHouse, WatermelonDB, Apache Doris, Milvus, StarRocks
nodejs-polars - nodejs front-end of polars
bitcoinbook - Mastering Bitcoin 3rd Edition - Programming the Open Blockchain