chdb
Meerschaum
chdb | Meerschaum | |
---|---|---|
18 | 17 | |
1,726 | 121 | |
3.4% | - | |
9.5 | 7.8 | |
about 14 hours ago | 4 months ago | |
C++ | Python | |
Apache License 2.0 | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
chdb
- FLaNK Stack Weekly 06 Nov 2023
-
DB Pilot: Query Postgres, files, S3 and more โ all at once, from your laptop
Hey HN, creator of DB Pilot here.
I first announced DB Pilot on HN back in April: https://news.ycombinator.com/item?id=35761979.
Since then a lot has improved: More databases are supported, most of the product can now be used for free, and most importantly:
The app now comes with an analytics workspace powered by an embedded ClickHouse instance, running locally on your machine. This allows you to query local files, files on S3, PostgreSQL, SQLite & more - and all of those at once.
Embedding ClickHouse was possible thanks to chDB (https://github.com/chdb-io/chdb). A recent discussion on HN about it: https://news.ycombinator.com/item?id=37985005
- ChDB: Embedded OLAP SQL Engine Powered by ClickHouse
-
DuckDB 0.9.0
I recommend using ClickHouse instead of DuckDB.
It has been around since 2016, and it covers and extends the feature set of DuckDB with a huge margin. Worth noting that it never has breaking changes in its table format MergeTree.
I'm tracking the progress of DuckDB and see that it is modeled after ClickHouse, but does not approach it in terms of feature completeness, stability, or performance.
The closest to DuckDB option is to use its self-contained version, clickhouse-local: https://clickhouse.com/blog/extracting-converting-querying-l... or an embedded version, chdb: https://github.com/chdb-io/chdb
-
Is ClickHouse Moving Away from Open Source?
Different beasts, but if by any chance you love ClickHouse already and just want to run OLAP queries in-process, there's chdb: https://github.com/chdb-io/chdb
- ChDB: An Embedded OLAP SQL Engine Powered by ClickHouse
-
PRQL, Pipelined Relational Query Language
> Can you embed it in Python as a library?
https://github.com/chdb-io/chdb
pip install chdb
-
Using SQL inside Python pipelines with Duckdb, Glaredb (and others?)
New kid on the block that I prefer over DuckDB is CHDB (https://github.com/chdb-io/chdb). Embedded ClickHouse, so once you out grow your laptop you can simply move to an actual OLAP that's Open-source.
- ClickHouse-local and chdb performance issue on clickbench Q.23 Q28
Meerschaum
-
Using SQL inside Python pipelines with Duckdb, Glaredb (and others?)
This sounds like a great use case for Meerschaum. You can organize your scripts into plugins and build out incremental transformations in SQL. We use Meerschaum Compose for client integrations and ETL in a similar workflow to yours.
-
Found a great new open source ELT Library - any pointers?
My company has been using a lot of PySpark, but we're working with not-large data (<1TB/source/day) so Spark can be a bit of overkill sometimes and I've been looking for a light-weight replacement. I think I found a replacement that fits all our needs called Meerschaum but I don't see a lot of other DEs talking about it.
-
Iโm struggling with how to ask for help with my task.
Do the tables have something like a datetime or integer index column? At my job, we use the ETL Python package Meerschaum to sync our tables, and for large ones, we split the sync into chunks with --begin (inclusive) and --end (exclusive).
-
For those of you who were self taught, what was your path into data engineering
I worked as the first data engineer for a student internship for two years, during which I rewrote the system several times until I had a time-series ETL system that fit their needs perfectly. After leaving, I took what I learned and started the ETL package Meerschaum, and after a few consulting contracts to deploy Meerschaum, I landed a DE job to manage Meerschaum deployments internally. A bit unconventional but worked out as I had hoped.
-
Wanted to share my open source incremental ETL framework: Meerschaum
There's a whole lot more that you can do with the framework, but this post is getting kinda long. Please check out the project homepage for more details, and I'd really love know what y'all think! Can you see a use case for the framework in your stack?
-
Python ETL - Jupyter/Pandas/Postgresql(DW) - Project Structure and Scripting
I'm the author of the ETL framework Meerschaum which is meant for this exact purpose. You can build an ETL pipeline in a few lines of Python, e.g. here's a quick video. Check out the Getting Started guide and the docs on writing your first plugin to get your data flowing!
-
Tools that allow you to use scripts to build/maintain data pipeline
You can prototype some scripts with a tool called Meerschaum that I built for this kind of purpose. Once you're ready to deploy your prototype, you could refactor it for something more suited for enterprise like Airflow.
- Meerschaum - Data Visualization Pipelines in Minutes
What are some alternatives?
risingwave - SQL stream processing, analytics, and management. PostgreSQL simplicity, unrivaled performance, and seamless elasticity. ๐ 10x more productive. ๐ 10x more cost-efficient.
Prefect - The easiest way to build, run, and monitor data pipelines at scale.
openvino_notebooks - ๐ Jupyter notebook tutorials for OpenVINOโข
glaredb - GlareDB: An analytics DBMS for distributed data
duckdb-wasm - WebAssembly version of DuckDB
gspreadsheet_fdw - Multicorn-based PostgreSQL foreign data wrapper for Google Spreadsheets
chdb-cli - Simple CLI / REPL for chdb made in Python
sqlite_blaster_python - A library for creating huge Sqlite indexes at breakneck speeds
techslamneggs - The code for my May 3, 2023 workshop at Greenville's Tech Slam 'N Eggs!
syncx - This Meerschaum plugin implements experimental syncing methods.