Meerschaum
duckdb
Meerschaum | duckdb | |
---|---|---|
17 | 52 | |
121 | 17,221 | |
- | 7.1% | |
6.7 | 10.0 | |
5 days ago | 4 days ago | |
Python | C++ | |
Apache License 2.0 | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
Meerschaum
-
Using SQL inside Python pipelines with Duckdb, Glaredb (and others?)
This sounds like a great use case for Meerschaum. You can organize your scripts into plugins and build out incremental transformations in SQL. We use Meerschaum Compose for client integrations and ETL in a similar workflow to yours.
-
Found a great new open source ELT Library - any pointers?
My company has been using a lot of PySpark, but we're working with not-large data (<1TB/source/day) so Spark can be a bit of overkill sometimes and I've been looking for a light-weight replacement. I think I found a replacement that fits all our needs called Meerschaum but I don't see a lot of other DEs talking about it.
-
I’m struggling with how to ask for help with my task.
Do the tables have something like a datetime or integer index column? At my job, we use the ETL Python package Meerschaum to sync our tables, and for large ones, we split the sync into chunks with --begin (inclusive) and --end (exclusive).
-
For those of you who were self taught, what was your path into data engineering
I worked as the first data engineer for a student internship for two years, during which I rewrote the system several times until I had a time-series ETL system that fit their needs perfectly. After leaving, I took what I learned and started the ETL package Meerschaum, and after a few consulting contracts to deploy Meerschaum, I landed a DE job to manage Meerschaum deployments internally. A bit unconventional but worked out as I had hoped.
-
Wanted to share my open source incremental ETL framework: Meerschaum
There's a whole lot more that you can do with the framework, but this post is getting kinda long. Please check out the project homepage for more details, and I'd really love know what y'all think! Can you see a use case for the framework in your stack?
-
Python ETL - Jupyter/Pandas/Postgresql(DW) - Project Structure and Scripting
I'm the author of the ETL framework Meerschaum which is meant for this exact purpose. You can build an ETL pipeline in a few lines of Python, e.g. here's a quick video. Check out the Getting Started guide and the docs on writing your first plugin to get your data flowing!
-
Tools that allow you to use scripts to build/maintain data pipeline
You can prototype some scripts with a tool called Meerschaum that I built for this kind of purpose. Once you're ready to deploy your prototype, you could refactor it for something more suited for enterprise like Airflow.
- Meerschaum - Data Visualization Pipelines in Minutes
duckdb
- 🪄 DuckDB sql hack : get things SORTED w/ constraint CHECK
- DuckDB: Move to push-based execution model (2021)
-
DuckDB performance improvements with the latest release
I'm not sure if the fix is reassuring or not: https://github.com/duckdb/duckdb/pull/9411/files
-
Building a Distributed Data Warehouse Without Data Lakes
It's an interesting question!
The problem is that the data is spread everywhere - no choice about that. So with that in mind, how do you query that data? Today, the idea is that you HAVE to put it into a central location. With tools like Bacalhau[1] and DuckDB [2], you no longer have to - a single query can be sharded amongst all your data - EFFECTIVELY giving you a lot of what you want from a data lake.
It's not a replacement, but if you can do a few of these items WITHOUT moving the data, you will be able to see really significant cost and time savings.
[1] https://github.com/bacalhau-project/bacalhau
[2] https://github.com/duckdb/duckdb
- DuckDB 0.9.0
-
Push or Pull, is this a question?
[4] Switch to Push-Based Execution Model by Mytherin · Pull Request #2393 · duckdb/duckdb (github.com)
-
Show HN: Hydra 1.0 – open-source column-oriented Postgres
it depends on your query obviously.
In general, I did very deep benchmarking of pg, clickhouse and duckdb, and I sure didn't make stupid mistakes like this: https://news.ycombinator.com/item?id=36990831
My dataset has 50B rows and 2tb of data, and I think columnar dbs are very overhiped and I chose pg because:
- pg performance is acceptable, maybe 2-3x times slower than clickhouse and duckdb on some queries if pg is configured correctly and run on compressed storage
- clickhouse and duckdb start falling apart very fast because they specialized on very narrow type of queries: https://github.com/ClickHouse/ClickHouse/issues/47520 https://github.com/ClickHouse/ClickHouse/issues/47521 https://github.com/duckdb/duckdb/discussions/6696
-
🦆 Effortless Data Quality w/duckdb on GitHub ♾️
This action installs duckdb with the version provided in input.
-
Using SQL inside Python pipelines with Duckdb, Glaredb (and others?)
Duckdb: https://github.com/duckdb/duckdb - seems pretty popular, been keeping an eye on this for close to a year now.
-
CSV or Parquet File Format
The Parquet-Go library is very complex, not yet success to use it. So I ask whether DuckDB can provide API https://github.com/duckdb/duckdb/issues/7776
What are some alternatives?
Prefect - The easiest way to build, run, and monitor data pipelines at scale.
ClickHouse - ClickHouse® is a free analytics DBMS for big data
glaredb - GlareDB: An analytics DBMS for distributed data
sqlite-worker - A simple, and persistent, SQLite database for Web and Workers.
chdb - chDB is an embedded OLAP SQL Engine 🚀 powered by ClickHouse
datasette - An open source multi-tool for exploring and publishing data
gspreadsheet_fdw - Multicorn-based PostgreSQL foreign data wrapper for Google Spreadsheets
octosql - OctoSQL is a query tool that allows you to join, analyse and transform data from multiple databases and file formats using SQL.
risingwave - SQL stream processing, analytics, and management. We decouple storage and compute to offer instant failover, dynamic scaling, speedy bootstrapping, and efficient joins.
metabase-clickhouse-driver - ClickHouse database driver for the Metabase business intelligence front-end
techslamneggs - The code for my May 3, 2023 workshop at Greenville's Tech Slam 'N Eggs!
datafusion - Apache DataFusion SQL Query Engine