Meerschaum VS covid

This sounds like a great use case for Meerschaum. You can organize your scripts into plugins and build out incremental transformations in SQL. We use Meerschaum Compose for client integrations and ETL in a similar workflow to yours.

My company has been using a lot of PySpark, but we're working with not-large data (<1TB/source/day) so Spark can be a bit of overkill sometimes and I've been looking for a light-weight replacement. I think I found a replacement that fits all our needs called Meerschaum but I don't see a lot of other DEs talking about it.

Do the tables have something like a datetime or integer index column? At my job, we use the ETL Python package Meerschaum to sync our tables, and for large ones, we split the sync into chunks with --begin (inclusive) and --end (exclusive).

I worked as the first data engineer for a student internship for two years, during which I rewrote the system several times until I had a time-series ETL system that fit their needs perfectly. After leaving, I took what I learned and started the ETL package Meerschaum, and after a few consulting contracts to deploy Meerschaum, I landed a DE job to manage Meerschaum deployments internally. A bit unconventional but worked out as I had hoped.

There's a whole lot more that you can do with the framework, but this post is getting kinda long. Please check out the project homepage for more details, and I'd really love know what y'all think! Can you see a use case for the framework in your stack?

I'm the author of the ETL framework Meerschaum which is meant for this exact purpose. You can build an ETL pipeline in a few lines of Python, e.g. here's a quick video. Check out the Getting Started guide and the docs on writing your first plugin to get your data flowing!

You can prototype some scripts with a tool called Meerschaum that I built for this kind of purpose. Once you're ready to deploy your prototype, you could refactor it for something more suited for enterprise like Airflow.

Here is the parent GitHub repository with usage instructions. I've formatted the scripts as Meerschaum plugins, which are just Python modules with fetch() functions to return DataFrames. Below are my individual scraping scripts:

A lightweight alternative to Meltano is Meerschaum (note — I'm the author of Meerschaum). Its use case is built around fetching immutable time-series data as DataFrames — great for prototyping though contained to that niche. As an example, here's a plugin I wrote to fetch COVID data from states. It also has built-in support for materializing time series SQL queries between databases, no scripting required.