meltano VS spark-rapids

- https://github.com/meltano/meltano

We'd love to hear what you think of Meltano (Cloud). If you join the Beta, you get 100 free credits (200 hourly or 100 daily runs) and a 20% discount on the pricing at GA (June 27). The first 100 to sign up get 1,000 credits -- that's 83 days of hourly runs or 3 years of dailies!

The team and I will be checking in here throughout the day, so don't hesitate to ask questions! If we don't get to you, feel free to join 3,500+ Meltano fans on https://meltano.com/slack and we'll chat there!

Meltano[0] might be of interest to you. Easy way to move data that should be very familiar for software engineers. If a connector doesn't exist our SDK makes it easy to build it.

[0] https://github.com/meltano/meltano

(disclaimer - I work at Meltano)

We currently don't do any process optimization on a per-stream basis when doing an extract. We have seen folks in the community running each tap separately for each stream which can speed it up. We've got an issue around this (Melturbo).

Meltano

I built my data pipeline to Extract some data from websites and CSV files, Load it into my database, and Transform it into a reporting-ready schema. I used Python and Pandas to extract and load some of the data and Meltano to load some additional supporting data. All of that data went into a PostgreSQL database hosted in the cloud on Azure where I then used dbt to create data models in the database optimized for reporting. Finally, I use Metabase to visualize the data. (whew! that's a lot of moving parts!)

I'm learning https://meltano.com/ right now, so am building custom Taps, mostly for fun. I'm enjoying it. I'm pulling in a variety of data from https://www.geonames.org/ and Canadian weather/climate data into BigQuery

https://meltano.com/ is ELT but I like it

His newer project, Ballista, was also donated to Apache Arrow. I hope to get the Rust skills to collaborate with him on open source work someday too. He's also doing really cool work on spark-rapids FYI.

The spark-rapids project allows one to run multi-GPU ETL workloads on a Spark cluster. https://github.com/NVIDIA/spark-rapids In such a setup, the GPU nodes are part of the Spark cluster. Multi-GPU nodes are viable, although an executor is currently limited to a single GPU.

So, in my day job at NVIDIA, I work on the RAPIDS Accelerator for Apache Spark, which is an open-source plugin that provides GPU-acceleration for ETL workloads, leveraging the RAPIDS cuDF GPU DataFrame library.