meltano VS pipelinewise-tap-mssql

- https://github.com/meltano/meltano

We'd love to hear what you think of Meltano (Cloud). If you join the Beta, you get 100 free credits (200 hourly or 100 daily runs) and a 20% discount on the pricing at GA (June 27). The first 100 to sign up get 1,000 credits -- that's 83 days of hourly runs or 3 years of dailies!

The team and I will be checking in here throughout the day, so don't hesitate to ask questions! If we don't get to you, feel free to join 3,500+ Meltano fans on https://meltano.com/slack and we'll chat there!

Meltano[0] might be of interest to you. Easy way to move data that should be very familiar for software engineers. If a connector doesn't exist our SDK makes it easy to build it.

[0] https://github.com/meltano/meltano

(disclaimer - I work at Meltano)

We currently don't do any process optimization on a per-stream basis when doing an extract. We have seen folks in the community running each tap separately for each stream which can speed it up. We've got an issue around this (Melturbo).

Meltano

I built my data pipeline to Extract some data from websites and CSV files, Load it into my database, and Transform it into a reporting-ready schema. I used Python and Pandas to extract and load some of the data and Meltano to load some additional supporting data. All of that data went into a PostgreSQL database hosted in the cloud on Azure where I then used dbt to create data models in the database optimized for reporting. Finally, I use Metabase to visualize the data. (whew! that's a lot of moving parts!)

I'm learning https://meltano.com/ right now, so am building custom Taps, mostly for fun. I'm enjoying it. I'm pulling in a variety of data from https://www.geonames.org/ and Canadian weather/climate data into BigQuery

https://meltano.com/ is ELT but I like it

You can use a singer tap for this. https://github.com/wintersrd/pipelinewise-tap-mssql

This is something I've been looking at as well. I've recently found a python-based implementation here driven by Github user `wintersrd` which might be a good candidate for adoption by meltano as that tap reaches maturity.