meltano
grouparoo
meltano | grouparoo | |
---|---|---|
9 | 27 | |
1,601 | 607 | |
2.7% | - | |
9.8 | 9.9 | |
1 day ago | about 2 years ago | |
Python | JavaScript | |
MIT License | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
meltano
-
meltano VS cloudquery - a user suggested alternative
2 projects | 2 Jun 2023
-
Show HN: Meltano Cloud (Gitlab spinout) – Managed infra for open source ELT
- https://github.com/meltano/meltano
We'd love to hear what you think of Meltano (Cloud). If you join the Beta, you get 100 free credits (200 hourly or 100 daily runs) and a 20% discount on the pricing at GA (June 27). The first 100 to sign up get 1,000 credits -- that's 83 days of hourly runs or 3 years of dailies!
The team and I will be checking in here throughout the day, so don't hesitate to ask questions! If we don't get to you, feel free to join 3,500+ Meltano fans on https://meltano.com/slack and we'll chat there!
-
Show HN: Sync’ing data to your customer’s Google Sheets
Meltano[0] might be of interest to you. Easy way to move data that should be very familiar for software engineers. If a connector doesn't exist our SDK makes it easy to build it.
[0] https://github.com/meltano/meltano
(disclaimer - I work at Meltano)
-
Meltano can now run any Airbyte source connector thanks to a community contribution
We currently don't do any process optimization on a per-stream basis when doing an extract. We have seen folks in the community running each tap separately for each stream which can speed it up. We've got an issue around this (Melturbo).
-
What is data integration?
Meltano
-
PostgreSQL to DuckDB - There and Quack Again
I built my data pipeline to Extract some data from websites and CSV files, Load it into my database, and Transform it into a reporting-ready schema. I used Python and Pandas to extract and load some of the data and Meltano to load some additional supporting data. All of that data went into a PostgreSQL database hosted in the cloud on Azure where I then used dbt to create data models in the database optimized for reporting. Finally, I use Metabase to visualize the data. (whew! that's a lot of moving parts!)
-
What should be the main point of a personal project?
I'm learning https://meltano.com/ right now, so am building custom Taps, mostly for fun. I'm enjoying it. I'm pulling in a variety of data from https://www.geonames.org/ and Canadian weather/climate data into BigQuery
-
What ETL tool you use with Postgres ?
https://meltano.com/ is ELT but I like it
- Airbyte vs Meltano community support
grouparoo
-
Reverse ETL recommendations?
Reverse ETL is on AirByte's roadmap under the "Future / Not prioritized" section. I wanted to use Grouparoo as a short term solution, but the repo was archived and I think they stopped taking new cloud customers (unknown if this is wrong/outdated).
-
Reference Data Stack for Data-Driven Startups
There are other tools that we will have to adopt in the future but haven’t yet due to lack of necessity. Specifically, one category that is popular in modern data stacks is Reverse ETL (Hightouch, Census, or Grouparoo). We currently don’t have a usecase for piping data back into 3rd party tools but it will definitely come up in the future.
-
Data pipeline suggestions
Reverse ETL: Grouparoo, Castled
-
Is Reverse ETL a new product or a new ETL/ELT feature?
Grouparoo, the open source Reverse ETL tool we are building, does all of these things. https://www.grouparoo.com
-
Where can I find free data engineering ( big data) projects online?
Ingestion / ETL: Airbyte, Singer, Jitsu Transformation: dbt Orchestration: Airflow, Dagster Testing: GreatExpectations Observability: Monosi Reverse ETL: Grouparoo, Castled Visualization: Lightdash, Superset
- Invite your company
-
Ask HN: Who is hiring? (December 2021)
Grouparoo | Remote (US) | Remote-OK | https://www.grouparoo.com
Grouparoo is a venture-backed software company building open source data tools that make data reliable, accessible, and actionable. We’re empowering teams to make great customer experiences, driven by data. While engineering teams have gotten good at storing and generating data about their customers, it’s rare that this data is used to its full potential in external applications. Grouparoo makes these integrations easy by providing a framework for defining your customer data and reliably syncing it to external tools.
To learn more about who we are, our engineering culture, and whether this is the right place for you, read our Key Values profile: https://www.keyvalues.com/grouparoo
Here are our open roles:
- Senior Backend / Lead Engineer: https://jobs.lever.co/grouparoo/6ba485d1-a5a4-41f0-9fa5-920a...
- Developer Advocate: https://jobs.lever.co/grouparoo/5e1531b4-7ec8-4c10-8e52-fc23...
Tech Stack: TypeScript / Javascript / Node.js, ActionHero, React + Next.js, Postgres & Redis, and whole lot of third-party APIs!
-
Launch HN: Hightouch (YC S19) – Sync data from data warehouses to SaaS tools
Congrats on the launch! Hightouch looks great and this need is real. Things seem to be going well, so I don't think I'm taking too much away by mentioning that we have been been working on Grouparoo, an open source alternative that solves similar pain points.
A few differences: git developer workflow focused (branches, CI, PRs, etc), ability to self host, segmentation in destinations (tagging people in mailchimp based on rules, for example)
https://www.grouparoo.com
-
Reverse ETL
We are building Grouparoo. Obviously, it's a biased sample but we are seeing a few trends at play.
-
What software or coding tools are you trying to get your company to invest in?
Has anyone heard or used of reverse etl or Hightouch or open sourced Grouparoo?
What are some alternatives?
Prefect - The easiest way to build, run, and monitor data pipelines at scale.
rotki - A portfolio tracking, analytics, accounting and management application that protects your privacy
nifi - Apache NiFi
airbyte - The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
pipelinewise - Data Pipeline Framework using the singer.io spec
TileDB - The Universal Storage Engine
pipelinewise-tap-mssql - Pipelinewise tap for Microsoft SQL Server
meltano
streamlit - Streamlit — A faster way to build and share data apps.
spark-rapids - Spark RAPIDS plugin - accelerate Apache Spark with GPUs
PostHog - 🦔 PostHog provides open-source product analytics, session recording, feature flagging and A/B testing that you can self-host.