Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality. Learn more →
Getting-started Alternatives
Similar projects and alternatives to getting-started
-
Mage
🧙 The modern replacement for Airflow. Mage is an open-source data pipeline tool for transforming and integrating data. https://github.com/mage-ai/mage-ai
-
airbyte
The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
AWS Data Wrangler
pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
astro
Discontinued Astro SDK allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow. [Moved to: https://github.com/astronomer/astro-sdk] (by astro-projects)
-
astro-sdk
Astro SDK allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.
-
datajob
Build and deploy a serverless data pipeline on AWS with no effort.
-
typhoon-orchestrator
Create elegant data pipelines and deploy to AWS Lambda or Airflow
-
data-pipelines-with-apache-airflow
Code for Data Pipelines with Apache Airflow
-
windmill
Open-source developer platform to turn scripts into workflows and UIs. Fastest workflow engine (5x vs Airflow). Open-source alternative to Airplane and Retool.
-
Moto
A library that allows you to easily mock out tests based on AWS infrastructure.
-
-
-
-
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
getting-started reviews and mentions
-
Data sources episode 2: AWS S3 to Postgres Data Sync using Singer
Singer is an open-source framework for data ingestion, which provides a standardized way to move data between various data sources and destinations (such as databases, APIs, and data warehouses). Singer offers a modular approach to data extraction and loading by leveraging two main components: Taps (data extractors) and Targets (data loaders). This design makes it an attractive option for data ingestion for several reasons:
- Design patter for Python ETL
-
Launch HN: Patterns (YC S21) – A much faster way to build and deploy data apps
Thanks for chipping in.
I’ve been leaning towards this direction. I think I/O is the biggest part that in the case of plain code steps still needs fixing. Input being data/stream and parameterization/config and output being some sort of typed data/stream.
My “let’s not reinvent the wheel” alarm is going of when I write that though. Examples that come to mind are text based (Unix / https://scale.com/blog/text-universal-interface) but also the Singer tap protocol (https://github.com/singer-io/getting-started/blob/master/doc...). And config obviously having many standard forms like ini, yaml, json, environment key value pairs and more.
At the same time, text feels horribly inefficient as encoding for some of the data objects being passed around in these flows. More specialized and optimized binary formats come to mind (Arrow, HDF5, Protobuf).
Plenty of directions to explore, each with their own advantages and disadvantages. I wonder which direction is favored by users of tools like ours. Will be good to poll (do they even care?).
PS Windmill looks equally impressive! Nice job
-
After Airflow. Where next for DE?
Mage uses the Singer Spec (https://github.com/singer-io/getting-started/blob/master/docs/SPEC.md), the data engineer community standard for building data integrations. This was created by Stitch and is widely adopted.
-
Basic data engineering question.
I like the Singer Protocol, and the various tools that use it. These include meltano, airbyte, stitch, pipelinewise, and a few others
-
Looking to build a database for BI reports
This is good advice and I think Airbyte created a great product here. I tried singer.io and pipewise but Airbyte is much better in my opinion and I love the UI.
-
Recommendation for approach for populating and refreshing new data lake
If you can't use Stitch, you can still develop with the singer.io library which is the basis for Stitch.
Suspect my question should have been regarding FREE systems, rather than BUYING a system. Sounds like singer.io will do what I need.
-
Datajob: Build and deploy a serverless data pipeline on AWS with no effort.
If i'm not mistaken, singer.io are scripts that move data around. Datajob can help you deploy and orchestrate these singer.io scripts to AWS Glue.
- Meltano ELT: Open-Source DataOps for the DevOps Era
-
A note from our sponsor - InfluxDB
www.influxdata.com | 28 Mar 2024
Stats
The primary programming language of getting-started is Makefile.
Popular Comparisons
- getting-started VS airbyte
- getting-started VS AWS Data Wrangler
- getting-started VS meltano
- getting-started VS Mage
- getting-started VS tap-hubspot
- getting-started VS tap-spreadsheets-anywhere
- getting-started VS singer-sdk
- getting-started VS astro-sdk
- getting-started VS astro
- getting-started VS data-pipelines-with-apache-airflow