getting-started

This repository is a getting started guide to Singer. (by singer-io)

Getting-started Alternatives

Similar projects and alternatives to getting-started

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a better getting-started alternative or higher similarity.

getting-started discussion

Log in or Post with

getting-started reviews and mentions

Posts with mentions or reviews of getting-started. We have used some of these posts to build our list of alternatives and similar projects. The last one was on 2023-05-04.
  • Why do companies still build data ingestion tooling instead of using a third-party tool like Airbyte?
    1 project | /r/dataengineering | 6 Dec 2023
    Coincidently, I saw a presentation today on a nice half-way-house solution: using embeddable Python libraries like Sling and dlt - both open-source. See https://www.youtube.com/watch?v=gAqOLgG2iYY There is also singer.io which is more of a protocol than a library, but can also be installed although it looks like it is a true community effort and not so well maintained.
  • Data sources episode 2: AWS S3 to Postgres Data Sync using Singer
    2 projects | dev.to | 4 May 2023
    Singer is an open-source framework for data ingestion, which provides a standardized way to move data between various data sources and destinations (such as databases, APIs, and data warehouses). Singer offers a modular approach to data extraction and loading by leveraging two main components: Taps (data extractors) and Targets (data loaders). This design makes it an attractive option for data ingestion for several reasons:
  • Design patter for Python ETL
    2 projects | /r/dataengineering | 2 Dec 2022
  • Launch HN: Patterns (YC S21) – A much faster way to build and deploy data apps
    6 projects | news.ycombinator.com | 30 Nov 2022
    Thanks for chipping in.

    I’ve been leaning towards this direction. I think I/O is the biggest part that in the case of plain code steps still needs fixing. Input being data/stream and parameterization/config and output being some sort of typed data/stream.

    My “let’s not reinvent the wheel” alarm is going of when I write that though. Examples that come to mind are text based (Unix / https://scale.com/blog/text-universal-interface) but also the Singer tap protocol (https://github.com/singer-io/getting-started/blob/master/doc...). And config obviously having many standard forms like ini, yaml, json, environment key value pairs and more.

    At the same time, text feels horribly inefficient as encoding for some of the data objects being passed around in these flows. More specialized and optimized binary formats come to mind (Arrow, HDF5, Protobuf).

    Plenty of directions to explore, each with their own advantages and disadvantages. I wonder which direction is favored by users of tools like ours. Will be good to poll (do they even care?).

    PS Windmill looks equally impressive! Nice job

  • After Airflow. Where next for DE?
    13 projects | /r/dataengineering | 15 Nov 2022
    Mage uses the Singer Spec (https://github.com/singer-io/getting-started/blob/master/docs/SPEC.md), the data engineer community standard for building data integrations. This was created by Stitch and is widely adopted.
  • Basic data engineering question.
    2 projects | /r/dataengineering | 16 Oct 2022
    I like the Singer Protocol, and the various tools that use it. These include meltano, airbyte, stitch, pipelinewise, and a few others
  • I have hundreds of API data endpoints with different schemas. How do I organize?
    1 project | /r/dataengineering | 10 Oct 2022
    Have you looked into using a dedicated data integration tool? Have you heard of Singer and the Singer Spec? https://github.com/singer-io/getting-started/blob/master/docs/SPEC.md
  • CDC (Change Data Capture) with 3rd party APIs
    1 project | /r/dataengineering | 23 Sep 2022
    Or you could build your own such system and run it on Airflow, Prefect, Dagster, etc. Check out the Singer project for a suite of Python packages designed for such a task. Quality varies greatly, though.
  • Questions about Integration Singer Specification with AWS Glue
    1 project | /r/dataengineering | 26 Aug 2022
    Our team is building out a data platform on AWS glue, and we pull from a variety of data sources including application databases and third party SaaS APIs. I have been looking into ways to standardize pulling data from different sources. The other day I came across the [Singer Specification](https://github.com/singer-io/getting-started) and was interested learning more about it. If anyone has experience working with Singer specifications, I would love to hear more about:
  • Anybody have experience creating singer taps and targets?
    1 project | /r/dataengineering | 30 Jan 2022
    I just read the readme of the Singer getting started repo and am excited to write my first tap! I’m thinking instead of writing a new Airflow DAG whenever I want to pipe API data into our data warehouse I could write a singer tap and use Stitch instead. Is that a stupid idea?
  • A note from our sponsor - InfluxDB
    www.influxdata.com | 19 Jun 2024
    Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality. Learn more →

Stats

Basic getting-started repo stats
16
1,239
0.0
over 1 year ago

Sponsored
Power Real-Time Data Analytics at Scale
Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
www.influxdata.com