SaaSHub helps you find the best software and product alternatives Learn more →
Top 11 data-ingestion Open-Source Projects
-
seatunnel
SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
paimon
Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.
-
multiwoven
🔥 Open Source Reverse ETL and Customer Data Platform (CDP). An open-source alternative to Hightouch, Census, and RudderStack.
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
squirrel-core
A Python library that enables ML teams to share, load, and transform data in a collaborative, flexible, and efficient way :chestnut:
-
Shift
Shift is a high performance better alternative to Airbyte, Singer, Meltano (by piyushsingariya)
Project mention: SeaTunnel – super high-performance, distributed data integration tool | news.ycombinator.com | 2024-04-28
You can actually have "background jobs" in very different ways in Elixir.
> I want background work to live on different compute capacity than http requests, both because they have very different resources usage
In Elixir, because of the way the BEAM works (the unit of parallelism is much cheaper and consume a low amount of memory), "incoming http requests" and related "workers" are not as expensive (a lot less actually) compared to other stacks (for instance Ruby and Python), where it is quite critical to release "http workers" and not hold the connection (which is what lead to the creation of background job tools like Resque, DelayedJob, Sidekiq, Celery...).
This means that you can actually hold incoming HTTP connections a lot longer without troubles.
A consequence of this is that implementing "reverse proxies", or anything calling third party servers _right in the middle_ of your own HTTP call, is usually perfectly acceptable (something I've done more than a couple of times, the latest one powering the reverse proxy behind https://transport.data.gouv.fr - code available at https://github.com/etalab/transport-site/tree/master/apps/un...).
As a consequence, what would be a bad pattern in Python or Ruby (holding the incoming HTTP connection) is not a problem with Elixir.
> because I want to have state or queues in front of background work so there's a well-defined process for retry, error handling, and back-pressure.
Unless you deal with immediate stuff like reverse proxying or cheap "one off async tasks" (like recording a metric), there also are solutions to have more "stateful" background works in Elixir, too.
A popular background job queue is https://github.com/sorentwo/oban (roughly similar to Sidekiq at al), which uses Postgres.
It handles retries, errors etc.
But it's not the only solution, as you have other tools dedicated to processing, such as Broadway (https://github.com/dashbitco/broadway), which handles back-pressure, fault-tolerance, batching etc natively.
You also have more simple options, such as flow (https://github.com/dashbitco/flow), gen_stage (https://github.com/elixir-lang/gen_stage), Task.async_stream (https://hexdocs.pm/elixir/1.12/Task.html#async_stream/5) etc.
It allows to use the "right tool for the job" quite easily.
It is also interesting to note there is no need to "go evented" if you need to fetch data from multiple HTTP servers: it can happen in the exact same process (even: in a background task attached to your HTTP server), as done here https://transport.data.gouv.fr/explore (if you zoom you will see vehicle moving in realtime, and ~80 data sources are being polled every 10 seconds & broadcasted to the visitors via pubsub & websockets).
18. Apache Paimon | Github | tutorial
Project mention: Multiwoven Reverse ETL (0.2.0) – Open-Source Alternative to Hightouch and Census | news.ycombinator.com | 2024-04-19Multiwoven is now a leading Open Source Alternative to Hightouch, Census, and Rudderstack.
It's been a great journey so far, and we are excited to announce a major update to Multiwoven - our new release, Multiwoven 0.2.0, is now available!
Repo: https://github.com/Multiwoven/multiwoven
This release brings a host of new features, enhancements, and bug fixes to streamline data syncs and user experience.
From new connectors to advanced reporting dashboards, as a team, we have been working hard on these updates based on the feedback and requests from our customers and the community.
- 10+ new connectors added to Multiwoven, including
As side hobby I started working on this personal project https://github.com/piyushsingariya/Kaku
data-ingestion related posts
- Multiwoven Reverse ETL (0.2.0) – Open-Source Alternative to Hightouch and Census
- Ask HN: Best way to mirror a Postgres database to parquet?
- Temporal.io: It Just Works
- The lightweight Open CDP and Reverse ETL for your data warehouse
- Why an open source Salesforce CDP alternative is needed
- Why companies need a open source Customer Data Platform (CDP)?
- Ubicloud wants to build open-source alternative to AWS in Ruby
-
A note from our sponsor - SaaSHub
www.saashub.com | 29 Apr 2024
Index
What are some of the best open-source data-ingestion projects? This list will help you:
Project | Stars | |
---|---|---|
1 | seatunnel | 7,223 |
2 | paradedb | 3,863 |
3 | ingestr | 2,308 |
4 | broadway | 2,300 |
5 | Pravega | 1,966 |
6 | paimon | 1,931 |
7 | multiwoven | 617 |
8 | cuelake | 284 |
9 | squirrel-core | 279 |
10 | squirrel-datasets-core | 43 |
11 | Shift | 8 |
Sponsored