Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality. Learn more →
Top 23 Pipeline Open-Source Projects
-
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
-
Mage
🧙 The modern replacement for Airflow. Mage is an open-source data pipeline tool for transforming and integrating data. https://github.com/mage-ai/mage-ai
-
-
-
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
-
-
-
-
meltano
Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.
-
-
onepanel
The open source, end-to-end computer vision platform. Label, build, train, tune, deploy and automate in a unified platform that runs on any cloud and on-premises.
-
covalent
Pythonic tool for orchestrating machine-learning/high performance/quantum-computing workflows in heterogeneous compute environments. (by AgnostiqHQ)
-
-
-
pypyr automation task runner
pypyr task-runner cli & api for automation pipelines. Automate anything by combining commands, different scripts in different languages & applications into one pipeline process.
-
PipelineC
A C-like hardware description language (HDL) adding high level synthesis(HLS)-like automatic pipelining as a language construct/compiler feature.
-
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
Like Argo Workflows?
https://github.com/argoproj/argo-workflows
I’ve been working in tech for more than five years. I started as a Data Scientist, and now I’m exploring and loving the DevRel 🥑 role for Taipy. Needless to say, evolving in the tech scene has been a ride full of ups, downs, and everything in between.
Ableton ⬩ Acima ⬩ ACKO ⬩ ActiveState ⬩ Adrima ⬩ AJR International ⬩ Alma ⬩ Astrosat ⬩ Ava ⬩ Avetta ⬩ Azara ⬩ Barmenia ⬩ Basiq ⬩ Beautiful Destinations ⬩ BEC Systems ⬩ Bekk ⬩ Bellroy ⬩ Bendyworks ⬩ Bernoulli Finance ⬩ Blue Fog Training ⬩ BravoTran ⬩ Brilliant ⬩ Budapest School ⬩ Buildr ⬩ Cachix ⬩ CalculoJuridico ⬩ CareRev ⬩ CARFAX ⬩ Caribou ⬩ carwow ⬩ CBANC ⬩ CircuitHub ⬩ CN Group CZ ⬩ CoinTracking ⬩ Concourse CI ⬩ Consensys ⬩ Cornell Tech ⬩ Corvus ⬩ Crowdstrike ⬩ Culture Amp ⬩ Day One ⬩ Deepgram ⬩ diesdas.digital ⬩ Dividat ⬩ Driebit ⬩ Drip ⬩ Emirates ⬩ eSpark ⬩ EXR ⬩ Featurespace ⬩ Field 33 ⬩ Fission ⬩ Flint ⬩ Folq ⬩ Ford ⬩ Forsikring ⬩ Foxhound Systems ⬩ Futurice ⬩ FörsäkringsGirot ⬩ Generative ⬩ Genesys ⬩ Geora ⬩ Gizra ⬩ GWI ⬩ HAMBS ⬩ Hatch ⬩ Hearken ⬩ hello RSE ⬩ HubTran ⬩ IBM ⬩ Idein ⬩ Illuminate ⬩ Improbable ⬩ Innovation through understanding ⬩ Insurello ⬩ iwantmyname ⬩ jambit ⬩ Jobvite ⬩ KOVnet ⬩ Kulkul ⬩ Logistically ⬩ Luko ⬩ Metronome Growth Systems ⬩ Microsoft ⬩ MidwayUSA ⬩ Mimo ⬩ Mind Gym ⬩ MindGym ⬩ Next DLP ⬩ NLX ⬩ Nomalab ⬩ Nomi ⬩ NoRedInk ⬩ Novabench ⬩ NZ Herald ⬩ Permutive ⬩ Phrase ⬩ PINATA ⬩ PinMeTo ⬩ Pivotal Tracker ⬩ PowerReviews ⬩ Practle ⬩ Prima ⬩ Rakuten ⬩ Roompact ⬩ SAVR ⬩ Scoville ⬩ Scrive ⬩ Scrivito ⬩ Serenytics ⬩ Smallbrooks ⬩ Snapview ⬩ SoPost ⬩ Splink ⬩ Spottt ⬩ Stax ⬩ Stowga ⬩ StructionSite ⬩ Studyplus For School ⬩ Symbaloo ⬩ Talend ⬩ Tallink & Silja Line ⬩ Test Double ⬩ thoughtbot ⬩ Travel Perk ⬩ TruQu ⬩ TWave ⬩ Tyler ⬩ Uncover ⬩ Unison ⬩ Veeva ⬩ Vendr ⬩ Verity ⬩ Vnator ⬩ Vy ⬩ W&W Interaction Solutions ⬩ Watermark ⬩ Webbhuset ⬩ Wejoinin ⬩ Zalora ⬩ ZEIT.IO ⬩ Zettle
Project mention: Show HN: JupySQL – a SQL client for Jupyter (ipython-SQL successor) | news.ycombinator.com | 2023-12-06- One-click sharing powered by Ploomber Cloud: https://ploomber.io
Documentation: https://jupysql.ploomber.io
Note that JupySQL is a fork of ipython-sql; which is no longer actively developed. Catherine, ipython-sql's creator, was kind enough to pass the project to us (check out ipython-sql's README).
We'd love to learn what you think and what features we can ship for JupySQL to be the best SQL client! Please let us know in the comments!
Project mention: meltano VS cloudquery - a user suggested alternative | libhunt.com/r/meltano | 2023-06-02
If you're looking for a visual and more intuitive way to load data to Neo4j, you might want to have a look at Apache Hop. Hop comes with tons of functionality to load data to Neo4j.
Pretty interesting request, if SSH is not used, i would try using something like dask which uses tcp to connect and execute assuming your workers are in another machine.I also think something like covalent can be used to extend your own custom plugin in their ecosystem to connect how you want. We have a very custom private plugin written on top of covalent's to have a custom protocol to connect our central on-prem GPU machines to our local laptops that is rpc based, mostly for high performance as well as some mandate security from where the GPU machines are. Once done it is pretty much something like
Project mention: Vector: A high-performance observability data pipeline | news.ycombinator.com | 2024-03-17We're building something similar at Tenzir, but more for operational security workloads. https://docs.tenzir.com
Differences to Vector:
- An agent has optional indexed storage, so you can store your data there and pick it up later. The storage is based on Apache Feather, Parquet's little brother.
- Pipelines operators both work with data frames (Arrow record batches) or chunks of bytes.
- Structured pipelines are multi-schema, i.e., a single pipeline can process streams of record batches with different schemas.
Project mention: 🦊 GitLab CI YAML Modifications: Tackling the Feedback Loop Problem | dev.to | 2023-12-18View on GitHub
Project mention: PipelineC Example: FM Radio Demodulation (FPGA SDR) | news.ycombinator.com | 2024-03-03Related: PipelineC: A C-like hardware description language (HDL):
https://github.com/JulianKemmerer/PipelineC
NOTE:
The open source projects on this list are ordered by number of github stars.
The number of mentions indicates repo mentiontions in the last 12 Months or
since we started tracking (Dec 2020).
Pipelines related posts
- PipelineC Example: FM Radio Demodulation (FPGA SDR)
- Generate non-CPU FPGA circuits from a C-like language
- Ask HN: What do you use to run background jobs?
- What makes C, Verilog, Java, Python, etc. so different?
- How to automate cypher query?
- What are your private FPGA projects and why?
- How to host React/Next "Cheaply" with a global audience? (NGO needs help)
-
A note from our sponsor - InfluxDB
www.influxdata.com | 24 Apr 2024
Index
What are some of the best open-source Pipeline projects? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | argo | 14,259 |
| 2 | Taipy | 8,371 |
| 3 | Concourse | 7,165 |
| 4 | Mage | 7,001 |
| 5 | orchest | 4,020 |
| 6 | zenml | 3,657 |
| 7 | polyaxon | 3,479 |
| 8 | ploomber | 3,369 |
| 9 | argo-events | 2,230 |
| 10 | elyra | 1,770 |
| 11 | MLJ.jl | 1,720 |
| 12 | meltano | 1,587 |
| 13 | hop | 850 |
| 14 | onepanel | 696 |
| 15 | covalent | 687 |
| 16 | tenzir | 609 |
| 17 | azure-devops-cli-extension | 609 |
| 18 | azure-aks-kubernetes-masterclass | 601 |
| 19 | pypyr automation task runner | 568 |
| 20 | glci | 567 |
| 21 | PipelineC | 541 |
| 22 | triggers | 532 |
| 23 | gitlab-monitor | 472 |
Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com