Top 5 Java Pipeline Projects

kestra

32 6,260 9.9 Java

Infinitely scalable, event-driven, language-agnostic orchestration and scheduling platform to manage millions of workflows declaratively in code.

Project mention: A High-Performance, Java-Based Orchestration Platform | /r/java | 2023-10-11

Kestra's communication is asynchronous and based on a queuing mechanism. It leverages the Micronaut framework and offers two runners: one that uses a database (JDBC) for both the message queue and resource storage, and another that uses Kafka as the message queue and Elasticsearch as the resource storage. The platform is fully extensible and plugin-based, providing a rich set of plugins for various workflow tasks, triggers, and data storage options. For those interested, the GitHub repository is available here: https://github.com/kestra-io/kestra

workflow-cps-plugin

1 172 8.7 Java
WorkOS

workos.com sponsored

The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
pipeline-github-plugin

1 158 5.7 Java

Pipeline: GitHub
safeql

4 15 0.0 Java

Composable / async / functional / type-safe / parallel-pipelined queries and relations without SQL injection or N+1s. (by karmakaze)

Project mention: Sketch of a Post-ORM | news.ycombinator.com | 2023-06-02

I want sum types.
I want a statically-typed way of constructing composable queries that follow SQL rather than reinvent a different thing. It doesn't have to be the same syntax but it has to be the same structuring.
I started writing one[0] and stopped before doing all the boilerplate code generation, having moved on from the JVM ecosystem for the time being. One thing it does is treat most things like sets so we don't end up with N+1 queries. Another trick it uses is collapsing constant expressions via an expression evaluation library[1].
[0] https://github.com/karmakaze/safeql
[1] https://github.com/karmakaze/moja

moja

4 1 1.8 Java

Composable computation pipelines for Java: Async, Lazy, Option, Try, Result, Multi (List), Stated, Reader, Logger, Writer.

Project mention: Sketch of a Post-ORM | news.ycombinator.com | 2023-06-02

I want sum types.
I want a statically-typed way of constructing composable queries that follow SQL rather than reinvent a different thing. It doesn't have to be the same syntax but it has to be the same structuring.
I started writing one[0] and stopped before doing all the boilerplate code generation, having moved on from the JVM ecosystem for the time being. One thing it does is treat most things like sets so we don't end up with N+1 queries. Another trick it uses is collapsing constant expressions via an expression evaluation library[1].
[0] https://github.com/karmakaze/safeql
[1] https://github.com/karmakaze/moja

InfluxDB

www.influxdata.com sponsored

Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).