Top 23 Kafka Open-Source Projects

1. pathway

   1 9 63,006 9.8 Python

   Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

   

   Project mention: GitHub's Fake Star Economy | news.ycombinator.com | 2026-04-20

2. SaaSHub

   www.saashub.com featured

   SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

   

3. data-engineering-zoomcamp

   2 127 42,355 8.6 Jupyter Notebook

   Data Engineering Zoomcamp is a free 9-week course on building production-ready data pipelines. The next cohort starts in January 2026. Join the course here 👇🏼

   

   Project mention: From APIs to Warehouses: AI-Assisted Data Ingestion with dlt | dev.to | 2026-03-01

   You can sign up here: https://github.com/DataTalksClub/data-engineering-zoomcamp/

4. Apache Kafka

   3 47 32,807 9.9 Java

   Apache Kafka - A distributed event streaming platform

   

   Project mention: Building Kafka Producer-Consumer Using Go and Docker | dev.to | 2026-06-08

   Kafka is a distributed streaming platform used to build real-time data pipelines and streaming applications. It allows producers to send messages to topics, which are then consumed by various consumers, making it ideal for event-driven architectures.

5. SpringBoot-Labs

   4 1 20,089 0.0 Java

   一个涵盖六个专栏：Spring Boot 2.X、Spring Cloud、Spring Cloud Alibaba、Dubbo、分布式消息队列、分布式事务的仓库。希望胖友小手一抖，右上角来个 Star，感恩 1024

   

6. Telegraf

   5 115 17,619 9.9 Go

   Agent for collecting, processing, aggregating, and writing metrics, logs, and other arbitrary data.

   

   Project mention: The Performance Battle the hardening of Vault and OWASP: What Matters | dev.to | 2026-05-07

   A common mistake we see is teams rejecting Vault hardening because they assume it will add 400ms+ p99 latency, a myth perpetuated by outdated blog posts from 2020. Our benchmarks of Vault 1.15 show that OWASP-compliant TLS 1.3 and rate limiting add only 12ms p99 latency, not 400ms. Before deciding against hardening, run the first code example (vault_owasp_benchmark.py) against your own Vault deployment to get real numbers for your workload. Use tools like https://github.com/influxdata/telegraf to collect latency metrics from production, and https://github.com/grafana/grafana to visualize percentiles over time. For read-heavy workloads (10k+ requests per second), the latency overhead is even lower (under 8ms p99) because TLS 1.3 has lower handshake overhead than TLS 1.2. We worked with a SaaS company that rejected hardening for 6 months due to latency fears, only to find after benchmarking that the overhead was 9ms p99, well within their 100ms SLA. They implemented hardening in 2 weeks, cut breach risk by 89%, and saw no customer impact. Always benchmark with your own traffic patterns, not generic numbers from the internet. Use the Go benchmark tool or Python script provided to test with your secret sizes, request patterns, and auth methods. If you see unexpected latency spikes, check for misconfigured cipher suites or rate limits that are too strict, not the hardening itself.

7. C++ Workflow

   6 9 14,369 7.4 C++

   C++ Parallel Computing and Asynchronous Networking Framework

   

8. debezium

   7 97 12,813 9.9 Java

   Change data capture for a variety of databases. Please log issues at https://github.com/debezium/dbz/issues.

   

   Project mention: 7 Free Tools for Data Pipeline Reconciliation and Cross-Source Validation | dev.to | 2026-05-13

   Debezium is an open-source CDC (change data capture) platform that streams database changes - inserts, updates, deletes - from supported databases (PostgreSQL, MySQL, MongoDB, SQL Server) to downstream consumers via Apache Kafka.

9. sarama

   8 3 12,493 9.4 Go

   Sarama is a Go library for Apache Kafka.

   

   Project mention: Kafka Consumer Health Checks: Dead or Alive | dev.to | 2025-12-13

   I’ve packaged this logic into kafka-pulse-go, a lightweight library that works with most popular Kafka clients. The core logic is decoupled from specific client implementations, with adapters provided for Sarama, segmentio/kafka-go, and Confluent’s client.

10. redpanda

    9 77 12,201 10.0 C++

    Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!

    

    Project mention: Top Open-Source Data Engineering Tools- Unravelling the Best in 2026 | dev.to | 2025-12-10

    Redpanda

11. kafka-ui

    10 50 12,055 5.8 Java

    Open-Source Web UI for Apache Kafka Management

    

    Project mention: Kafka UI in Action: Monitoring and Managing Kafka Like a Pro | dev.to | 2025-06-29

    GitHub: https://github.com/provectus/kafka-ui

12. kubeshark

    11 19 11,949 8.9 Go

    eBPF-powered network observability for Kubernetes. Indexes L4/L7 traffic with full K8s context, decrypts TLS without keys. Queryable by AI agents via MCP and humans via dashboard.

    

    Project mention: API Traffic Analyzer for Kubernetes | news.ycombinator.com | 2026-03-09

13. kafka-manager

    12 13 11,938 0.0 Scala

    CMAK is a tool for managing Apache Kafka clusters

    

14. automq

    13 28 10,005 9.4 Java

    Diskless Kafka® on S3. 10x Cost-Effective. No Cross-AZ Traffic Cost. Autoscale in seconds. Single-digit ms latency. Multi-AZ Availability.

    

    Project mention: Top 10 noteworthy Java errors in 2025 | dev.to | 2025-12-24

    Let's move on to the error from AutoMQ:

15. watermill

    14 29 9,750 8.2 Go

    Building event-driven applications the easy way in Go.

    

    Project mention: How I built Upple: A modern uptime monitor with Go and React | dev.to | 2026-01-02

    I'm using Watermill for the event bus with Redis Streams as the backend. Redis Streams has this concept of consumer groups; consumers in the same group split messages between them, while different groups each receive all messages.

16. risingwave

    15 45 9,077 9.9 Rust

    Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.

    

    Project mention: Building a Real-Time Crypto Arbitrage Monitoring System | dev.to | 2025-11-24

    In crypto markets, these price differences, or spreads, appear and vanish in milliseconds. If your data pipeline takes five seconds to process a batch of prices, the opportunity is already gone. This post demonstrates how to use RisingWave—an open-source real-time event streaming platform—to detect arbitrage opportunities with sub-second latency using standard SQL.

17. connect

    16 7 8,678 9.9 Go

    Fancy stream processing made operationally mundane (by redpanda-data)

    

18. kafka-go

    17 16 8,570 4.8 Go

    Kafka library in Go

    

    Project mention: Opinion: Why You Should Use NATS 2.10 Over Kafka for Edge Messaging | dev.to | 2026-04-28

19. DevOps-Bash-tools

    18 1 8,295 9.9 Shell

    1000+ DevOps Bash Scripts - AWS, GCP, Kubernetes, Docker, CI/CD, APIs, SQL, PostgreSQL, MySQL, Hive, Impala, Kafka, Hadoop, Jenkins, GitHub, GitLab, BitBucket, Azure DevOps, TeamCity, Spotify, MP3, LDAP, Code/Build Linting, pkg mgmt for Linux, Mac, Python, Perl, Ruby, NodeJS, Golang, Advanced dotfiles: .bashrc, .vimrc, .gitconfig, .screenrc, tmux..

    

    Project mention: Level Up Your DevOps Workflow with Hari Sekhon's Bash Tools! | dev.to | 2025-06-23

    View the Project on GitHub

20. graylog

    19 48 8,061 10.0 Java

    Free and open log management

    

21. CAP

    20 8 7,088 7.5 C#

    Distributed transaction solution in micro-service base on eventually consistency, also an eventbus with Outbox pattern (by dotnetcore)

    

22. Faust

    21 8 6,823 1.4 Python

    Python Stream Processing

    

23. flink-cdc

    22 6 6,432 9.4 Java

    Flink CDC is a streaming data integration tool

    

    Project mention: Flink CDC from the Trenches: Handling JSON in Pipelines with UDFs | dev.to | 2025-12-16

    A critically important class that Flink CDC ships to perform type conversions internally is the DataTypeConverter, which you can find the source code for right here.

24. materialize

    23 132 6,316 10.0 Rust

    The live data layer for apps and AI agents. Create up-to-the-second views into your business, just using SQL (by MaterializeInc)

    

    Project mention: ANN v3: 200ms p99 query latency over 100B vectors | news.ycombinator.com | 2026-01-25

    I agree our sample may not be representative but we try to stay focused on the current and next crop of tpuf customers. So far "CI prohibits network access during tests" just hasn't come up as a pain point for any of them, but as I mentioned in another comment [0], we're definitely keeping an open mind about introducing an offline dev experience.

    At my last company an engineer spent a year implementing Bazel [0][1] only to have it ripped out after they left [2] due to the maintenance burden. You might say it was a little bit of a hassle. :)

    [0]: https://news.ycombinator.com/item?id=46758156

    [1]: https://github.com/MaterializeInc/materialize/pull/24243

    [2]: https://github.com/MaterializeInc/materialize/pull/31006

    [3]: https://github.com/MaterializeInc/materialize/pull/33895

Kafka related posts

• Cursor made me do it

  3 projects | dev.to | 4 Jun 2026

• Made a Tool to Streams Changes from Microsoft SQL Server to Apache Kafka

  2 projects | news.ycombinator.com | 31 May 2026

• FastStream 0.7: MQTT support – in-memory tests, AsyncAPI generation and more

  1 project | news.ycombinator.com | 1 Jun 2026

• Usage-Based Billing for AI Agents with FastAPI and Kong

  2 projects | dev.to | 26 May 2026

• Show HN: Diom – Back end primitives (queue, rate limit, etc.) in one Rust binary

  1 project | news.ycombinator.com | 20 May 2026

• Show HN: Diom – Open-source back end primitives with no runtime dependencies

  1 project | news.ycombinator.com | 13 May 2026

• 7 Free Tools for Data Pipeline Reconciliation and Cross-Source Validation

  4 projects | dev.to | 13 May 2026
• A note from our sponsor - SaaSHub
  www.saashub.com | 13 Jun 2026

  SaaSHub helps you find the best software and product alternatives Learn more →

Index

Sponsored

SaaSHub - Software Alternatives and Reviews

SaaSHub helps you find the best software and product alternatives

www.saashub.com