Kafka

Top 23 Kafka Open-Source Projects

  • Apache Kafka

    Mirror of Apache Kafka

    Project mention: Implementing tagged fields for Kafka Protocol | /r/apachekafka | 2023-05-27
  • data-engineering-zoomcamp

    Free Data Engineering course!

    Project mention: Building a project in DBT | dev.to | 2024-02-23

    For Week 4 of DataTalksClub's data engineering zoomcamp, we had to install dbt and create a project. This was a formidable task. dbt is a data transformation tool that enables data analysts and engineers to transform data in a cloud analytics warehouse, BigQuery in our case. It took me a very long time to do this, and in this case I needed the homework extension.

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  • SpringBoot-Labs

    一个涵盖六个专栏:Spring Boot 2.X、Spring Cloud、Spring Cloud Alibaba、Dubbo、分布式消息队列、分布式事务的仓库。希望胖友小手一抖,右上角来个 Star,感恩 1024

  • Thingsboard

    Open-source IoT Platform - Device management, data collection, processing and visualization.

    Project mention: ThingsBoard Microservices Installation Update Problem | /r/ThingsBoard | 2023-11-29

    Until recently I was still using TB v3.2.1, now I've set about updating the installation to the latest version. I proceeded as described at https://github.com/thingsboard/thingsboard/tree/master/docker:

  • C++ Workflow

    C++ Parallel Computing and Asynchronous Networking Framework

  • kafka-manager

    CMAK is a tool for managing Apache Kafka clusters

    Project mention: FLaNK Stack Weekly 16 October 2023 | dev.to | 2023-10-17
  • sarama

    Sarama is a Go library for Apache Kafka. (by IBM)

    Project mention: Shopify/sarama is now IBM/sarama | /r/golang | 2023-07-11
  • WorkOS

    The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  • kubeshark

    The API traffic analyzer for Kubernetes providing real-time K8s protocol-level visibility, capturing and monitoring all traffic and payloads going in, out and across containers, pods, nodes and clusters. Inspired by Wireshark, purposely built for Kubernetes

    Project mention: Show HN: Alaz: Open-Source, Self-Hosted, eBPF-Based K8s Monitoring | news.ycombinator.com | 2023-09-06

    The one similar product I had come across is Kubeshark (https://github.com/kubeshark/kubeshark). But admittedly the eBPF way seems more performant theoretically (given you can afford to have a modern-enough kernel). I'm really excited to see how this project develops out.

    The eBPF-mode of innovation is pretty exciting, truly a fresh lens to building software. I'm also following Akita Software - the company building an eBPF paradigm of monitoring.

  • debezium

    Change data capture for a variety of databases. Please log issues at https://issues.redhat.com/browse/DBZ.

    Project mention: Choosing Between a Streaming Database and a Stream Processing Framework in Python | dev.to | 2024-02-10

    They manage data in the application layer and your original data stays where it is. This way data consistency is no longer an issue as it was with streaming databases. You can use Change Data Capture (CDC) services like Debezium by directly connecting to your primary database, doing computational work, and saving the result back or sending real-time data to output streams.

  • openreplay

    Session replay and analytics tool you can self-host. Ideal for reproducing issues, co-browsing with users and optimizing your product.

    Project mention: Show HN: How Cobrowsing Works in Session Replay | news.ycombinator.com | 2024-03-14
  • redpanda

    Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!

    Project mention: Choosing Between a Streaming Database and a Stream Processing Framework in Python | dev.to | 2024-02-10

    Stream-processing platforms such as Apache Kafka, Apache Pulsar, or Redpanda are specifically engineered to foster event-driven communication in a distributed system and they can be a great choice for developing loosely coupled applications. Stream processing platforms analyze data in motion, offering near-zero latency advantages. For example, consider an alert system for monitoring factory equipment. If a machine's temperature exceeds a certain threshold, a streaming platform can instantly trigger an alert and engineers do timely maintenance.

  • kafka-ui

    Open-Source Web UI for Apache Kafka Management

    Project mention: FLaNK Stack Weekly 16 October 2023 | dev.to | 2023-10-17
  • Benthos

    Fancy stream processing made operationally mundane

    Project mention: Ask HN: Who is hiring? (December 2023) | news.ycombinator.com | 2023-12-01
  • librdkafka

    The Apache Kafka C/C++ library

    Project mention: Do you use Rust in your professional career? | /r/rust | 2023-05-09

    recent PR: https://github.com/confluentinc/librdkafka/pull/4275

  • kafka-go

    Kafka library in Go

    Project mention: book about golang and kafka | /r/golang | 2023-06-04

    There are two main libraries that people use to write clients Confluent Kafka and segment io kafka

  • graylog

    Free and open log management

    Project mention: graylog VS openobserve - a user suggested alternative | libhunt.com/r/graylog2-server | 2023-09-07
  • Faust

    Python Stream Processing

    Project mention: Faust VS quix-streams - a user suggested alternative | libhunt.com/r/faust | 2023-12-07
  • watermill

    Building event-driven applications the easy way in Go.

    Project mention: Microservices communication | /r/golang | 2023-12-09

    I’ve successfully worked on projects using an asynchronous event-driven way of connecting services. I really like the decoupling of business logic and the events triggering it. I highly recommend https://github.com/ThreeDotsLabs/watermill to be more flexible when it comes to choosing the actual technology driving the async patter. It might be NATS today but requirements might change and you need to change. Watermill prepares you for this.

  • CAP

    Distributed transaction solution in micro-service base on eventually consistency, also an eventbus with Outbox pattern

    Project mention: Mechanism for managing faulty consumer in asynchronous event broadcast in microservices / modular monolith | /r/dotnet | 2023-05-22

    You might want to implement the outbox pattern. There’s a decent library that’ll help and is really easy to integrate. In particular, I’d look at using the transaction functionality

  • ksql

    The database purpose-built for stream processing applications.

  • materialize

    The data warehouse for operational workloads. (by MaterializeInc)

    Project mention: Choosing Between a Streaming Database and a Stream Processing Framework in Python | dev.to | 2024-02-10

    To fully leverage the data is the new oil concept, companies require a special database designed to manage vast amounts of data instantly. This need has led to different database forms, including NoSQL databases, vector databases, time-series databases, graph databases, in-memory databases, and in-memory data grids. Recent years have seen the rise of cloud-based streaming databases such as RisingWave, Materialize, DeltaStream, and TimePlus. While they each have distinct commercial and technical approaches, their overarching goal remains consistent: to offer users cloud-based streaming database services.

  • kafka-python

    Python client for Apache Kafka

    Project mention: kafka-python VS quix-streams - a user suggested alternative | libhunt.com/r/kafka-python | 2023-12-07
  • kafdrop

    Kafka Web UI

    Project mention: FLaNK Stack Weekly 16 October 2023 | dev.to | 2023-10-17
  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2024-03-14.

Kafka related posts

Index

What are some of the best open-source Kafka projects? This list will help you:

Project Stars
1 Apache Kafka 27,123
2 data-engineering-zoomcamp 22,027
3 SpringBoot-Labs 18,260
4 Thingsboard 15,409
5 C++ Workflow 12,255
6 kafka-manager 11,646
7 sarama 10,892
8 kubeshark 10,462
9 debezium 9,774
10 openreplay 8,713
11 redpanda 8,647
12 kafka-ui 8,266
13 Benthos 7,367
14 librdkafka 7,240
15 kafka-go 7,030
16 graylog 7,014
17 Faust 6,673
18 watermill 6,650
19 CAP 6,406
20 ksql 5,795
21 materialize 5,543
22 kafka-python 5,470
23 kafdrop 5,127
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com