Go Kafka

Open-source Go projects categorized as Kafka

Top 23 Go Kafka Projects

  • sarama

    Sarama is a Go library for Apache Kafka.

    Project mention: Klient - a native, statically-compiled, command line client for Kafka | reddit.com/r/golang | 2023-02-10

    I've used mainly sarama wrapped with a bit of bespoke helper libraries. Never really looked into others, just grabbed one that was actively maintained and went for it.

  • kubeshark

    The API traffic analyzer for Kubernetes providing real-time K8s protocol-level visibility, capturing and monitoring all traffic and payloads going in, out and across containers, pods, nodes and clusters.. Think TCPDump and Wireshark re-invented for Kubernetes

    Project mention: Can't make kubeshark work | reddit.com/r/kubernetes | 2023-04-27

    Can you send us the logs using: `kubeshark logs` (you can also use the issues page or slack)

  • InfluxDB

    Access the most powerful time series database as a service. Ingest, store, & analyze all types of time series data in a fully-managed, purpose-built database. Keep data forever with low-cost storage and superior data compression.

  • Benthos

    Fancy stream processing made operationally mundane

    Project mention: Show HN: Open-source Auth0 alternative Ory Kratos v0.13 released – nearing v1.0 | news.ycombinator.com | 2023-04-19
  • kafka-go

    Kafka library in Go

    Project mention: Klient - a native, statically-compiled, command line client for Kafka | reddit.com/r/golang | 2023-02-10

    Unlike the standard scripts, and many binary clients, it's a native, statically-compiled, binary. It uses segmentio/go-kafka internally, which means CGO can be disabled during compilation.

  • watermill

    Building event-driven applications the easy way in Go.

    Project mention: Small EDA/Micro service Project | reddit.com/r/golang | 2023-05-04

    These conditions have led me to discover two packages watermill.io and go-kit. Both of these seem to do what I want and I am thinking of implementing the project using watermill and go Chanels and deploying to a server or some sort.

  • Jocko

    Kafka implemented in Golang with built-in coordination (No ZK dep, single binary install, Cloud Native)

    Project mention: Ask HN: What interesting problems are you working on? ( 2022 Edition) | news.ycombinator.com | 2022-09-16

    Still early stages but building on top of this. - https://github.com/travisjeffery/jocko

  • Gizmo

    A Microservice Toolkit from The New York Times

    Project mention: GoLang — Simplifying Complexity “The Beginning” | dev.to | 2023-05-27

    . Web backend (with various frameworks available) . Web Assembly (one of them is vugu framework) . Microservices (some frameworks: Go Micro, Go Kit, Gizmo, Kite) . Fragments services (Term mentioned by @jeffotoni in a microservices discussion group) . Lambdas (FaaS example) . Client Server . Terminal applications (using the tview lib) . IoT (some frameworks) . Bots (some here) . Client Applications using Web technology . Desktop using Qt+QML, Native Win Lib (example Qt, Qt widgets, Qml) . Network Applications . Protocol applications . REST Applications . SOAP Applications . GraphQL Applications . RPC Applications . TCP Applications . gRPC Applications . WebSocket Applications . GopherJS (compiles Go to JavaScript)

  • CodiumAI

    TestGPT | Generating meaningful tests for busy devs. Get non-trivial tests (and trivial, too!) suggested right inside your IDE, so you can code smart, create more value, and stay confident when you push.

  • go-zero-looklook

    🔥基于go-zero(go zero) 微服务全技术栈开发最佳实践项目。Develop best practice projects based on the full technology stack of go zero (go zero) microservices.

  • goka

    Goka is a compact yet powerful distributed stream processing library for Apache Kafka written in Go.

  • plumber

    A swiss army knife CLI tool for interacting with Kafka, RabbitMQ and other messaging systems.

    Project mention: Pulling CDC data from Postgres | reddit.com/r/dataengineering | 2023-04-30

    I recommend Streamdal. The connecting agent is open source and distributed by default, so it will scale horizontally WAY better than Debezium. All data ingested is indexed into parquet as well, and you can do serverless functions/transforms on the platform to reduce Snowflake compute costs.

  • kafka_exporter

    Kafka exporter for Prometheus

    Project mention: Kafka Devs: Are there any important metrics that you are unable to monitor with current tools? | reddit.com/r/apachekafka | 2023-05-10

    For Prometheus there is the Kafka exporter.

  • eventhorizon

    Event Sourcing for Go!

    Project mention: Help to migrate a simple app from Elm to React, Angular or VanillaJS. | reddit.com/r/elm | 2022-11-06

    The todo app

  • go-streams

    A lightweight stream processing library for Go

    Project mention: When is go not a good choice? | reddit.com/r/golang | 2023-03-03

    Have you looked at this streaming library? https://github.com/reugn/go-streams

  • gnomock

    Test your code without writing mocks with ephemeral Docker containers 📦 Setup popular services with just a couple lines of code ⏱️ No bash, no yaml, only code 💻

    Project mention: How to unit test your database interactions with Docker | reddit.com/r/golang | 2023-03-24

    https://github.com/orlangure/gnomock does this pretty well, but I liked your solution too, it is neat.

  • franz-go

    franz-go contains a feature complete, pure Go library for interacting with Kafka from 0.8.0 through 3.4+. Producing, consuming, transacting, administrating, etc.

    Project mention: New Winner of Kafka Consumers: Scala to Go Journey | reddit.com/r/golang | 2023-04-18

    Can you post the code for that? Franz-go itself publishes benchmark samples for all clients here, so easy to run on your own machine and also review/adapt settings as needed: https://github.com/twmb/franz-go/tree/master/examples/bench

  • vFlow

    Enterprise Network Flow Collector (IPFIX, sFlow, Netflow)

  • go-stash

    go-stash is a high performance, free and open source server-side data processing pipeline that ingests data from Kafka, processes it, and then sends it to ElasticSearch.

  • goflow

    The high-scalability sFlow/NetFlow/IPFIX collector used internally at Cloudflare.

    Project mention: Sflow/netflow platform with ArubaOS-CX | reddit.com/r/ArubaNetworks | 2022-12-14

    All in one platform would be preferred, but Docker could sway me to something else. Meaning, I don't want to install a bunch of puzzle pieces to get up and running. GoFlow would be an example here.

  • koperator

    Oh no! Yet another Apache Kafka operator for Kubernetes

    Project mention: How to create "ordinal id" for normal Deployments? | reddit.com/r/kubernetes | 2023-05-21

    Typo? You probably meant https://github.com/banzaicloud/koperator

  • kafkactl

    Command Line Tool for managing Apache Kafka

  • components-contrib

    Community driven, reusable components for distributed apps

    Project mention: istributed Application Runtime (Dapr) v1.10 Released | reddit.com/r/AZURE | 2023-03-19

    Microsoft introduced Dapr in October 2019 and has had several releases since its production-ready release V1.0 in February 2021. Since the V1.0 release, Dapr has been approved by Cloud Native Computing Foundation (CNCF) as an incubation project, received approximately 43 new components, and has over 2500 community contributors.

  • terraform-provider-kafka

    Terraform provider for managing Apache Kafka Topics + ACLs

    Project mention: Least-privilege for Kafka data | reddit.com/r/apachekafka | 2023-03-29

    Interesting. In my current company we use this terraform plugin https://github.com/Mongey/terraform-provider-kafka . Everything is disabled by default and you need to create a small Pull Request in a centralized repo that specifies where a service will be producing/consuming. It's not as neat as the solution presented in this article, but it's good enough for us.

  • tiflow

    This repo maintains DM (a data migration platform) and TiCDC (change data capture for TiDB)

  • SonarLint

    Clean code begins in your IDE with SonarLint. Up your coding game and discover issues early. SonarLint is a free plugin that helps you find & fix bugs and security issues from the moment you start writing code. Install from your favorite IDE marketplace today.

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2023-05-27.

Go Kafka related posts


What are some of the best open-source Kafka projects in Go? This list will help you:

Project Stars
1 sarama 10,013
2 kubeshark 9,090
3 Benthos 6,203
4 kafka-go 6,146
5 watermill 5,603
6 Jocko 4,718
7 Gizmo 3,705
8 go-zero-looklook 3,221
9 goka 2,131
10 plumber 1,928
11 kafka_exporter 1,755
12 eventhorizon 1,462
13 go-streams 1,442
14 gnomock 1,182
15 franz-go 1,068
16 vFlow 988
17 go-stash 944
18 goflow 726
19 koperator 701
20 kafkactl 598
21 components-contrib 454
22 terraform-provider-kafka 436
23 tiflow 353
ONLYOFFICE Docs — document collaboration in your environment
Powerful document editing and collaboration in your app or environment. Ultimate security, API and 30+ ready connectors, SaaS or on-premises