Get non-trivial tests (and trivial, too!) suggested right inside your IDE, so you can code smart, create more value, and stay confident when you push. Learn more →
Top 23 Go Kafka Projects
-
Project mention: Klient - a native, statically-compiled, command line client for Kafka | reddit.com/r/golang | 2023-02-10
I've used mainly sarama wrapped with a bit of bespoke helper libraries. Never really looked into others, just grabbed one that was actively maintained and went for it.
-
kubeshark
The API traffic analyzer for Kubernetes providing real-time K8s protocol-level visibility, capturing and monitoring all traffic and payloads going in, out and across containers, pods, nodes and clusters.. Think TCPDump and Wireshark re-invented for Kubernetes
Can you send us the logs using: `kubeshark logs` (you can also use the issues page or slack)
-
InfluxDB
Access the most powerful time series database as a service. Ingest, store, & analyze all types of time series data in a fully-managed, purpose-built database. Keep data forever with low-cost storage and superior data compression.
-
Project mention: Show HN: Open-source Auth0 alternative Ory Kratos v0.13 released – nearing v1.0 | news.ycombinator.com | 2023-04-19
-
Project mention: Klient - a native, statically-compiled, command line client for Kafka | reddit.com/r/golang | 2023-02-10
Unlike the standard scripts, and many binary clients, it's a native, statically-compiled, binary. It uses segmentio/go-kafka internally, which means CGO can be disabled during compilation.
-
These conditions have led me to discover two packages watermill.io and go-kit. Both of these seem to do what I want and I am thinking of implementing the project using watermill and go Chanels and deploying to a server or some sort.
-
Jocko
Kafka implemented in Golang with built-in coordination (No ZK dep, single binary install, Cloud Native)
Project mention: Ask HN: What interesting problems are you working on? ( 2022 Edition) | news.ycombinator.com | 2022-09-16Still early stages but building on top of this. - https://github.com/travisjeffery/jocko
-
. Web backend (with various frameworks available) . Web Assembly (one of them is vugu framework) . Microservices (some frameworks: Go Micro, Go Kit, Gizmo, Kite) . Fragments services (Term mentioned by @jeffotoni in a microservices discussion group) . Lambdas (FaaS example) . Client Server . Terminal applications (using the tview lib) . IoT (some frameworks) . Bots (some here) . Client Applications using Web technology . Desktop using Qt+QML, Native Win Lib (example Qt, Qt widgets, Qml) . Network Applications . Protocol applications . REST Applications . SOAP Applications . GraphQL Applications . RPC Applications . TCP Applications . gRPC Applications . WebSocket Applications . GopherJS (compiles Go to JavaScript)
-
CodiumAI
TestGPT | Generating meaningful tests for busy devs. Get non-trivial tests (and trivial, too!) suggested right inside your IDE, so you can code smart, create more value, and stay confident when you push.
-
go-zero-looklook
🔥基于go-zero(go zero) 微服务全技术栈开发最佳实践项目。Develop best practice projects based on the full technology stack of go zero (go zero) microservices.
-
goka
Goka is a compact yet powerful distributed stream processing library for Apache Kafka written in Go.
-
plumber
A swiss army knife CLI tool for interacting with Kafka, RabbitMQ and other messaging systems.
I recommend Streamdal. The connecting agent is open source and distributed by default, so it will scale horizontally WAY better than Debezium. All data ingested is indexed into parquet as well, and you can do serverless functions/transforms on the platform to reduce Snowflake compute costs.
-
Project mention: Kafka Devs: Are there any important metrics that you are unable to monitor with current tools? | reddit.com/r/apachekafka | 2023-05-10
For Prometheus there is the Kafka exporter.
-
Project mention: Help to migrate a simple app from Elm to React, Angular or VanillaJS. | reddit.com/r/elm | 2022-11-06
The todo app
-
Have you looked at this streaming library? https://github.com/reugn/go-streams
-
gnomock
Test your code without writing mocks with ephemeral Docker containers 📦 Setup popular services with just a couple lines of code ⏱️ No bash, no yaml, only code 💻
Project mention: How to unit test your database interactions with Docker | reddit.com/r/golang | 2023-03-24https://github.com/orlangure/gnomock does this pretty well, but I liked your solution too, it is neat.
-
franz-go
franz-go contains a feature complete, pure Go library for interacting with Kafka from 0.8.0 through 3.4+. Producing, consuming, transacting, administrating, etc.
Project mention: New Winner of Kafka Consumers: Scala to Go Journey | reddit.com/r/golang | 2023-04-18Can you post the code for that? Franz-go itself publishes benchmark samples for all clients here, so easy to run on your own machine and also review/adapt settings as needed: https://github.com/twmb/franz-go/tree/master/examples/bench
-
-
go-stash
go-stash is a high performance, free and open source server-side data processing pipeline that ingests data from Kafka, processes it, and then sends it to ElasticSearch.
-
All in one platform would be preferred, but Docker could sway me to something else. Meaning, I don't want to install a bunch of puzzle pieces to get up and running. GoFlow would be an example here.
-
Project mention: How to create "ordinal id" for normal Deployments? | reddit.com/r/kubernetes | 2023-05-21
Typo? You probably meant https://github.com/banzaicloud/koperator
-
-
Project mention: istributed Application Runtime (Dapr) v1.10 Released | reddit.com/r/AZURE | 2023-03-19
Microsoft introduced Dapr in October 2019 and has had several releases since its production-ready release V1.0 in February 2021. Since the V1.0 release, Dapr has been approved by Cloud Native Computing Foundation (CNCF) as an incubation project, received approximately 43 new components, and has over 2500 community contributors.
-
Interesting. In my current company we use this terraform plugin https://github.com/Mongey/terraform-provider-kafka . Everything is disabled by default and you need to create a small Pull Request in a centralized repo that specifies where a service will be producing/consuming. It's not as neat as the solution presented in this article, but it's good enough for us.
-
-
SonarLint
Clean code begins in your IDE with SonarLint. Up your coding game and discover issues early. SonarLint is a free plugin that helps you find & fix bugs and security issues from the moment you start writing code. Install from your favorite IDE marketplace today.
NOTE:
The open source projects on this list are ordered by number of github stars.
The number of mentions indicates repo mentiontions in the last 12 Months or
since we started tracking (Dec 2020).
The latest post mention was on 2023-05-27.
Go Kafka related posts
- Which email marketing tool do you use? Do you like/dislike it?
- Unlimited Devices is a Scam Advertise
- How to create "ordinal id" for normal Deployments?
- Kafka Devs: Are there any important metrics that you are unable to monitor with current tools?
- Why does our webpage take so long to load?
- How to Deploy Apache Kafka® on Kubernetes If You're in a Time Crunch
- Pulling CDC data from Postgres
-
A note from our sponsor - CodiumAI
codium.ai | 31 May 2023
Index
What are some of the best open-source Kafka projects in Go? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | sarama | 10,013 |
| 2 | kubeshark | 9,090 |
| 3 | Benthos | 6,203 |
| 4 | kafka-go | 6,146 |
| 5 | watermill | 5,603 |
| 6 | Jocko | 4,718 |
| 7 | Gizmo | 3,705 |
| 8 | go-zero-looklook | 3,221 |
| 9 | goka | 2,131 |
| 10 | plumber | 1,928 |
| 11 | kafka_exporter | 1,755 |
| 12 | eventhorizon | 1,462 |
| 13 | go-streams | 1,442 |
| 14 | gnomock | 1,182 |
| 15 | franz-go | 1,068 |
| 16 | vFlow | 988 |
| 17 | go-stash | 944 |
| 18 | goflow | 726 |
| 19 | koperator | 701 |
| 20 | kafkactl | 598 |
| 21 | components-contrib | 454 |
| 22 | terraform-provider-kafka | 436 |
| 23 | tiflow | 353 |
ONLYOFFICE Docs — document collaboration in your environment
Powerful document editing and collaboration in your app or environment. Ultimate security, API and 30+ ready connectors, SaaS or on-premises
www.onlyoffice.com