Confluent Kafka Golang Client
Centrifugo
Confluent Kafka Golang Client | Centrifugo | |
---|---|---|
12 | 31 | |
4,435 | 7,924 | |
1.2% | 1.4% | |
8.0 | 8.9 | |
6 days ago | 7 days ago | |
HTML | Go | |
Apache License 2.0 | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
Confluent Kafka Golang Client
-
book about golang and kafka
There are two main libraries that people use to write clients Confluent Kafka and segment io kafka
-
Getting sum type values from a map
As my first "real world" (ish) project in Vlang, I'm trying to copy https://github.com/confluentinc/confluent-kafka-go, which is a Go wrapper for Kafka C client library, https://github.com/edenhill/librdkafka
-
Fix it, Fork it, Fuck off
You are right, but in practice that's not what happens. Companies do not rely on open source libraries, the developers working for such companies do.
I can give you a realistic example. If you want to use Kafka and Go, your probably only option is to use https://github.com/confluentinc/confluent-kafka-go. Its LICENSE explicitly says "no warranty". Now, what if I find a bug in the library? Only two realistic solutions from my side:
1. I submit the issue and hope for the maintainers to fix it
2. I dig deeper and try to fix the issue. I submit the PR
None of the above scenarios are guaranteed to have a happy ending. The issue could be ignored, or piled up among thousand of other (maybe higher prio) issues. My solution may not be optimal and could be rejected (or if it's optimal, nobody is taking a look at it, and it could remain open for weeks/months).
> If that is a problem for you, negotiate a different contract up front - with the maintainer or someone else willing to do the work. That probably means paying them.
In the real world that would mean that I go to my manager and asks them to pay money to the maintainers of confluent-kafka-go to fix the issue I found. I don't think my manager would approve that, but let's imagine he does. The guys at confluent-kafka-go may not want money to fix the issue. These guys have probably already jobs that pay them well, and they work on the library at will.
Note: I'm talking about confluent-kafka-go, which I know is behind the Confluent software company. But I could as well be talking about libraries maintained by individuals like https://github.com/edenhill/librdkafka
-
What are Golang competitors in 2022 when it comes to one-file binary deployment?
it can be completely statically linked binaries. example: https://github.com/confluentinc/confluent-kafka-go/blob/db57ef6235/kafka/librdkafka_vendor/README.md
-
Benthos - Fancy stream processing made operationally mundane
If you find the kafka input slow, try kafka_franz. It might be a bit faster, since it’s based on https://github.com/twmb/franz-go. The kafka one is based on https://github.com/Shopify/sarama. You can also write a custom input based on https://github.com/confluentinc/confluent-kafka-go, but this library relies on CGo, which can be annoying.
-
Sharing event schema ( type ) between producer and a consumer
Last time I checked Confluent does not have a Schema Registry for Go, only for Java, so instead of that I rely on using the guidelines defined for the serialized data, specifically I've used gPRC+Protobuf for doing this, together with buf to detect breaking changes; buf has their own schema registry perhaps that could be something you could explore as well.
-
Hunting down a C memory leak in a Go program
So, in the interests of full transparency - we at Zendesk are actually running a fork of confluent-kafka-go, which I forked to add, amongst other things, context support: https://github.com/confluentinc/confluent-kafka-go/pull/626
This bug actually happened because I mis-merged upstream into our fork and missed an important call to rd_kafka_poll_set_consumer: https://github.com/zendesk/confluent-kafka-go/commit/6e2d889...
-
Create page view analytics system using Kafka, Go, Postgres & GraphQL in 5 steps
Setup Kafka Producer using confluent-kakfka-go
-
Is segmentio/kafka-go production ready ?
I'd suggest https://github.com/confluentinc/confluent-kafka-go we switched from sarama-cluster with minimal work and it works fine. And we process approx 1.2M messages per hour.
-
Go and Kafka
In my company we use this https://github.com/confluentinc/confluent-kafka-go,
Centrifugo
-
WebSockets vs. Server-Sent-Events vs. Long-Polling vs. WebRTC vs. WebTransport
Hello, I am author of https://github.com/centrifugal/centrifugo. Our users can choose from WebSocket, EventSource, WebTransport (experimental stabilize in the future). WebRTC is out of scope as the main purpose is central server based real-time json/binary messaging, and WebRTC makes things much more complex since it shines for peer-to-peer and rich media communications.
What I'd like to add is that Centrifugo also supports HTTP-streaming – not mentioned by the OP – but this is a transport which has advantages over Eventsource - like possibility to send POST body on initial request from web browser (with SSE you can not), it supports binary, and with Readable Streams browser API it's widely supported by modern browsers.
Another thing I'd like to mention about Centrifugo - it supports bidirectional WebSocket fallbacks with EventSource and HTTP-streaming, and does this without sticky sessions requirement. I guess nobody else have this at this point. See https://centrifugal.dev/blog/2022/07/19/centrifugo-v4-releas.... Which solves one more practical concern. Sticky sessions is an optimization in Centrifugo case, not a requirement.
If you are interested in topic, we also have a post about WebSocket scalability - https://centrifugal.dev/blog/2020/11/12/scaling-websocket - it covers some design decisions made in Centrifugo.
- Centrifugo v5.1.0 released, with new powers for real-time messaging tasks, now with proxy GRPC subscription streams – similar to WebSocketd but over the network
-
Integrating websockets into my current app
Check out https://github.com/centrifugal/centrifugo - it was initially designed to be a standalone language-agnostic real-time messaging server. So it may be used with Django without radical change in the existing application and using ASGI. It can also provide a much better performance if you care about it.
- Millions of Active WebSockets with Node.js
-
Show HN: DriftDB is an open source WebSocket back end for real-time apps
https://github.com/centrifugal/centrifugo
It's a complete solution, including server, admin panel and client library.
We're an European company and use OVH, Hetzner and others.
-
Laravel Websockets vs Soketi vs Laravel Echo Server
Hello! Theoretically you can take a look at https://github.com/centrifugal/centrifugo - which is a standalone self-hosted real-time messaging server. It does not have native support for Laravel and not compatible with Pusher protocol, though integrating with any backend system, including Laravel: see the blog post https://centrifugal.dev/blog/2021/12/14/laravel-multi-room-chat-tutorial, also has some helper packages:
-
Is Python a good option to implement Websockets?
Hello, it's also possible to design an app in a way that its core will be built with Python, but WebSocket part delegated to something external and efficient like https://github.com/centrifugal/centrifugo – the benefit of the approach is that application business logic is completely decoupled from the real-time transport layer. This may lead to a scalable design with graceful degradation. I think this is especially useful when you already have backend built with Django and need to handle millions of concurrent connections.
- Centrifugo – real-time messaging server (WebSocket, etc.) which scales well and integrates with any backend. SDKs for browser and mobile development included
-
What is the coolest Go open source projects you have seen?
Centrifugo https://centrifugal.dev/ https://github.com/centrifugal/centrifugo
- Golang updating the front-end with almost real-time events from the backend server
What are some alternatives?
sarama - Sarama is a Go library for Apache Kafka. [Moved to: https://github.com/IBM/sarama]
Socket.io - Realtime application framework (Node.JS server)
kafka-go - Kafka library in Go
NATS - Golang client for NATS, the cloud native messaging system.
goka - Goka is a compact yet powerful distributed stream processing library for Apache Kafka written in Go.
Mercure - 🪽 An open, easy, fast, reliable and battery-efficient solution for real-time communications
Benthos - Fancy stream processing made operationally mundane
laravel-websockets - Websockets for Laravel. Done right.
confluent-kafka-python - Confluent's Kafka Python Client
soketi - Next-gen, Pusher-compatible, open-source WebSockets server. Simple, fast, and resilient. 📣
Uniqush-Push - Uniqush is a free and open source software system which provides a unified push service for server side notification to apps on mobile devices.