librdkafka
kcat
Our great sponsors
librdkafka | kcat | |
---|---|---|
18 | 18 | |
7,292 | 5,243 | |
1.2% | - | |
8.3 | 0.0 | |
4 days ago | 4 months ago | |
C | C | |
GNU General Public License v3.0 or later | GNU General Public License v3.0 or later |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
librdkafka
-
Do you use Rust in your professional career?
recent PR: https://github.com/confluentinc/librdkafka/pull/4275
-
JR, quality Random Data from the Command line, part I
# Kafka configuration # https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md bootstrap.servers= security.protocol=SASL_SSL sasl.mechanisms=PLAIN sasl.username= sasl.password= compression.type=gzip compression.level=9 statistics.interval.ms=1000
-
A Critical Detail about Kafka Partitioners
But what about Kafka producer clients in other languages? The excellent librdkafka project is a C/C++ implementation of Kafka clients and is widely used for non-JVM Kafka applications. Additionally, Kafka clients in other languages (Python, C#) build on top of it. The default partitioner for librdkafka uses the CRC32 hash function to get the correct partition for a key.
-
Horizontally scaling Kafka consumers with rendezvous hashing
We could have made some changes at the librdkafka level (see this), but we didn’t really want to pursue this (at least not yet).
-
Events with same key going to different partitions
You want records with the same key to always land on the same partition, so you need all the clients to use the same hashing algorithm. The easiest way to do that is to make sure the librdkafka client uses the java compatible murmur2_random hash algorithm. See “Partitioner” section here: https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md
-
Getting sum type values from a map
As my first "real world" (ish) project in Vlang, I'm trying to copy https://github.com/confluentinc/confluent-kafka-go, which is a Go wrapper for Kafka C client library, https://github.com/edenhill/librdkafka
-
Installing node-rdkafka on M1 for use with SASL
If you're using Kafka in a Node.js app, it's likely that you'll need node-rdkafka. This is a library that wraps the librdkafka library and makes it available in Node.js. According to the project's README, "All the complexity of balancing writes across partitions and managing (possibly ever-changing) brokers should be encapsulated in the library."
-
Introduction to Key Apache KafkaⓇ Concepts
# Parse the configuration. # See https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md config_parser = ConfigParser() config_parser.read_file(args.config_file) config = dict(config_parser['default']) # Create Producer instance producer = Producer(config)
-
video analytics on edge
• git clone https://github.com/edenhill/librdkafka.git
- librdkafka - the Apache Kafka C/C++ client library
kcat
-
JR, quality Random Data from the Command line, part I
So, is JR yet another faking library written in Go? Yes and no. JR indeed implements most of the APIs in fakerjs and Go fake it, but it's also able to stream data directly to stdout, Kafka, Redis and more (Elastic and MongoDB coming). JR can talk directly to Confluent Schema Registry, manage json-schema and Avro schemas, easily maintain coherence and referential integrity. If you need more than what is OOTB in JR, you can also easily pipe your data streams to other cli tools like kcat thanks to its flexibility.
-
Deploy Apache Kafka® on Kubernetes
This deployment creates a kcat container we can use to produce and consume messages.
-
How to Build a Kafka Producer in Rust with Partitioning
Now we don't see any additional output. To verify it worked, let's use kafkacat to consume the topic's events. (We install kafkacat in the Dev Container. Please run the following command in VSCode's terminal)
-
Apache Kafka: A Quickstart Guide for Developers
Before we come to an end here, let's explore one additional helpful tool: kcat (formerly known as kafkacat).
-
AdTech using SingleStoreDB, Kafka and Metabase
Let's look at the data in the ad_events topic from the Kafka broker and see if we can identify the problem. We'll install kcat (formerly kafkacat):
-
Getting Started as a Kafka Developer
kcat (formerly KafkaCat) - https://github.com/edenhill/kcat
-
Your Experience Learning and Implementing Kafka
Start with multiple consumers and produce events (this gives a sense about consistency or need for reliable data) - Producer could be command line or kafkacat
-
Running Apache Kafka on Containers
kcat is an awesome tool to make our life easier, it allows us to read and write from kafka topics without tons of scripts and in a more user-friendly way.
- Unreadable data/log files created by Kafka Producer
-
⌨️ Pipe xlsx files into/from Kafka... From cli with (k)cat 🙀
kcat
What are some alternatives?
CVE-2022-27254 - PoC for vulnerability in Honda's Remote Keyless System(CVE-2022-27254)
kafka-python - Python client for Apache Kafka
sarama - Sarama is a Go library for Apache Kafka. [Moved to: https://github.com/IBM/sarama]
rskafka - A minimal Rust client for Apache Kafka
Karafka - Ruby and Rails efficient multithreaded Kafka processing framework
console - Redpanda Console is a developer-friendly UI for managing your Kafka/Redpanda workloads. Console gives you a simple, interactive approach for gaining visibility into your topics, masking data, managing consumer groups, and exploring real-time data with time-travel debugging.
kafka-go - Kafka library in Go
templates - Repository for Dev Container Templates that are managed by Dev Container spec maintainers. See https://github.com/devcontainers/template-starter to create your own!
rsyslog - a Rocket-fast SYStem for LOG processing
jr - JR: streaming quality random data from the command line
rust-kafka-101 - Getting started with Rust and Kafka
kubernetes - Production-Grade Container Scheduling and Management