Go Apache Kafka

Open-source Go projects categorized as Apache Kafka

Top 7 Go Apache Kafka Projects

  • console

    Redpanda Console is a developer-friendly UI for managing your Kafka/Redpanda workloads. Console gives you a simple, interactive approach for gaining visibility into your topics, masking data, managing consumer groups, and exploring real-time data with time-travel debugging. (by redpanda-data)

  • Project mention: What are your favorite tools or components in the Kafka ecosystem? | /r/apachekafka | 2023-05-31
  • kafkactl

    Command Line Tool for managing Apache Kafka

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo
  • transfer

    Database replication platform that leverages change data capture. Stream production data from databases to your data warehouse (Snowflake, BigQuery, Redshift) in real-time.

  • Project mention: Migrate mongodb Datawarehouse to snowflake | /r/snowflake | 2023-12-04
  • xk6-kafka

    k6 extension to load test Apache Kafka with support for various serialization formats, SASL, TLS, compression, Schema Registry client and beyond

  • Project mention: Performance testing Kafka server using xk6-kafka | dev.to | 2024-01-01

    import { Writer, Reader, SCHEMA_TYPE_STRING, SchemaRegistry, GROUP_BALANCER_ROUND_ROBIN, BALANCER_ROUND_ROBIN, } from "k6/x/kafka"; import { check, sleep } from "k6"; const bootstrapServers = [ 'kafka-bootsrap-1:9001', ]; let vus_amount = 3; let total_written = 0; let total_read = 0; export const options = { vus: vus_amount, iterations: "3", thresholds: { kafka_writer_error_count: ["count == 0"], kafka_reader_error_count: ["count == 0"], }, }; const topicName = "my-topic"; const schemaRegistry = new SchemaRegistry(); const producer = new Writer({ brokers: bootstrapServers, topic: topicName, balancer: BALANCER_ROUND_ROBIN, // or pick another balancer https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L66 // ... auth config }); export function teardown(data) { producer.close(); } export default function () { const consumer = new Reader({ brokers: bootstrapServers, // it is important to set groupID, groupTopics and groupBalancers when using Kafka bootstrap server // topic ReaderConfig param doesn't quite work with bootstrap server groupID: topicName + "-group", groupTopics: [topicName], groupBalancers: [GROUP_BALANCER_ROUND_ROBIN], // or pick different balancer https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L75 }); let messageAmount = 1000; if (__VU == 1) { console.log('VU 1, writing messages. Iter ' + __ITER); for (let index = 0; index < messageAmount; index++) { let messages = [ { value: schemaRegistry.serialize({ data: "test-value-string-" + index + "-vu-" + __VU + "-iter-" + __ITER, schemaType: SCHEMA_TYPE_STRING, }), }, ]; producer.produce({ messages: messages }); total_written += messages.length; } } let consumerMsgAmount = Math.floor(messageAmount / vus_amount); let messages = consumer.consume({ limit: consumerMsgAmount}); total_read += messages.length; console.log("Amount of msgs received: " + messages.length + ", VU " + __VU + ", iter " + __ITER); check(messages, { "all messages returned": (msgs) => msgs.length == consumerMsgAmount, }); let firstMessageValue = schemaRegistry.deserialize({ data: messages[0].value, schemaType: SCHEMA_TYPE_STRING, }); let lastMessageValue = schemaRegistry.deserialize({ data: messages[consumerMsgAmount - 1].value, schemaType: SCHEMA_TYPE_STRING, }); check(messages[0], { "Topic equals to": (msg) => msg["topic"] == topicName }); console.log("First msg value " + firstMessageValue + ", offset" + messages[0]["offset"] + ", partition " + messages[0]["partition"] + ", VU " + __VU + ", iter " + __ITER); console.log("Last msg value " + lastMessageValue + ", offset" + messages[consumerMsgAmount - 1]["offset"] + ", partition " + messages[0]["partition"] + ", VU " + __VU + ", iter " + __ITER); consumer.close(); }

  • kafta

    Kafta is a command line for managing Kafka clusters

  • goneli

    Implementation of the NELI leader election protocol for Go and Kafka

  • rtdl

    rtdl makes it easy to build and maintain a real-time data lake (by realtimedatalake)

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    SaaSHub logo
NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).

Go Apache Kafka related posts

  • Performance testing Kafka server using xk6-kafka

    1 project | dev.to | 1 Jan 2024
  • 🔥 What is Apache Kafka and how to perform performance tests on it (Part 2) 👨🏻‍💻

    2 projects | dev.to | 18 Dec 2023
  • Stream MySQL changes

    3 projects | /r/golang | 3 Feb 2023
  • xk6-kafka v0.12.0 is released! 🎉

    1 project | /r/golang | 4 Jul 2022
  • Hacking PostgreSQL Internals to Deliver Push Notifications

    4 projects | dev.to | 20 Jan 2022
  • Understanding Kafka with Factorio

    2 projects | news.ycombinator.com | 26 Nov 2021
  • How to Load Test Your Kafka Producers and Consumers using k6

    2 projects | dev.to | 21 Sep 2021
  • A note from our sponsor - InfluxDB
    www.influxdata.com | 24 May 2024
    Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality. Learn more →

Index

What are some of the best open-source Apache Kafka projects in Go? This list will help you:

Project Stars
1 console 3,619
2 kafkactl 755
3 transfer 540
4 xk6-kafka 131
5 kafta 92
6 goneli 63
7 rtdl 43

Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com