Go Apache Kafka

Open-source Go projects categorized as Apache Kafka

Top 7 Go Apache Kafka Projects

  • console

    Redpanda Console is a developer-friendly UI for managing your Kafka/Redpanda workloads. Console gives you a simple, interactive approach for gaining visibility into your topics, masking data, managing consumer groups, and exploring real-time data with time-travel debugging. (by redpanda-data)

    Project mention: What are your favorite tools or components in the Kafka ecosystem? | /r/apachekafka | 2023-05-31
  • kafkactl

    Command Line Tool for managing Apache Kafka

  • WorkOS

    The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  • transfer

    Database replication platform that leverages change data capture. Stream production data from databases to your data warehouse (Snowflake, BigQuery, Redshift) in real-time.

    Project mention: Migrate mongodb Datawarehouse to snowflake | /r/snowflake | 2023-12-04
  • xk6-kafka

    k6 extension to load test Apache Kafka with support for various serialization formats, SASL, TLS, compression, Schema Registry client and beyond

    Project mention: Performance testing Kafka server using xk6-kafka | dev.to | 2024-01-01

    import { Writer, Reader, SCHEMA_TYPE_STRING, SchemaRegistry, GROUP_BALANCER_ROUND_ROBIN, BALANCER_ROUND_ROBIN, } from "k6/x/kafka"; import { check, sleep } from "k6"; const bootstrapServers = [ 'kafka-bootsrap-1:9001', ]; let vus_amount = 3; let total_written = 0; let total_read = 0; export const options = { vus: vus_amount, iterations: "3", thresholds: { kafka_writer_error_count: ["count == 0"], kafka_reader_error_count: ["count == 0"], }, }; const topicName = "my-topic"; const schemaRegistry = new SchemaRegistry(); const producer = new Writer({ brokers: bootstrapServers, topic: topicName, balancer: BALANCER_ROUND_ROBIN, // or pick another balancer https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L66 // ... auth config }); export function teardown(data) { producer.close(); } export default function () { const consumer = new Reader({ brokers: bootstrapServers, // it is important to set groupID, groupTopics and groupBalancers when using Kafka bootstrap server // topic ReaderConfig param doesn't quite work with bootstrap server groupID: topicName + "-group", groupTopics: [topicName], groupBalancers: [GROUP_BALANCER_ROUND_ROBIN], // or pick different balancer https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L75 }); let messageAmount = 1000; if (__VU == 1) { console.log('VU 1, writing messages. Iter ' + __ITER); for (let index = 0; index < messageAmount; index++) { let messages = [ { value: schemaRegistry.serialize({ data: "test-value-string-" + index + "-vu-" + __VU + "-iter-" + __ITER, schemaType: SCHEMA_TYPE_STRING, }), }, ]; producer.produce({ messages: messages }); total_written += messages.length; } } let consumerMsgAmount = Math.floor(messageAmount / vus_amount); let messages = consumer.consume({ limit: consumerMsgAmount}); total_read += messages.length; console.log("Amount of msgs received: " + messages.length + ", VU " + __VU + ", iter " + __ITER); check(messages, { "all messages returned": (msgs) => msgs.length == consumerMsgAmount, }); let firstMessageValue = schemaRegistry.deserialize({ data: messages[0].value, schemaType: SCHEMA_TYPE_STRING, }); let lastMessageValue = schemaRegistry.deserialize({ data: messages[consumerMsgAmount - 1].value, schemaType: SCHEMA_TYPE_STRING, }); check(messages[0], { "Topic equals to": (msg) => msg["topic"] == topicName }); console.log("First msg value " + firstMessageValue + ", offset" + messages[0]["offset"] + ", partition " + messages[0]["partition"] + ", VU " + __VU + ", iter " + __ITER); console.log("Last msg value " + lastMessageValue + ", offset" + messages[consumerMsgAmount - 1]["offset"] + ", partition " + messages[0]["partition"] + ", VU " + __VU + ", iter " + __ITER); consumer.close(); }

  • kafta

    Kafta is a command line for managing Kafka clusters

  • goneli

    Implementation of the NELI leader election protocol for Go and Kafka

  • rtdl

    rtdl makes it easy to build and maintain a real-time data lake (by realtimedatalake)

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2024-01-01.

Go Apache Kafka related posts

Index

What are some of the best open-source Apache Kafka projects in Go? This list will help you:

Project Stars
1 console 3,575
2 kafkactl 732
3 transfer 517
4 xk6-kafka 123
5 kafta 92
6 goneli 63
7 rtdl 44
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com