Top 7 Go Avro Projects

• kafkactl

  1 750 8.7 Go

  Command Line Tool for managing Apache Kafka

  

• srclient

  1 221 2.4 Go

  Golang Client for Schema Registry

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• xk6-kafka

  4 131 6.3 Go

  k6 extension to load test Apache Kafka with support for various serialization formats, SASL, TLS, compression, Schema Registry client and beyond

Project mention: Performance testing Kafka server using xk6-kafka | dev.to | 2024-01-01

import { Writer, Reader, SCHEMA_TYPE_STRING, SchemaRegistry, GROUP_BALANCER_ROUND_ROBIN, BALANCER_ROUND_ROBIN, } from "k6/x/kafka"; import { check, sleep } from "k6"; const bootstrapServers = [ 'kafka-bootsrap-1:9001', ]; let vus_amount = 3; let total_written = 0; let total_read = 0; export const options = { vus: vus_amount, iterations: "3", thresholds: { kafka_writer_error_count: ["count == 0"], kafka_reader_error_count: ["count == 0"], }, }; const topicName = "my-topic"; const schemaRegistry = new SchemaRegistry(); const producer = new Writer({ brokers: bootstrapServers, topic: topicName, balancer: BALANCER_ROUND_ROBIN, // or pick another balancer https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L66 // ... auth config }); export function teardown(data) { producer.close(); } export default function () { const consumer = new Reader({ brokers: bootstrapServers, // it is important to set groupID, groupTopics and groupBalancers when using Kafka bootstrap server // topic ReaderConfig param doesn't quite work with bootstrap server groupID: topicName + "-group", groupTopics: [topicName], groupBalancers: [GROUP_BALANCER_ROUND_ROBIN], // or pick different balancer https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L75 }); let messageAmount = 1000; if (__VU == 1) { console.log('VU 1, writing messages. Iter ' + __ITER); for (let index = 0; index < messageAmount; index++) { let messages = [ { value: schemaRegistry.serialize({ data: "test-value-string-" + index + "-vu-" + __VU + "-iter-" + __ITER, schemaType: SCHEMA_TYPE_STRING, }), }, ]; producer.produce({ messages: messages }); total_written += messages.length; } } let consumerMsgAmount = Math.floor(messageAmount / vus_amount); let messages = consumer.consume({ limit: consumerMsgAmount}); total_read += messages.length; console.log("Amount of msgs received: " + messages.length + ", VU " + __VU + ", iter " + __ITER); check(messages, { "all messages returned": (msgs) => msgs.length == consumerMsgAmount, }); let firstMessageValue = schemaRegistry.deserialize({ data: messages[0].value, schemaType: SCHEMA_TYPE_STRING, }); let lastMessageValue = schemaRegistry.deserialize({ data: messages[consumerMsgAmount - 1].value, schemaType: SCHEMA_TYPE_STRING, }); check(messages[0], { "Topic equals to": (msg) => msg["topic"] == topicName }); console.log("First msg value " + firstMessageValue + ", offset" + messages[0]["offset"] + ", partition " + messages[0]["partition"] + ", VU " + __VU + ", iter " + __ITER); console.log("Last msg value " + lastMessageValue + ", offset" + messages[consumerMsgAmount - 1]["offset"] + ", partition " + messages[0]["partition"] + ", VU " + __VU + ", iter " + __ITER); consumer.close(); }

• jr

  4 114 9.4 Go

  JR: streaming quality random data from the command line (by ugol)

Project mention: JR, quality Random Data from the Command line, part II | dev.to | 2023-06-01

We have seen how to use JR in more advanced use cases, streaming quality random data with referential integrity. In the next part of this series, we will see how to use REST apis with JR. In the meanwhile, happy streaming!

• avro

  0 45 0.0 Go

  Apache AVRO for go (by khezen)

• singlestore-logistics-sim

  5 32 2.5 Go

  Scalable package delivery logistics simulator built using SingleStore and Vectorized Redpanda

  

• schema-registry-statistics

  2 24 7.7 Go

  Schema Registry Statistics Tool

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

Go Avro related posts

• Schema Registry Statistics Tool is a small utility that allows you to easily identify the usage of different schema versions within a topic. Using this tool, you can consume from a topic, while calculating the percentage of each schema version.

  1 project | /r/dataengineering | 18 Feb 2023

• Open source proxy layer in front of Kafka for strongly typed events?

  2 projects | /r/apachekafka | 22 Oct 2021

• BigQuery alternatives and the future of data warehousing

  1 project | /r/dataengineering | 2 Jul 2021

• I built a reference architecture for global package logistics and learned a bunch about Terraform in the process + scaled up to 400k packages delivered per second!

  1 project | /r/devops | 28 Jun 2021

• I built an e2e simulation of global package logistics... then scaled it to 125x yearly volume! Code, terraform, and dashboards included - run everything in docker locally or deploy it to Google Cloud

  1 project | /r/coding | 28 Jun 2021

• Decided to simulate global package delivery on my computer... Ended up scaling it up to 125x!

  1 project | /r/programming | 28 Jun 2021

• Scaling Worldwide Parcel Logistics with SingleStore and Vectorized

  2 projects | news.ycombinator.com | 24 Jun 2021
• A note from our sponsor - InfluxDB
  www.influxdata.com | 10 May 2024

  Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality. Learn more →

Index

Sponsored

SaaSHub - Software Alternatives and Reviews

SaaSHub helps you find the best software and product alternatives

www.saashub.com

Do not miss the trending Go projects with our weekly report!