xk6-kafka
singlestore-logistics-sim
Our great sponsors
xk6-kafka | singlestore-logistics-sim | |
---|---|---|
4 | 5 | |
125 | 32 | |
- | - | |
6.3 | 1.8 | |
9 days ago | over 2 years ago | |
Go | Go | |
Apache License 2.0 | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
xk6-kafka
-
Performance testing Kafka server using xk6-kafka
import { Writer, Reader, SCHEMA_TYPE_STRING, SchemaRegistry, GROUP_BALANCER_ROUND_ROBIN, BALANCER_ROUND_ROBIN, } from "k6/x/kafka"; import { check, sleep } from "k6"; const bootstrapServers = [ 'kafka-bootsrap-1:9001', ]; let vus_amount = 3; let total_written = 0; let total_read = 0; export const options = { vus: vus_amount, iterations: "3", thresholds: { kafka_writer_error_count: ["count == 0"], kafka_reader_error_count: ["count == 0"], }, }; const topicName = "my-topic"; const schemaRegistry = new SchemaRegistry(); const producer = new Writer({ brokers: bootstrapServers, topic: topicName, balancer: BALANCER_ROUND_ROBIN, // or pick another balancer https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L66 // ... auth config }); export function teardown(data) { producer.close(); } export default function () { const consumer = new Reader({ brokers: bootstrapServers, // it is important to set groupID, groupTopics and groupBalancers when using Kafka bootstrap server // topic ReaderConfig param doesn't quite work with bootstrap server groupID: topicName + "-group", groupTopics: [topicName], groupBalancers: [GROUP_BALANCER_ROUND_ROBIN], // or pick different balancer https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L75 }); let messageAmount = 1000; if (__VU == 1) { console.log('VU 1, writing messages. Iter ' + __ITER); for (let index = 0; index < messageAmount; index++) { let messages = [ { value: schemaRegistry.serialize({ data: "test-value-string-" + index + "-vu-" + __VU + "-iter-" + __ITER, schemaType: SCHEMA_TYPE_STRING, }), }, ]; producer.produce({ messages: messages }); total_written += messages.length; } } let consumerMsgAmount = Math.floor(messageAmount / vus_amount); let messages = consumer.consume({ limit: consumerMsgAmount}); total_read += messages.length; console.log("Amount of msgs received: " + messages.length + ", VU " + __VU + ", iter " + __ITER); check(messages, { "all messages returned": (msgs) => msgs.length == consumerMsgAmount, }); let firstMessageValue = schemaRegistry.deserialize({ data: messages[0].value, schemaType: SCHEMA_TYPE_STRING, }); let lastMessageValue = schemaRegistry.deserialize({ data: messages[consumerMsgAmount - 1].value, schemaType: SCHEMA_TYPE_STRING, }); check(messages[0], { "Topic equals to": (msg) => msg["topic"] == topicName }); console.log("First msg value " + firstMessageValue + ", offset" + messages[0]["offset"] + ", partition " + messages[0]["partition"] + ", VU " + __VU + ", iter " + __ITER); console.log("Last msg value " + lastMessageValue + ", offset" + messages[consumerMsgAmount - 1]["offset"] + ", partition " + messages[0]["partition"] + ", VU " + __VU + ", iter " + __ITER); consumer.close(); }
-
🔥 What is Apache Kafka and how to perform performance tests on it (Part 2) 👨🏻💻
Apache Kafka, like any other service, platform or server, has its performance limitations. In order to know them, you need to run performance tests on Apache Kafka's Cluster, which will show acceptation thresholds. One project that can help with this is xk6-kafka.
-
xk6-kafka v0.12.0 is released! 🎉
I am thrilled to announce that I released xk6-kafka v0.12.0, a complete revamp of the JavaScript API plus many other features and fixes. You can read more about it on the changelog of the release: https://github.com/mostafa/xk6-kafka/releases/tag/v0.12.0
-
How to Load Test Your Kafka Producers and Consumers using k6
The xk6-kafka extension provides some convenient functions for interacting with Kafka Producers and Consumers. It serves as a producer that can send a high volume of messages per second, allowing you to monitor the system under test (SUT) and test how the application will keep up with the load.
singlestore-logistics-sim
-
BigQuery alternatives and the future of data warehousing
Logistics Github Repo
-
I built a reference architecture for global package logistics and learned a bunch about Terraform in the process + scaled up to 400k packages delivered per second!
tldr; step by step instructions and Terraform module
- I built an e2e simulation of global package logistics... then scaled it to 125x yearly volume! Code, terraform, and dashboards included - run everything in docker locally or deploy it to Google Cloud
- Decided to simulate global package delivery on my computer... Ended up scaling it up to 125x!
-
Scaling Worldwide Parcel Logistics with SingleStore and Vectorized
Any technical blog post is only as good as the code, so I encourage you to check it out here: https://github.com/singlestore-labs/singlestore-logistics-si.... You can spin it up on your computer using Docker or you can deploy it directly to Google Cloud using Terraform by following the instructions in the readme.
What are some alternatives?
k6-reporter - Output K6 test run results as formatted & easy to read HTML reports
srclient - Golang Client for Schema Registry
ddosify - Effortless Kubernetes Monitoring and Performance Testing. Available on CLI, Self-Hosted, and Cloud
gcsim - monte carlo combat simulation for genshin impact
postman-to-k6 - Converts Postman collections to k6 script code
avro - Apache AVRO for go
xk6 - Build k6 with extensions
examples - Apache Kafka and Confluent Platform examples and demos
xk6-redis - A k6 extension to test the performance of a Redis instance.
hugedbbench - huge list of database benchmark
kafkarest-processor - Apache NiFi 1.10.0 Processor to consume 1 Kafka message at a time, easily to tie into a REST Proxy
kafkactl - Command Line Tool for managing Apache Kafka