How to Build a Kafka Producer in Rust with Partitioning

• tracing

  52 4,958 7.8 Rust

  Application level tracing for Rust.

  

To showcase how to produce Kafka events, we will leverage Tokio's tracing crate to generate log data.

• templates

  12 790 7.3 Shell

  Repository for Dev Container Templates that are managed by Dev Container spec maintainers. See https://github.com/devcontainers/template-starter to create your own! (by devcontainers)

  

$ cd tracing_publisher $ mkdir .devcontainer $ cat > .devcontainer/devcontainer.json // For format details, see https://aka.ms/devcontainer.json. For config options, see the // README at: https://github.com/devcontainers/templates/tree/main/src/rust { "name": "Rust", "service": "rust-log-processing", "dockerComposeFile": "../docker-compose.yml", "features": { "ghcr.io/devcontainers/features/rust:1": {} }, "workspaceFolder": "/workspaces/${localWorkspaceFolderBasename}", "shutdownAction": "stopCompose" }

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• tracing-blog-post

  1 10 10.0 Rust

  Example code for a blog post about tracing

// Credit: https://github.com/bryanburgers/tracing-blog-post/blob/main/examples/figure_3/custom_layer.rs use std::collections::BTreeMap; use tracing_subscriber::Layer; pub struct CustomLayer; impl Layer for CustomLayer where S: tracing::Subscriber, { fn on_event( &self, event: &tracing::Event<'_>, _ctx: tracing_subscriber::layer::Context<'_, S>, ) { // Covert the values into a JSON object let mut fields = BTreeMap::new(); let mut visitor = JsonVisitor(&mut fields); event.record(&mut visitor); // Output the event in JSON let output = serde_json::json!({ "target": event.metadata().target(), "name": event.metadata().name(), "level": format!("{:?}", event.metadata().level()), "fields": fields, }); println!("{}", serde_json::to_string_pretty(&output).unwrap()); } } struct JsonVisitor<'a>(&'a mut BTreeMap); impl<'a> tracing::field::Visit for JsonVisitor<'a> { fn record_f64(&mut self, field: &tracing::field::Field, value: f64) { self.0 .insert(field.name().to_string(), serde_json::json!(value)); } fn record_i64(&mut self, field: &tracing::field::Field, value: i64) { self.0 .insert(field.name().to_string(), serde_json::json!(value)); } fn record_u64(&mut self, field: &tracing::field::Field, value: u64) { self.0 .insert(field.name().to_string(), serde_json::json!(value)); } fn record_bool(&mut self, field: &tracing::field::Field, value: bool) { self.0 .insert(field.name().to_string(), serde_json::json!(value)); } fn record_str(&mut self, field: &tracing::field::Field, value: &str) { self.0 .insert(field.name().to_string(), serde_json::json!(value)); } fn record_error( &mut self, field: &tracing::field::Field, value: &(dyn std::error::Error + 'static), ) { self.0.insert( field.name().to_string(), serde_json::json!(value.to_string()), ); } fn record_debug(&mut self, field: &tracing::field::Field, value: &dyn std::fmt::Debug) { self.0.insert( field.name().to_string(), serde_json::json!(format!("{:?}", value)), ); } }

• kcat

  18 5,253 0.0 C

  Generic command line non-JVM Apache Kafka producer and consumer

Now we don't see any additional output. To verify it worked, let's use kafkacat to consume the topic's events. (We install kafkacat in the Dev Container. Please run the following command in VSCode's terminal)

• rust-kafka-producer-partitioner-example

  2 1 1.7 Rust

  Example Code for "How to Build a Kafka Producer in Rust with Partitioning"

Find the project's source code on GitHub.

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

Suggest a related project