rust-rdkafka VS nalgebra

I use this library a lot and it's got some nice touches for how to handle wrapping a C library: https://github.com/fede1024/rust-rdkafka

For this example, we're using rdkafka to build producers and consumers, because it allows us to specify custom headers for each record.

kafka

You might find https://crates.io/crates/rdkafka helpful

I am playing with tokio and rust-rdkafka library, following the examples like this one: https://github.com/fede1024/rust-rdkafka/blob/6fb2c37/examples/asynchronous_processing.rs

The source for the current version of the library can be found on Github. I had to increase the major version because I needed to break the API in order to support all formats supported by the current Schema Registry version. I also added the possibility to set an API key, so it can be used with Confluent Cloud, the cloud offering from Confluent. As part of the latest major refactoring it's also supporting async. This might improve performance of your app, and is also the default for the major Kafka client, more information about why you would want to use async can be found in the async book. The schemas retrieved from the Schema Registry are cached. This way the schema is only retrieved once for each id, and reused for other messages with the same id.

What's wrong with rust-rdkafka?

This is a two-part series to help you get started with Rust and Kafka. We will be using the rust-rdkafka crate which itself is based on librdkafka (C library).

Ok, so this uses https://rapier.rs/ which is very cool

Rapier, alongside https://nalgebra.org/ (which it uses underneath) has seriously good documentation and some advanced features like cross-platform determinism (something made hard by the way floating point differs between platforms)

> Some GA libraries[1][2] define types for the different kind of objects (grades)

That's nice!

This reminds of me things like, linear algebra libraries that will type-check matrices so that a 2x2 matrix can't be added to a 2x3 matrix (but then you can have a dynamic matrix that will error only in runtime), like https://nalgebra.org/ and others.

Maybe start by looking at https://nalgebra.org/ to see what rust math libraries might look like

And Nalgebra, while better, also seems to have slowed down on commits and responses to issues and PRs. I have a PR there for a relatively simple wrapper type for row vectors which was explicitly requested by a maintainer, which hasn't even gotten a comment since for two weeks.

nalgebra (similar to Eigen in cpp)

nalgebra

nalgebra is for linear algebra.

Take a look into math libraries, like glam, nalgebra, and cgmath. I've only used these through game engines, though, so I can't offer per-basis reviews/advice.

If you do not need GPU then I would recommend looking into Eigen in C++, nalgebra in Rust (with a BLAS in both cases for improved performance) or one of the above options (Julia / Python+JAX).