kafka-connect-datagen VS Apache Avro

The first part will keep things relatively simple - it's all about get started easily. I will be using the Kafka Connect Datagen source connector to pump data some sample data into MSK topic and then use the AWS DynamoDB sink connector to persist that data in a DynamoDB table.

For the remaining scenarios, we will use a producer component to generate records. The Kafka Connect Datagen connector is our friend. It is meant for generating mock data for testing, so let’s put it to good use!

Neither kafka-connect-datagen nor voluble offer the ability to generate headers, unfortunately. It's a cool idea though.

Apache AVRO [1] is one but it has been largely replaced by Parquet [2] which is a hybrid row/columnar format

[1] https://avro.apache.org/

The most common format for describing schema in this scenario is Apache Avro.

I am testing a new schema registry which loads and retrieves different kinds of avro schemas. In the process of testing, I need to create a bunch of different types of avro schemas. As it involves a lot of permutations, I decided to create the schema programmatically.I am using the apache avro SchemaBuilder to do so.

Other serialization alternatives have a schema validation option: e.g., Avro, Kryo and Protocol Buffers. Interestingly enough, gRPC uses Protobuf to offer RPC across distributed components:

Apache Avro is a data serialization system, for more information visit Apache Avro

Once things like JSON became more popular Apache Avro appeared. You can define Avro files which can then be generated into Python, Java C, Ruby, etc.. classes.

Avro, a data serialization system based on JSON schemas.

Supporting multiple versions of an event schema is a solved problem. Apache Avro with a published schema hash in a message header is one solution.

https://avro.apache.org/