Our great sponsors
-
Use the header for metadata about the payload, such as the OpenTelemetry trace IDs. It can also be useful to duplicate some of the fields from the payload itself, if they are used for routing or filtering the data. In secure systems, intermediate components may not have access to the whole payload, so putting the data in the header can expose just the appropriate fields there. Also consider that, for larger payloads, the overhead of deserializing can be non-trivial. Being able to access just a couple of fields while keeping the system moving can help performance, too.
-
There are other alternatives, notably Protocol Buffers, known as ProtoBuf. It achieves similar goals by generating code to use in your own application, making it available on fewer tech stacks. If it's available for yours, it's worth a look.
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
Avro: Small and schema-driven Apache Avro is a serialisation system that keeps the data tidy and small, which is ideal for Kafka records. The data structure is described with a schema (example below) and messages can only be created if they conform with the requirements of the schema. The producer takes the data and the schema, produces a message that goes to the kafka broker, and registers the schema with a schema registry. The consumers do the same in reverse: take the message, ask the schema registry for the schema, and assemble the full data structure. Avro has a strong respect for data types, requires all payloads conform with the schema, and since data such as fieldnames is encoded in the schema rather than repeated in every payload, the overall payload size is reduced.
