OpenTelemetry at Scale: Using Kafka to handle bursty traffic

• opentelemetry-collector-contrib

  43 2,546 10.0 Go

  Contrib repository for the OpenTelemetry Collector

  

This arch is how the big players do it at scale (ie. datadog, new relic - the second it passes their edge it lands in a kafka queue). Also otel components lack rate limiting(1) meaning its super easy to overload your backend storage (s3).

Grafana has some posts how they softened the s3 blow with memcached(2,3).

1. https://github.com/open-telemetry/opentelemetry-collector-co...

• opentelemetry-collector-co

  10 - -

This arch is how the big players do it at scale (ie. datadog, new relic - the second it passes their edge it lands in a kafka queue). Also otel components lack rate limiting(1) meaning its super easy to overload your backend storage (s3).

Grafana has some posts how they softened the s3 blow with memcached(2,3).

1. https://github.com/open-telemetry/opentelemetry-collector-co...

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

Suggest a related project