kafka-lag-exporter VS jmx_exporter

- https://github.com/prometheus/jmx_exporter - https://github.com/lightbend/kafka-lag-exporter (Export Kafka Topics Lag Metrics) - https://github.com/danielqsj/kafka_exporter (Export Kafka Metrics) - https://github.com/obsidiandynamics/kafdrop (Topic Administration tool - create/edit/delete topics & view messages)

Kafka Lag Exporter is not part of the Apache Kafka project nor the Confluent Platform. It’s an open-source project under Apache-2.0 License to export Consumer Lag with reporters like Prometheus, Graphite, or InfluxDB. Kafka Lag Exporter enables to monitor Consumer Lag but it also allows to estimate the Time Lag. This metric shows how far a Consumer group is behind the last produced record in terms of time. It shows the actual latency of a consumer application. For more information about this estimation, you can read the chapter of the documentation Estimate Consumer Group Time Lag

not quite sure if this is where I should be asking for help but i'm kind of befuddled about this whole situation. I'm trying to set up https://github.com/prometheus/jmx_exporter for our containerized Java application on our cluster. Specifically the JavaAgent as we are interested in getting the CPU and memory metrics especially. However, getting it initialized I am faced with this:

Prometheus doesn't deal with logs at all - only metrics. Even just for metrics, you have several different choices depending on how your Java application is running. You may be able to use the JMX Exporter to expose the metrics that you would then scrape with Prometheus. For the logs, you'll need something like Fluent Bit as an agent to grab ('tail' in Fluent Bit input terms) the logs and send them to Elasticsearch or Grafana Loki.

- https://github.com/prometheus/jmx_exporter - https://github.com/lightbend/kafka-lag-exporter (Export Kafka Topics Lag Metrics) - https://github.com/danielqsj/kafka_exporter (Export Kafka Metrics) - https://github.com/obsidiandynamics/kafdrop (Topic Administration tool - create/edit/delete topics & view messages)

Usually I use a javaagent that transforms the JMX metrics into prometheus style metrics, for example: https://github.com/prometheus/jmx_exporter/blob/116a1c4b3fafcfbb9e566b34e130ccc805f331fc/example_configs/kafka-0-8-2.yml

Another good one is to add the jmx_exporter agent plugin. This allows easy collection of real-time performance metrics.

In our case, we add sink connectors for AWS Kinesis and Google Cloud PubSub. We also add a Prometheus exporter JAR that scrapes the Kafka Connect JMX metrics and exposes them as Prometheus metrics.