What are your favorite tools or components in the Kafka ecosystem?

This page summarizes the projects mentioned and recommended in the original post on /r/apachekafka

Our great sponsors
  • Mergify - Updating dependencies is time-consuming.
  • Sonar - Write Clean Python Code. Always.
  • InfluxDB - Collect and Analyze Billions of Data Points in Real Time
  • kafka-ml

    Kafka-ML: connecting the data stream with ML/AI frameworks (now TensorFlow and PyTorch!)

    For example, CLIs, UIs, monitoring tools / integrations, cluster administration, stream processing libraries (Flink, Kafka Streams, smaller / newer libs), etc? Anything in the ML / AI space (e.g. a quick Google search came up with https://github.com/ertis-research/kafka-ml).

  • console

    Redpanda Console is a developer-friendly UI for managing your Kafka/Redpanda workloads. Console gives you a simple, interactive approach for gaining visibility into your topics, masking data, managing consumer groups, and exploring real-time data with time-travel debugging. (by redpanda-data)

  • Mergify

    Updating dependencies is time-consuming.. Solutions like Dependabot or Renovate update but don't merge dependencies. You need to do it manually while it could be fully automated! Add a Merge Queue to your workflow and stop caring about PR management & merging. Try Mergify for free.

  • materialize

    Materialize is an Operational Data Warehouse: A cloud data warehouse with streaming internals, built for work that needs action on what’s happening right now. (by MaterializeInc)

  • bytewax

    Python Stream Processing

  • river

    🌊 Online machine learning in Python

    River - https://github.com/online-ml/river (Online machine learning, best used with Bytewax for Kafka integration)

  • python-fake-data-producer-for-apache-kafka

    The Python fake data producer for Apache Kafka® is a complete demo app allowing you to quickly produce JSON fake streaming datasets and push it to an Apache Kafka topic.

    Fake data utility - https://github.com/aiven/python-fake-data-producer-for-apache-kafka

  • debezium

    Change data capture for a variety of databases. Please log issues at https://issues.redhat.com/browse/DBZ.

    Debezium: https://debezium.io/ (connector for cdc)

  • Sonar

    Write Clean Python Code. Always.. Sonar helps you commit clean code every time. With over 225 unique rules to find Python bugs, code smells & vulnerabilities, Sonar finds the issues while you focus on the work.

  • datagen

    Generate authentic looking mock data based on a SQL, JSON or Avro schema and produce to Kafka in JSON or Avro format.

    For fake data, shameless plug for https://github.com/MaterializeInc/datagen/tree/main

  • conduktor-gateway

    Conduktor Gateway is a tool to intercept and then filter and transform requests and responses between clients and Apache Kafka

    They also provide an open-source Kafka proxy which can be used to enhance Kafka with 'interceptors'.

  • kloadgen

    KLoadGen is kafka load generator plugin for jmeter designed to work with AVRO, JSON and PROTOL-BUFFERS schema Registries. (by sngular)

    Sngular kloadgen for fake/synthetic data and load testing is great if you already use java/jmeter

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts