Top 23 Python Kafka Projects

• Faust

  8 6,675 1.4 Python

  Python Stream Processing

  

Project mention: Faust VS quix-streams - a user suggested alternative | libhunt.com/r/faust | 2023-12-07

• kafka-python

  8 5,489 6.4 Python

  Python client for Apache Kafka

Project mention: kafka-python VS quix-streams - a user suggested alternative | libhunt.com/r/kafka-python | 2023-12-07

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• faststream

  13 1,802 9.7 Python

  FastStream is a powerful and easy-to-use Python framework for building asynchronous services interacting with event streams such as Apache Kafka, RabbitMQ, NATS and Redis.

  

Project mention: FastStream v0.4.0: Introducing Confluent Kafka Integration with Async Support | news.ycombinator.com | 2024-01-30

• pathway

  3 1,733 9.6 Python

  Pathway is a high-throughput, low-latency data processing framework that handles live data & streaming for you. Made with ❤️ for Python & ML/AI developers.

  

Project mention: Choosing Between a Streaming Database and a Stream Processing Framework in Python | dev.to | 2024-02-10

We understood how streaming databases differ from traditional databases, stream processing engines, conventional analytics databases, or OLAP databases. Now let’s focus on when and why we can use stream data processing frameworks for Python as an alternative to streaming databases. Python is the go-to language for data science and machine learning. There are some stream-processing libraries and frameworks in Python such as Bytewax, Quix, GlassFlow, Pathway. They have been developed to cope with the challenges Python Engineers face with Apache Kafka or Flink since they do not natively support Python.

• faust

  11 1,454 7.9 Python

  Python Stream Processing. A Faust fork (by faust-streaming)

  

Project mention: faust VS quix-streams - a user suggested alternative | libhunt.com/r/faust-streaming/faust | 2023-12-07

• nagios-plugins

  2 1,119 8.4 Python

  450+ AWS, Hadoop, Cloud, Kafka, Docker, Elasticsearch, RabbitMQ, Redis, HBase, Solr, Cassandra, ZooKeeper, HDFS, Yarn, Hive, Presto, Drill, Impala, Consul, Spark, Jenkins, Travis CI, Git, MySQL, Linux, DNS, Whois, SSL Certs, Yum Security Updates, Kubernetes, Cloudera etc...

• aiokafka

  3 1,051 8.6 Python

  asyncio client for kafka

  

Project mention: FastStream: Python's framework for Efficient Message Queue Handling | dev.to | 2023-10-16

Our journey with FastStream started when we needed to integrate our machine learning models into a customer's Apache Kafka environment. To streamline this process, we created FastKafka using AIOKafka, AsyncAPI, and asyncio. It was our first step in making message queue management easier.

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

• DataEngineeringProject

  5 985 0.0 Python

  Example end to end data engineering project.

• cp-all-in-one

  9 882 8.3 Python

  docker-compose.yml files for cp-all-in-one , cp-all-in-one-community, cp-all-in-one-cloud, Apache Kafka Confluent Platform

  

Project mention: My local Kafka instance stuck in "auto leader balancing" | /r/apachekafka | 2023-11-22

# https://github.com/confluentinc/cp-all-in-one/blob/7.0.1-post/cp-all-in-one/docker-compose.yml version: '3' services: zookeeper: image: confluentinc/cp-zookeeper:7.3.0 container_name: zookeeper ports: - "2181:2181" environment: ZOOKEEPER_CLIENT_PORT: 2181 ZOOKEEPER_TICK_TIME: 2000 broker: image: confluentinc/cp-kafka:7.3.0 container_name: broker ports: - "9092:9092" depends_on: - zookeeper environment: KAFKA_BROKER_ID: 1 KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181" KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092 KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1 KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1 mongodb: container_name: mongo_c image: mongo:6.0 volumes: - ./db:/data/db ports: - "27017:27017" environment: MONGO_INITDB_ROOT_USERNAME: root MONGO_INITDB_ROOT_PASSWORD: example

• quix-streams

  25 692 9.0 Python

  A Python library for building containerized ML and Generative AI applications with Apache Kafka.

  

Project mention: Show HN: Streaming DataFrames–a Pandas-like syntax for real-time data | news.ycombinator.com | 2024-04-23

• KQ

  1 571 0.0 Python

  Kafka-based Job Queue for Python

• streamify

  4 474 0.0 Python

  A data engineering project with Kafka, Spark Streaming, dbt, Docker, Airflow, Terraform, GCP and much more!

• Propan

  16 471 8.8 Python

  Propan is a powerful and easy-to-use Python framework for building event-driven applications that interact with any MQ Broker

Project mention: FastStream: Python's framework for Efficient Message Queue Handling | dev.to | 2023-10-16

Later, we discovered Propan, a library created by Nikita Pastukhov, which solved similar problems but for RabbitMQ. Recognizing the potential for collaboration, we joined forces with Nikita to build a unified library that could work seamlessly with both Kafka and RabbitMQ. And that's how FastStream came to be—a solution born out of the need for simplicity and efficiency in microservices development.

• clickhouse-sink-connector

  1 175 9.8 Python

  Replicate data from MySQL, Postgres and MongoDB to ClickHouse

  

• kafka-ml

  1 147 7.9 Python

  Kafka-ML: connecting the data stream with ML/AI frameworks (now TensorFlow and PyTorch!)

  

Project mention: What are your favorite tools or components in the Kafka ecosystem? | /r/apachekafka | 2023-05-31

For example, CLIs, UIs, monitoring tools / integrations, cluster administration, stream processing libraries (Flink, Kafka Streams, smaller / newer libs), etc? Anything in the ML / AI space (e.g. a quick Google search came up with https://github.com/ertis-research/kafka-ml).

• inferencedb

  9 77 0.0 Python

  🚀 Stream inferences of real-time ML models in production to any data lake (Experimental)

  

• strimzi-kafka-cli

  1 76 8.9 Python

  Command Line Interface for the Strimzi Kafka Operator

  

• python-fake-data-producer-for-apache-kafka

  32 77 2.7 Python

  The Python fake data producer for Apache Kafka® is a complete demo app allowing you to quickly produce JSON fake streaming datasets and push it to an Apache Kafka topic.

  

Project mention: ElephantSQL Is Shutting Down | news.ycombinator.com | 2024-04-07

I had good experience with Aiven in the past, we needed something located in the EU: https://aiven.io/

• spark_app_twitter

  3 60 0.0 Python

  A data engineering project (Twitter monitor app)

• kafka-crypto-questdb

  1 58 1.7 Python

  Using Kafka to track cryptocurrency price trends

• journalpump

  1 58 6.0 Python

  systemd journald to aws_cloudwatch, elasticsearch, google cloud logging, kafka, rsyslog or logplex log sender

  

Project mention: FLaNK Stack Weekly for 20 Nov 2023 | dev.to | 2023-11-20

• kafkaml-anomaly-detection

  1 48 0.0 Python

  Project for real-time anomaly detection using Kafka and python

• makinage

  4 38 2.7 Python

  Stream Processing Made Easy

  

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

Python Kafka related posts

• Show HN: Streaming DataFrames–a Pandas-like syntax for real-time data

  1 project | news.ycombinator.com | 23 Apr 2024

• 🦿🛴Smarcity garbage reporting automation w/ ollama

  6 projects | dev.to | 31 Jan 2024

• FastStream v0.4.0: Introducing Confluent Kafka Integration with Async Support

  1 project | news.ycombinator.com | 30 Jan 2024

• Easy Guide to Integrating Kafka: Practical Solutions for Managing Blob Data

  5 projects | dev.to | 22 Jan 2024

• Show HN: Confluent Kafka support added to FastStream v0.4.0rc0

  2 projects | news.ycombinator.com | 12 Jan 2024

• Knative switchboard series, part 1. Setup Knative Eventing with Kafka from scratch, scale based on events volume, and monitor

  4 projects | dev.to | 4 Jan 2024

• faust VS quix-streams - a user suggested alternative

  2 projects | 7 Dec 2023
• A note from our sponsor - InfluxDB
  www.influxdata.com | 4 May 2024

  Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality. Learn more →

Index

Sponsored

SaaSHub - Software Alternatives and Reviews

SaaSHub helps you find the best software and product alternatives

www.saashub.com

Do not miss the trending Python projects with our weekly report!