Python Kafka

Open-source Python projects categorized as Kafka

Top 23 Python Kafka Projects

  • Faust

    Python Stream Processing

    Project mention: Kafka ETL tool, is there any? | /r/apachekafka | 2023-02-14

    If you really want a "modern" language (I assume you just want Python based on your other comments), there's Robinhood's Faust, though it's been deprecated for a while. It'll still probably do what you want given your criteria, but it's not really suitable for long-term use given it hasn't been updated since October 2020.

  • kafka-python

    Python client for Apache Kafka

    Project mention: Improving Kafka interfaces | /r/apachekafka | 2023-06-07

    kafka-python - https://github.com/dpkp/kafka-python

  • Onboard AI

    Learn any GitHub repo in 59 seconds. Onboard AI learns any GitHub repo in minutes and lets you chat with it to locate functionality, understand different parts, and generate new code. Use it for free at www.getonboard.dev.

  • faust

    Python Stream Processing. A Faust fork (by faust-streaming)

    Project mention: New release of FastKafka supporting Redpanda | /r/Python | 2023-03-24

    We have many similar ideas and paradigms, as one would expect from modern frameworks tackling the same problem. However, Faust has been deprecated and no longer managed or supported (there is a fork being maintained by the community at https://github.com/faust-streaming/faust). I guess Robinhood at some point decided not to spend any more resources on it, an often destiny of such side projects by large corporations. On the other hand, we hope to stay around for a long time and build to a framework and a community that loves it. This initial version was built using many libraries in order to have a working system and to gather feedback from the large community. We plan to replace eventually all, or at least most of dependancies, and replace then with Rust lib with Python bindings.

  • nagios-plugins

    450+ AWS, Hadoop, Cloud, Kafka, Docker, Elasticsearch, RabbitMQ, Redis, HBase, Solr, Cassandra, ZooKeeper, HDFS, Yarn, Hive, Presto, Drill, Impala, Consul, Spark, Jenkins, Travis CI, Git, MySQL, Linux, DNS, Whois, SSL Certs, Yum Security Updates, Kubernetes, Cloudera etc...

  • aiokafka

    asyncio client for kafka

    Project mention: Improving Kafka interfaces | /r/apachekafka | 2023-06-07

    aiokafka - https://github.com/aio-libs/aiokafka

  • DataEngineeringProject

    Example end to end data engineering project.

    Project mention: What are your favourite GitHub repos that shows how data engineering should be done? | /r/dataengineering | 2022-11-18
  • cp-all-in-one

    docker-compose.yml files for cp-all-in-one , cp-all-in-one-community, cp-all-in-one-cloud, Apache Kafka Confluent Platform

    Project mention: Apache Kafka Using Docker | /r/apachekafka | 2023-04-26

    Hi everyone,i'm using Kafka on Docker (https://github.com/confluentinc/cp-all-in-one/blob/7.3.3-post/cp-all-in-one/docker-compose.yml), when I run producer.py, it runs very smooth and consumer.py as well. however when I check the schema-register at localhost:8081 it is null and so is the Confluent Ui (localhost:9021). Is there anything missing? Thanks for your help!

  • Revelo Payroll

    Free Global Payroll designed for tech teams. Building a great tech team takes more than a paycheck. Zero payroll costs, get AI-driven insights to retain best talent, and delight them with amazing local benefits. 100% free and compliant.

  • faststream

    FastStream is a powerful and easy-to-use Python framework for building asynchronous services that interact with event streams such as Apache Kafka and RabbitMQ.

    Project mention: Introducing FastStream: the easiest way to write microservices for Apache Kafka and RabbitMQ in Python | /r/opensource | 2023-09-29

    FastStream simplifies the process of writing producers and consumers for message queues, handling all the parsing, networking and documentation generation automatically. It is a new package based on the ideas and experiences gained from FastKafka and Propan. By joining our forces, we picked up the best from both packages and created a unified way to write services capable of processing streamed data regardless of the underlying protocol. We'll continue to maintain both packages, but new development will be in this project.

  • KQ

    Kafka-based Job Queue for Python

  • Propan

    Propan is a powerful and easy-to-use Python framework for building event-driven applications that interact with any MQ Broker

    Project mention: Introducing FastStream: the easiest way to write microservices for Apache Kafka and RabbitMQ in Python | /r/opensource | 2023-09-29

    FastStream simplifies the process of writing producers and consumers for message queues, handling all the parsing, networking and documentation generation automatically. It is a new package based on the ideas and experiences gained from FastKafka and Propan. By joining our forces, we picked up the best from both packages and created a unified way to write services capable of processing streamed data regardless of the underlying protocol. We'll continue to maintain both packages, but new development will be in this project.

  • streamify

    A data engineering project with Kafka, Spark Streaming, dbt, Docker, Airflow, Terraform, GCP and much more!

    Project mention: Where can I find online projects end-to-end? | /r/dataengineering | 2023-03-21
  • pathway

    Pathway is an open framework for high-throughput and low-latency real-time data processing.

    Project mention: How to build a custom GPT enabled full-stack app for real-time data | dev.to | 2023-09-05

    Maintain a data snapshot to observe variations in sales prices over time, as Pathway provides a built-in feature to compute differences between two alterations.

  • kafka-ml

    Kafka-ML: connecting the data stream with ML/AI frameworks (now TensorFlow and PyTorch!)

    Project mention: What are your favorite tools or components in the Kafka ecosystem? | /r/apachekafka | 2023-05-31

    For example, CLIs, UIs, monitoring tools / integrations, cluster administration, stream processing libraries (Flink, Kafka Streams, smaller / newer libs), etc? Anything in the ML / AI space (e.g. a quick Google search came up with https://github.com/ertis-research/kafka-ml).

  • inferencedb

    🚀 Stream inferences of real-time ML models in production to any data lake (Experimental)

  • strimzi-kafka-cli

    Command Line Interface for the Strimzi Kafka Operator

  • python-fake-data-producer-for-apache-kafka

    The Python fake data producer for Apache Kafka® is a complete demo app allowing you to quickly produce JSON fake streaming datasets and push it to an Apache Kafka topic.

    Project mention: Scale up: a MySQL bug story, or why Aiven works | dev.to | 2023-07-07

    One of the hardest questions we answer for our large enterprise customers is why they should choose Aiven instead of managing their own database and streaming services. It can seem counterintuitive that paying extra for a managed service can save you money. However, when we factor in economies of scale - particularly in regards to access to specialized knowledge and tooling - the case for managed services becomes clear. This was certainly the case for some of our MySQL clients earlier this year, where their investments in Aiven paid off in the form of a quietly managed bug fix.

  • spark_app_twitter

    A data engineering project (Twitter monitor app)

  • kafka-crypto-questdb

    Using Kafka to track cryptocurrency price trends

  • kafkaml-anomaly-detection

    Project for real-time anomaly detection using Kafka and python

  • makinage

    Stream Processing Made Easy

  • esque

    esque - an operational kafka tool.

  • slack-influence-bot

    A slack bot that helps you understand and influence your slack community

    Project mention: Analyzing Interactions in a Slack Communication Network | dev.to | 2023-03-14

    Our team spent Hackathon week building a graph application on top of streaming Slack data. If you want to jump right to the code, check out the GitHub repo, and if you want to learn more about it, join our Discord Community Chat!

  • Realtime-ticket-booking-solution

    A simple demo showing how to use Ably and fastAPI to route messages into Kafka for stream processing

  • InfluxDB

    Collect and Analyze Billions of Data Points in Real Time. Manage all types of time series data in a single, purpose-built database. Run at any scale in any environment in the cloud, on-premises, or at the edge.

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2023-09-29.

Python Kafka related posts

Index

What are some of the best open-source Kafka projects in Python? This list will help you:

Project Stars
1 Faust 6,600
2 kafka-python 5,316
3 faust 1,215
4 nagios-plugins 1,107
5 aiokafka 935
6 DataEngineeringProject 874
7 cp-all-in-one 764
8 faststream 726
9 KQ 558
10 Propan 436
11 streamify 325
12 pathway 126
13 kafka-ml 115
14 inferencedb 75
15 strimzi-kafka-cli 66
16 python-fake-data-producer-for-apache-kafka 64
17 spark_app_twitter 58
18 kafka-crypto-questdb 56
19 kafkaml-anomaly-detection 43
20 makinage 37
21 esque 23
22 slack-influence-bot 20
23 Realtime-ticket-booking-solution 14
Write Clean Python Code. Always.
Sonar helps you commit clean code every time. With over 225 unique rules to find Python bugs, code smells & vulnerabilities, Sonar finds the issues while you focus on the work.
www.sonarsource.com