Manage all types of time series data in a single, purpose-built database. Run at any scale in any environment in the cloud, on-premises, or at the edge. Learn more →
Top 23 Python Kafka Projects
-
If you really want a "modern" language (I assume you just want Python based on your other comments), there's Robinhood's Faust, though it's been deprecated for a while. It'll still probably do what you want given your criteria, but it's not really suitable for long-term use given it hasn't been updated since October 2020.
-
kafka-python - https://github.com/dpkp/kafka-python
-
Onboard AI
Learn any GitHub repo in 59 seconds. Onboard AI learns any GitHub repo in minutes and lets you chat with it to locate functionality, understand different parts, and generate new code. Use it for free at www.getonboard.dev.
-
We have many similar ideas and paradigms, as one would expect from modern frameworks tackling the same problem. However, Faust has been deprecated and no longer managed or supported (there is a fork being maintained by the community at https://github.com/faust-streaming/faust). I guess Robinhood at some point decided not to spend any more resources on it, an often destiny of such side projects by large corporations. On the other hand, we hope to stay around for a long time and build to a framework and a community that loves it. This initial version was built using many libraries in order to have a working system and to gather feedback from the large community. We plan to replace eventually all, or at least most of dependancies, and replace then with Rust lib with Python bindings.
-
nagios-plugins
450+ AWS, Hadoop, Cloud, Kafka, Docker, Elasticsearch, RabbitMQ, Redis, HBase, Solr, Cassandra, ZooKeeper, HDFS, Yarn, Hive, Presto, Drill, Impala, Consul, Spark, Jenkins, Travis CI, Git, MySQL, Linux, DNS, Whois, SSL Certs, Yum Security Updates, Kubernetes, Cloudera etc...
-
aiokafka - https://github.com/aio-libs/aiokafka
-
Project mention: What are your favourite GitHub repos that shows how data engineering should be done? | /r/dataengineering | 2022-11-18
-
cp-all-in-one
docker-compose.yml files for cp-all-in-one , cp-all-in-one-community, cp-all-in-one-cloud, Apache Kafka Confluent Platform
Hi everyone,i'm using Kafka on Docker (https://github.com/confluentinc/cp-all-in-one/blob/7.3.3-post/cp-all-in-one/docker-compose.yml), when I run producer.py, it runs very smooth and consumer.py as well. however when I check the schema-register at localhost:8081 it is null and so is the Confluent Ui (localhost:9021). Is there anything missing? Thanks for your help!
-
Revelo Payroll
Free Global Payroll designed for tech teams. Building a great tech team takes more than a paycheck. Zero payroll costs, get AI-driven insights to retain best talent, and delight them with amazing local benefits. 100% free and compliant.
-
faststream
FastStream is a powerful and easy-to-use Python framework for building asynchronous services that interact with event streams such as Apache Kafka and RabbitMQ.
Project mention: Introducing FastStream: the easiest way to write microservices for Apache Kafka and RabbitMQ in Python | /r/opensource | 2023-09-29FastStream simplifies the process of writing producers and consumers for message queues, handling all the parsing, networking and documentation generation automatically. It is a new package based on the ideas and experiences gained from FastKafka and Propan. By joining our forces, we picked up the best from both packages and created a unified way to write services capable of processing streamed data regardless of the underlying protocol. We'll continue to maintain both packages, but new development will be in this project.
-
-
Propan
Propan is a powerful and easy-to-use Python framework for building event-driven applications that interact with any MQ Broker
Project mention: Introducing FastStream: the easiest way to write microservices for Apache Kafka and RabbitMQ in Python | /r/opensource | 2023-09-29FastStream simplifies the process of writing producers and consumers for message queues, handling all the parsing, networking and documentation generation automatically. It is a new package based on the ideas and experiences gained from FastKafka and Propan. By joining our forces, we picked up the best from both packages and created a unified way to write services capable of processing streamed data regardless of the underlying protocol. We'll continue to maintain both packages, but new development will be in this project.
-
streamify
A data engineering project with Kafka, Spark Streaming, dbt, Docker, Airflow, Terraform, GCP and much more!
-
Project mention: How to build a custom GPT enabled full-stack app for real-time data | dev.to | 2023-09-05
Maintain a data snapshot to observe variations in sales prices over time, as Pathway provides a built-in feature to compute differences between two alterations.
-
Project mention: What are your favorite tools or components in the Kafka ecosystem? | /r/apachekafka | 2023-05-31
For example, CLIs, UIs, monitoring tools / integrations, cluster administration, stream processing libraries (Flink, Kafka Streams, smaller / newer libs), etc? Anything in the ML / AI space (e.g. a quick Google search came up with https://github.com/ertis-research/kafka-ml).
-
inferencedb
🚀 Stream inferences of real-time ML models in production to any data lake (Experimental)
-
-
python-fake-data-producer-for-apache-kafka
The Python fake data producer for Apache Kafka® is a complete demo app allowing you to quickly produce JSON fake streaming datasets and push it to an Apache Kafka topic.
One of the hardest questions we answer for our large enterprise customers is why they should choose Aiven instead of managing their own database and streaming services. It can seem counterintuitive that paying extra for a managed service can save you money. However, when we factor in economies of scale - particularly in regards to access to specialized knowledge and tooling - the case for managed services becomes clear. This was certainly the case for some of our MySQL clients earlier this year, where their investments in Aiven paid off in the form of a quietly managed bug fix.
-
-
-
-
-
-
Our team spent Hackathon week building a graph application on top of streaming Slack data. If you want to jump right to the code, check out the GitHub repo, and if you want to learn more about it, join our Discord Community Chat!
-
Realtime-ticket-booking-solution
A simple demo showing how to use Ably and fastAPI to route messages into Kafka for stream processing
-
InfluxDB
Collect and Analyze Billions of Data Points in Real Time. Manage all types of time series data in a single, purpose-built database. Run at any scale in any environment in the cloud, on-premises, or at the edge.
NOTE:
The open source projects on this list are ordered by number of github stars.
The number of mentions indicates repo mentiontions in the last 12 Months or
since we started tracking (Dec 2020).
The latest post mention was on 2023-09-29.
Python Kafka related posts
- FLaNK Stack Weekly 2 October 2023
- Introducing FastStream: the easiest way to write microservices for Apache Kafka and RabbitMQ in Python
- FastStream: the easiest way to add Kafka and RabbitMQ support to FastAPI services
- FastStream: Effortless Kafka/RabbitMQ Event Stream Integration for Microservices
- Announcing pyKLI: interactive command line client for ksqlDB
- Show HN: LLM App – build a realtime LLM app in 30 lines, with no vector database
- Embed graphql server in nginx (aggregate multiple data sources: restful, webservice, database, kafka, etc.)
-
A note from our sponsor - InfluxDB
www.influxdata.com | 4 Oct 2023
Index
What are some of the best open-source Kafka projects in Python? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | Faust | 6,600 |
| 2 | kafka-python | 5,316 |
| 3 | faust | 1,215 |
| 4 | nagios-plugins | 1,107 |
| 5 | aiokafka | 935 |
| 6 | DataEngineeringProject | 874 |
| 7 | cp-all-in-one | 764 |
| 8 | faststream | 726 |
| 9 | KQ | 558 |
| 10 | Propan | 436 |
| 11 | streamify | 325 |
| 12 | pathway | 126 |
| 13 | kafka-ml | 115 |
| 14 | inferencedb | 75 |
| 15 | strimzi-kafka-cli | 66 |
| 16 | python-fake-data-producer-for-apache-kafka | 64 |
| 17 | spark_app_twitter | 58 |
| 18 | kafka-crypto-questdb | 56 |
| 19 | kafkaml-anomaly-detection | 43 |
| 20 | makinage | 37 |
| 21 | esque | 23 |
| 22 | slack-influence-bot | 20 |
| 23 | Realtime-ticket-booking-solution | 14 |
Write Clean Python Code. Always.
Sonar helps you commit clean code every time. With over 225 unique rules to find Python bugs, code smells & vulnerabilities, Sonar finds the issues while you focus on the work.
www.sonarsource.com