SaaSHub helps you find the best software and product alternatives Learn more →
Top 23 Python Kafka Projects
-
Project mention: Show HN: Pathway – Build Mission Critical ETL and RAG in Python (NATO, F1 Used) | news.ycombinator.com | 2024-06-13
The main factor impacting the RAM requirement of the instance is the size of the data that you feed into it, especially if you need an in-memory index. (If you are curious about peak memory use etc., you can profile Pathway memory use in Grafana: https://github.com/pathwaycom/pathway/tree/main/examples/pro....)
One point to clarify is that "Pathway Community" is self-hosted, and the "8GB RAM - 4 cores" value is just a limit on the dimension of your own/cloud machine that the framework will effectively use. Currently, if you would like to get a "free" cloud machine to go with your project, we suggest going for "Pathway Scale" and reaching out through the #Developer Assist link - add a mention that you are interested in cloud credits. You can also go with 3rd party hosting providers like http://render.com/ who have a (somewhat modest) free tier for Docker instances, or reasonably priced ones like fly.io https://fly.io/docs/about/pricing/.
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
-
-
-
faststream
FastStream is a powerful and easy-to-use Python framework for building asynchronous services interacting with event streams such as Apache Kafka, RabbitMQ, NATS and Redis.
Project mention: FastStream: A powerful library for building services with event streams | news.ycombinator.com | 2024-10-29FastStream (https://github.com/airtai/faststream) simplifies the process of writing producers and consumers for message queues, handling all the parsing, networking and documentation generation automatically. It is a new package based on the ideas and experiences gained from FastKafka and Propan. By joining our forces, we picked up the best from both packages and created a unified way to write services capable of processing streamed data regardless of the underlying protocol. We'll continue to maintain both packages, but new development will be in this project.
Making streaming microservices has never been easier. Designed with junior developers in mind, FastStream simplifies your work while keeping the door open for more advanced use cases. Here's a look at the core features that make FastStream a go-to framework for modern, data-centric microservices.
Multiple Brokers: FastStream provides a unified API to work across multiple message brokers (Apache Kafka, RabbitMQ, NATS and Redis)
-
-
quix-streams
A Python library for building containerized ML and Generative AI applications with Apache Kafka.
Project mention: Show HN: Denormalized – Embeddable Stream Processing in Rust and DataFusion | news.ycombinator.com | 2024-08-15Congratulations on launching your project! We spoke back in March at a Kafka Summit London social meetup and talked all things Python and Kafka (I work on https://github.com/quixio/quix-streams). Always great to see a new stream processing project tackle a new segment
-
-
-
nagios-plugins
450+ AWS, Hadoop, Cloud, Kafka, Docker, Elasticsearch, RabbitMQ, Redis, HBase, Solr, Cassandra, ZooKeeper, HDFS, Yarn, Hive, Presto, Drill, Impala, Consul, Spark, Jenkins, Travis CI, Git, MySQL, Linux, DNS, Whois, SSL Certs, Yum Security Updates, Kubernetes, Cloudera etc...
-
cp-all-in-one
docker-compose.yml files for cp-all-in-one , cp-all-in-one-community, cp-all-in-one-cloud, Apache Kafka Confluent Platform
-
kaskade
kaskade is a text user interface for kafka, which allows you to interact and consume topics from your terminal in style!
-
-
streamify
A data engineering project with Kafka, Spark Streaming, dbt, Docker, Airflow, Terraform, GCP and much more!
-
Propan
Propan is a powerful and easy-to-use Python framework for building event-driven applications that interact with any MQ Broker
-
-
-
-
-
python-fake-data-producer-for-apache-kafka
The Python fake data producer for Apache Kafka® is a complete demo app allowing you to quickly produce JSON fake streaming datasets and push it to an Apache Kafka topic.
Aiven
-
inferencedb
🚀 Stream inferences of real-time ML models in production to any data lake (Experimental)
-
-
-
journalpump
systemd journald to aws_cloudwatch, elasticsearch, google cloud logging, kafka, rsyslog or logplex log sender
NOTE:
The open source projects on this list are ordered by number of github stars.
The number of mentions indicates repo mentiontions in the last 12 Months or
since we started tracking (Dec 2020).
Python Kafka discussion
Python Kafka related posts
-
Show HN: Kaskade version 3 was released
-
FastStream: A powerful library for building services with event streams
-
Build a real-time crypto analytics dashboard with Beavers and Perspective
-
Industry Standard for Cloud Instance Initialization: Cloud-Init
-
Show HN: Streaming DataFrames–a Pandas-like syntax for real-time data
-
🦿🛴Smarcity garbage reporting automation w/ ollama
-
FastStream v0.4.0: Introducing Confluent Kafka Integration with Async Support
-
A note from our sponsor - SaaSHub
www.saashub.com | 15 Jan 2025
Index
What are some of the best open-source Kafka projects in Python? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | pathway | 12,374 |
| 2 | Faust | 6,754 |
| 3 | kafka-python | 5,663 |
| 4 | faststream | 3,329 |
| 5 | faust | 1,692 |
| 6 | quix-streams | 1,259 |
| 7 | aiokafka | 1,187 |
| 8 | DataEngineeringProject | 1,167 |
| 9 | nagios-plugins | 1,136 |
| 10 | cp-all-in-one | 983 |
| 11 | kaskade | 864 |
| 12 | KQ | 571 |
| 13 | streamify | 561 |
| 14 | Propan | 484 |
| 15 | tributary | 444 |
| 16 | clickhouse-sink-connector | 238 |
| 17 | kafka-ml | 177 |
| 18 | strimzi-kafka-cli | 83 |
| 19 | python-fake-data-producer-for-apache-kafka | 82 |
| 20 | inferencedb | 78 |
| 21 | spark_app_twitter | 77 |
| 22 | kafka-crypto-questdb | 67 |
| 23 | journalpump | 60 |
Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com