confluent-kafka-python VS faststream

Responding to popular demand, the latest 0.4.0rc0 version introduces support for Kafka stream processing using Confluent Kafka's Python library - https://github.com/confluentinc/confluent-kafka-python.

- confluent kafka library (https://github.com/confluentinc/confluent-kafka-python) uses librdkafka, and is faster that kafka-python - https://www.bytewax.io/(https://github.com/bytewax/bytewax) is another option. Handles the dirty work of exchanging data across processes for parallel stateful processing

Here are examples https://github.com/confluentinc/confluent-kafka-python

I just wanted to mention that pykafka is currently unmaintained and archived on GitHub:

https://github.com/Parsely/pykafka

pykafka was originally developed and maintained by my team at Parse.ly, but we no longer maintain it. We instead encourage folks to use confluent-kafka-python, which is what we have ourselves switched to in our production systems:

https://github.com/confluentinc/confluent-kafka-python

(pykafka was developed at a time before Confluent invested in their own Python binding. Some of the history of the project is described in this 2016 blog post[1] and our original 2015 announcement[2].)

[1]: https://blog.parse.ly/pykafka-now/

[2]: https://blog.parse.ly/announcing-pykafka-python-support-for-...

confluent-kafka

I also found that there is a library provided by confluent https://github.com/confluentinc/confluent-kafka-python which maybe helpful here. Is that right? I see a section "SSL certificates" but couldnt get much insight out of this.

ProtobugDeserializer does not allow anything but the protoBuf message type (see here):

I'd start off with some simple producer/consumer code to get an understanding of interacting with Kafka's APIs from Python - there's a Python client with some good examples of the producer and consumer as well as more complex examples.

FastStream - https://github.com/airtai/faststream, a stream processing framework, already supports Kafka stream processing using the aiokafka library, as well as other brokers such as Redis, RabbitMQ, and NATS.

FastStream is a powerful and easy-to-use FOSS framework for building asynchronous services interacting with event streams such as Apache Kafka, RabbitMQ and NATS. It simplifies the process of writing producers and consumers for message queues, handling all the parsing, networking and documentation generation automatically.

Ready to join the FastStream revolution? Head over to our GitHub repository and show your support by starring it. By doing so, you'll stay in the loop with the latest developments, updates, and enhancements as we continue to refine and expand FastStream.

After two months of hard work, we presented the newly released FastStream framework at Infobip Shift conference and got featured at ShiftMag. The framework now supports both Apache Kafka and RabbitMQ, but also NATS protocol with the plan to add more protocols in the near future. The overall code is much cleaner and the implementation is streamlined with abstractions covering the common functionality across the protocols. We deprecated both FastKafka and Propan, but promised to fix bugs as long as needed. However, it seems like the community already decided to switch over to gain new functionalities.

faststream-gen(https://github.com/airtai/faststream-gen/) uses GPT models to automatically generate microservices using the FastStream(https://github.com/airtai/faststream) framework for Apache Kafka, RabbitMQ and NATS. Simply describe your microservice in plain English, and it will generate a production-level FastStream application ready to deploy in a few minutes and under $1 cost, together with unit and integration tests, documentation and Docker images.

faststream-gen is a Python library that uses generative AI to automatically generate FastStream applications. Simply describe your microservice in plain English, and faststream-gen will generate a production-level FastStream application ready to deploy in a few minutes and under $1 cost.

FastStream simplifies the process of writing producers and consumers for message queues, handling all the parsing, networking and documentation generation automatically. It is a new package based on the ideas and experiences gained from FastKafka and Propan. By joining our forces, we picked up the best from both packages and created a unified way to write services capable of processing streamed data regardless of the underlying protocol. We'll continue to maintain both packages, but new development will be in this project.

FastStream (https://github.com/airtai/faststream) is a new Python framework, born from Propan and FastKafka teams' collaboration (both are deprecated now). It extremely simplifies event-driven system development, handling all the parsing, networking, and documentation generation automatically. Now FastStream supports RabbitMQ and Kafka, but supported brokers are constantly growing (wait for NATS and Redis a bit). FastStream itself is a really great tool to build event-driven services. Also, it has a native FastAPI integration. Just create a StreamRouter (very close to APIRouter) and register event handlers the same with the regular HTTP-endpoints way: