confluent-kafka-python VS kafka-python

Responding to popular demand, the latest 0.4.0rc0 version introduces support for Kafka stream processing using Confluent Kafka's Python library - https://github.com/confluentinc/confluent-kafka-python.

- confluent kafka library (https://github.com/confluentinc/confluent-kafka-python) uses librdkafka, and is faster that kafka-python - https://www.bytewax.io/(https://github.com/bytewax/bytewax) is another option. Handles the dirty work of exchanging data across processes for parallel stateful processing

Here are examples https://github.com/confluentinc/confluent-kafka-python

I just wanted to mention that pykafka is currently unmaintained and archived on GitHub:

https://github.com/Parsely/pykafka

pykafka was originally developed and maintained by my team at Parse.ly, but we no longer maintain it. We instead encourage folks to use confluent-kafka-python, which is what we have ourselves switched to in our production systems:

https://github.com/confluentinc/confluent-kafka-python

(pykafka was developed at a time before Confluent invested in their own Python binding. Some of the history of the project is described in this 2016 blog post[1] and our original 2015 announcement[2].)

[1]: https://blog.parse.ly/pykafka-now/

[2]: https://blog.parse.ly/announcing-pykafka-python-support-for-...

confluent-kafka

I also found that there is a library provided by confluent https://github.com/confluentinc/confluent-kafka-python which maybe helpful here. Is that right? I see a section "SSL certificates" but couldnt get much insight out of this.

ProtobugDeserializer does not allow anything but the protoBuf message type (see here):

I'd start off with some simple producer/consumer code to get an understanding of interacting with Kafka's APIs from Python - there's a Python client with some good examples of the producer and consumer as well as more complex examples.

Kafka-python is a producer-consumer library for one message at a time applications. Quix Streams is a Python stream processing library for ML and AI applications. Use them together in your event streaming architecture.

kafka-python - https://github.com/dpkp/kafka-python

If you're using kafka-python take a look at it's sourcecode.

contribute to opensource. companies care about distributed systems at the moment, so if you can contribute to something like kafka https://kafka.apache.org/project or a popular wrapper https://github.com/dpkp/kafka-python, or a distributed systems platform https://github.com/redpanda-data/redpanda, that would be a strong signal that you can provide value and work with others as you deliver code.

Eg this https://kafka-python.readthedocs.io/

Related issue thread - https://github.com/dpkp/kafka-python/issues/2290

I've been working on a pure-Racket Kafka client[1] off and on (for fun) since February and it's a good amount of work. When the official protocol docs and Wireshark fail me, I usually look at librdkafka and kafka-python[2] to figure out how things are supposed to fit together. Kudos to the authors of both libraries for writing code that's easy to follow!

[1]: https://defn.io/2022/03/12/ann-racket-kafka/

[2]: https://github.com/dpkp/kafka-python