kafka-python VS quix-streams

Kafka-python is a producer-consumer library for one message at a time applications. Quix Streams is a Python stream processing library for ML and AI applications. Use them together in your event streaming architecture.

kafka-python - https://github.com/dpkp/kafka-python

If you're using kafka-python take a look at it's sourcecode.

contribute to opensource. companies care about distributed systems at the moment, so if you can contribute to something like kafka https://kafka.apache.org/project or a popular wrapper https://github.com/dpkp/kafka-python, or a distributed systems platform https://github.com/redpanda-data/redpanda, that would be a strong signal that you can provide value and work with others as you deliver code.

Eg this https://kafka-python.readthedocs.io/

Related issue thread - https://github.com/dpkp/kafka-python/issues/2290

I've been working on a pure-Racket Kafka client[1] off and on (for fun) since February and it's a good amount of work. When the official protocol docs and Wireshark fail me, I usually look at librdkafka and kafka-python[2] to figure out how things are supposed to fit together. Kudos to the authors of both libraries for writing code that's easy to follow!

[1]: https://defn.io/2022/03/12/ann-racket-kafka/

[2]: https://github.com/dpkp/kafka-python

Airflow for Streaming

Debezium is a change data capture framework for streaming. Connect it to your databases to detect changes and produce those change events to Kafka. Quix Streams is a Python stream processing library for ML and AI applications. It builds on the Confluent Python library to add a state store with RocksDB and adds a Streaming DataFrames API with declarative operations like Windows. It is designed for analytics and data engineering workloads. Use it together with Debezium to process your CDC data in real-time.