demo-scene VS confluent-kafka-python

Hi HN!

I recently created a website that displays your votes on classically debated questions like 'tabs or spaces?'?

One of my favorite things about it is that if you leave the result progress bars up, and someone else votes, you can see the results change in real time.

I used the confluent-kafka-javascript(https://github.com/confluentinc/confluent-kafka-javascript) client which is currently in early access.

Votes are sent to a Kafka topic deployed in Confluent Cloud, and the code for the frontend is on Amplify while the code creating the route and Kafka clients is on EC2.

I put a version of the code here(https://github.com/confluentinc/demo-scene/tree/master/confl...) so you can follow along and deploy a similar project for yourself. I'm hoping it makes a fun way for you to learn this Kafka JavaScript client.

In the future, I'd like to implement features like allowing you to change your vote, or a board summarizing votes like "30% of developers who voted for X voted for Y".

Enjoy! Looking forward to your feedback.

Here’s a great place to dig into things: https://developer.confluent.io

Once you’ve decided on a language to focus on, it’s time to start filling those knowledge gaps. Don’t be discouraged by this step. We all have knowledge gaps, and filling them can be very rewarding. First, you’ll want to make sure you have a good understanding of Kafka basics. Fortunately, there are many resources to help you with this. A web search will turn up many great books and other resources. And, of course, Confluent Developer offers interactive courses ranging from introductory (Apache Kafka 101) to advanced (Kafka Internals), full documentation, and other content to help you get started.

Confluent Developer: Learn Apache Kafka through Confluent Developer tutorials, documentation, courses, blog posts, and examples.

Spend some time on https://developer.confluent.io , especially https://developer.confluent.io/get-started/java/ and look at https://github.com/confluentinc/examples . Those will give really practical examples for how to set up your configs and start consuming

FWIW you'll find a bunch of promo codes over at developer.confluent.io

Kafka streams is a Java library built on the producer/consumer APIs designed to create event-driven microservices on top of Kafka. In general, I recommend exploring https://developer.confluent.io to learn more

For example: https://github.com/confluentinc/demo-scene/tree/master/kafka-connect-zero-to-hero

should I be reading a different material for a first Kafka project and working with a different kind of setup? Now, I can't be unbiased on this one ;) One of the things we're doing with Confluent Developer is to try and create a resource for people to learn Kafka from the ground up, whether they ultimately decide to pursue it on Confluent or not. The fundamentals of Kafka that you'll be learning are going to be as applicable whether you're using Apache Kafka self-managed, or Confluent, or AWS' MSK, or whatever else. Personally I'd this stage I'd use whatever setup you find easiest and least friction to your learning journey. As u/louisvell mentioned, /u/stephanemaarek's courses on Udemy are also very popular, if you wanted a "second opinion" on how to approach learning Kafka.

Responding to popular demand, the latest 0.4.0rc0 version introduces support for Kafka stream processing using Confluent Kafka's Python library - https://github.com/confluentinc/confluent-kafka-python.

- confluent kafka library (https://github.com/confluentinc/confluent-kafka-python) uses librdkafka, and is faster that kafka-python - https://www.bytewax.io/(https://github.com/bytewax/bytewax) is another option. Handles the dirty work of exchanging data across processes for parallel stateful processing

Here are examples https://github.com/confluentinc/confluent-kafka-python

I just wanted to mention that pykafka is currently unmaintained and archived on GitHub:

https://github.com/Parsely/pykafka

pykafka was originally developed and maintained by my team at Parse.ly, but we no longer maintain it. We instead encourage folks to use confluent-kafka-python, which is what we have ourselves switched to in our production systems:

https://github.com/confluentinc/confluent-kafka-python

(pykafka was developed at a time before Confluent invested in their own Python binding. Some of the history of the project is described in this 2016 blog post[1] and our original 2015 announcement[2].)

[1]: https://blog.parse.ly/pykafka-now/

[2]: https://blog.parse.ly/announcing-pykafka-python-support-for-...

confluent-kafka

I also found that there is a library provided by confluent https://github.com/confluentinc/confluent-kafka-python which maybe helpful here. Is that right? I see a section "SSL certificates" but couldnt get much insight out of this.

ProtobugDeserializer does not allow anything but the protoBuf message type (see here):

I'd start off with some simple producer/consumer code to get an understanding of interacting with Kafka's APIs from Python - there's a Python client with some good examples of the producer and consumer as well as more complex examples.