kafkacat VS Docker Compose

To communicate with Kafka, you can use Kafkacat, a command-line tool that allows to produce and consume Kafka messages using a very simple syntax. It also allows you to view the topics' metadata.

Feels like you could get pretty far with kafkacat and a SQLite database.

This is the second release version of Kafcat. Kafcat is a Rust fully async rewrite of kafkacat.

Consume from the Kafka topic my.topic with kafkacat:

From the code above, you can see that to process the headers, simply use the ConsumerRecord.headers() method to return the headers. In our example above, we’re printing the headers out to the console for demonstration purposes. Once you have access to the headers, you can process them as needed. For reading headers from the command line, KIP-431 adds support for optionally printing headers from the ConsoleConsumer, which will be available in the Apache Kafka 2.7.0 release.You can also use kafkacat to view headers from the command line. Here’s an example command:

Note: In the example above, we have used kafkacat to consume the topics. The option -o-1 is used to only consume the latest message

“Compose simplifies the control of your entire application stack, making it easy to manage services, networks, and volumes in a single, comprehensible YAML configuration file. Then, with a single command, you create and start all the services from your configuration file.” - Docker documentation

curl -fsSL https://get.docker.com -o get-docker.sh sudo sh get-docker.sh # Install docker compose sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose # Apply executable permissions to the binary sudo chmod +x /usr/local/bin/docker-compose # Run Project docker-compose up -d

Docker,

This foundation now opens the door to even more powerful concepts. You can explore more advanced concepts such as container networking, streamlining the management of complex applications with Docker Compose, and how to make your application data persistent using volumes.

In many projects that are containerized, especially in cases where development is also done locally with docker-compose, teams often have two Dockerfiles, 1 for Development, the other for Production. If you happen to have multiple environments like pre-prod, staging and so on, some teams could have different Dockerfiles for these environments.

sudo curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose

You can also use Docker Compose to deploy the cluster, without having to write all the previous commands.

[ec2-user]$ sudo yum update -y [ec2-user]$ sudo yum install -y docker [ec2-user]$ sudo service docker start [ec2-user]$ sudo curl -L https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose [ec2-user]$ sudo chmod +x /usr/local/bin/docker-compose [ec2-user]$ docker --version Docker version 20.10.23, build 7155243 [ec2-user]$ docker-compose --version Docker Compose version v2.18.1

Docker Compose Documentation

groupadd -g 997 docker gpasswd -a jenkins docker curl -L https://github.com/docker/compose/releases/tag/1.29.2/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose chmod +x /usr/local/bin/docker-compose #confirm the docker-compose is installed docker-compose version