Top 23 Java Kafka Projects
Mirror of Apache KafkaProject mention: How-to-Guide: Contributing to Open Source | reddit.com/r/dataengineering | 2022-06-11
一个涵盖六个专栏：Spring Boot 2.X、Spring Cloud、Spring Cloud Alibaba、Dubbo、分布式消息队列、分布式事务的仓库。希望胖友小手一抖，右上角来个 Star，感恩 1024
Less time debugging, more time building. Scout APM allows you to find and fix performance issues with no hassle. Now with error monitoring and external services monitoring, Scout is a developer's best friend when it comes to application development.
Open-source IoT Platform - Device management, data collection, processing and visualization.Project mention: Zelensky: “Ukraine has a lot to offer the world, products and services created by Ukrainians. Ukrainian companies – join. World – use. Tell your foreign friends about it. Spend with Ukraine to stand for Ukraine.“ | reddit.com/r/ukraine | 2022-06-23
The company I work for uses https://thingsboard.io and the team from Ukraine that built it are awesome and the company is still responding even though they are going through this hell of a war.
Change data capture for a variety of databases. Please log issues at https://issues.redhat.com/browse/DBZ.Project mention: Hard things in Computer Science | dev.to | 2022-06-27
While working on this problem, I discovered Change-Data-Capture. The idea behind CDC is to send updates to a single store and stream the diffs of the new state to the other one. To implement it by oneself is not trivial. I'd recommend using an existing product: I've used Debezium successfully in the past for my demos.
Free and open source log managementProject mention: ELK Stack or an alternative in 2022? | reddit.com/r/sysadmin | 2022-06-09
That's not ELK, you need a logging observability tool. For example Grafana Loki or maybe Graylog, which uses elastic, but is not ELK..
The database purpose-built for stream processing applications.Project mention: PranaDB | news.ycombinator.com | 2022-04-28
Kafka Web UIProject mention: What kind of monitoring tools are people using for their Kafka Deployment? | reddit.com/r/apachekafka | 2022-01-27
Kafdrop (https://github.com/obsidiandynamics/kafdrop) is very popular and pretty solid. I'd start there. Kafka itself also comes with some cli tools and you can also extract metrics out of it via JMX/jolokia.
Clean code begins in your IDE with SonarLint. Up your coding game and discover issues early. SonarLint is a free plugin that helps you find & fix bugs and security issues from the moment you start writing code. Install from your favorite IDE marketplace today.
Apache Kafka® running on KubernetesProject mention: Kafka cluster and its maintenance, what is it like? | reddit.com/r/apachekafka | 2022-06-07
Kafka GUI for Apache Kafka to manage topics, topics data, consumers group, schema registry, connect and more...Project mention: Show HN: After AKHQ (KafkaHQ), I Made Kestra Open-Source Airflow Alternative | news.ycombinator.com | 2022-03-24
The project start as a side project (yet another side project I do the night and weekend) but was quickly promoted and used in a French Big Retail Company.
This one trust on the project and decide to go production with Kestra. So they decide to inject some resource in order to develop some features that need and that is missing.
But basically, not so much people for now. We are trying to start a community around the product and started to communicate around the product since few weeks only, I hope community will follow us! And I hope to succeed like on my other open source project: https://github.com/tchiotludo/akhq
Cruise-control is the first of its kind to fully automate the dynamic workload rebalance and self-healing of a Kafka cluster. It provides great value to Kafka users by simplifying the operation of Kafka clusters.Project mention: I'm looking to add more instances in a kafka cluster, how does kafka know that there are more instances in a cluster? How do we tell kafka that there are more instances provisioned? | reddit.com/r/apachekafka | 2022-06-22
I used this at my last job and it made kafka mgmt a lot easier - https://github.com/linkedin/cruise-control
Confluent REST Proxy for KafkaProject mention: Kafka Rest API vs client library | reddit.com/r/apachekafka | 2022-03-21
When you say the Rest API, are you referring to the Confluent REST Proxy?
Secor is a service implementing Kafka log persistenceProject mention: Storing kakfa messages to dynamoDb | reddit.com/r/apachekafka | 2022-02-11
Since you are trying to archive this data, a simpler solution is to use something like secor to archive the data in kafka to S3. It’s much cheaper than dynamodb too.
Confluent Schema Registry for KafkaProject mention: Testing a Kafka consumer with Avro schema messages in your Spring Boot application with Testcontainers | dev.to | 2022-02-22
So that means we can configure the Kafka producer and consumer with an imaginary schema registry url, that only needs to start with “mock://” and you automatically get to work with the MockSchemaRegistryClient. This way you don't need to explicitly initiate the MockSchemaRegistryClient and configure everything accordingly. That also eradicates the need for the Confluent Schema Registry Container. Running the Kafka Testcontainer with the embedded Zookeeper, we no longer need an extra Zookeeper container and we are down to one Testcontainer for the messaging. This way I ended up with only two Testcontainers: Kafka and the database.
A web clone of the card game Cards Against Humanity.
Distributed Stream and Batch ProcessingProject mention: Updating data files, commits vs. pull requests | dev.to | 2021-08-15
A distributed event bus that implements a RESTful API abstraction on top of Kafka-like queues
Kafka Connect connector for JDBC-compatible databasesProject mention: How Sendoso is using Kafka for Event-Driven Architecture | dev.to | 2021-08-31
Event sourcing is an effective architectural pattern to record changes to the application state. Event sequence is important - we need changes as they were originally applied. incoming events are first persisted into Kafka and then processed by services independently. Kafka, hence, becomes our source of truth (SOT), a data source that gives a complete picture of the data object as a whole. This, however, meant dramatic changes in our core application. Our source of truth is still the core database, but we generate events in Kafka when data gets persisted. To ensure transactional behavior, we employed Transactional Outbox pattern. Essentially, we a) created a new events table in the database, b) wrote event data in the same transaction when we update our SOT table. Kafka Connect is subsequently used to read this table and insert records in relevant Kafka topics. This ensures that we never have an inconsistent situation where data was inserted in the database but the event is not added to Kafka topic or vice versa. We evaluated a few connectors for sourcing data from Mysql (JdbcSourceConnector, and Debezium). Our scenario was supported out of the box in JdbcSourceConnector, making it possible to have one event table in Mysql where different rows could be routed to a relevant topic based on the topic field.
Fast and reliable message broker built on top of Kafka. (by allegro)Project mention: Got fed up with AMQP, so I wrote a HTTP message broker on top of RabbitMQ | reddit.com/r/programming | 2022-03-27
Let's draw some parallels in the Kafka ecosystem, in which there are Confluent REST Proxy and Hermes by Allegro. You could say that since Kafka uses a binary protocol over TCP, both of these projects are "moot and useless". Despite that, Allegro aka the Poland's Amazon still maintains that project and uses it in production as a means of HTTP-based communication between their 700+ microservices arguing that it makes their development process easier, since it allows them to abstract away the intricacies of Kafka and use plain HTTP instead.
Learn Kafka Streams with several examples!
Container auto-configurations for Spring Boot based integration testsProject mention: Building a Scalable Live Stream Chat Service with Spring WebFlux, Redis PubSub, RSocket and Auth0 | dev.to | 2021-08-20
We will use the Playtika testcontainers-springboot library to easily spin up a Redis docker container while running the integration tests locally. This can then be used as our PubSub message broker as we test the functionality of the connection server.
Kafka Connect Elasticsearch connector
A community-developed, free, open source, microservices API automation and load testing framework built using JUnit core runners for Http REST, SOAP, Security, Database, Kafka and much more. Zerocode Open Source enables you to create, change, orchestrate and maintain your automated test cases declaratively with absolute ease.Project mention: 20+ Trending and Popular Java Open Source Project | dev.to | 2022-05-10
A resilient Ethereum event listener that bridges your smart contract events and backend microservicesProject mention: Services for listening to events on smart contracts | reddit.com/r/ethereum | 2022-01-17
I am creating a system to accept erc20 payments on my website. A few years back I came across Eventeum. I used in in a prototype (for a different purpose) and it worked great.
Java Kafka related posts
I'm looking to add more instances in a kafka cluster, how does kafka know that there are more instances in a cluster? How do we tell kafka that there are more instances provisioned?
1 project | reddit.com/r/apachekafka | 22 Jun 2022
Bring your own REST APIs for Apache Kafka
1 project | reddit.com/r/apachekafka | 22 Jun 2022
An open source API gateway for Kafka — thoughts?
1 project | reddit.com/r/apachekafka | 8 Jun 2022
Kafkaer: Manage Kafka topics and configurations as code
2 projects | reddit.com/r/apachekafka | 7 Jun 2022
Kafka: Manage kafka topics and configs as code
1 project | reddit.com/r/java | 7 Jun 2022
Apache Camel Integration with Spring-boot and Kafka
1 project | reddit.com/r/SpringBoot | 21 May 2022
2 projects | news.ycombinator.com | 28 Apr 2022
What are some of the best open-source Kafka projects in Java? This list will help you:
Are you hiring? Post a new remote job listing for free.