Top 23 Java Kafka Projects
-
Project mention: How-to-Guide: Contributing to Open Source | reddit.com/r/dataengineering | 2022-06-11
Apache Kafka
-
SpringBoot-Labs
一个涵盖六个专栏:Spring Boot 2.X、Spring Cloud、Spring Cloud Alibaba、Dubbo、分布式消息队列、分布式事务的仓库。希望胖友小手一抖,右上角来个 Star,感恩 1024
-
Scout APM
Less time debugging, more time building. Scout APM allows you to find and fix performance issues with no hassle. Now with error monitoring and external services monitoring, Scout is a developer's best friend when it comes to application development.
-
Thingsboard
Open-source IoT Platform - Device management, data collection, processing and visualization.
Project mention: Zelensky: “Ukraine has a lot to offer the world, products and services created by Ukrainians. Ukrainian companies – join. World – use. Tell your foreign friends about it. Spend with Ukraine to stand for Ukraine.“ | reddit.com/r/ukraine | 2022-06-23The company I work for uses https://thingsboard.io and the team from Ukraine that built it are awesome and the company is still responding even though they are going through this hell of a war.
-
debezium
Change data capture for a variety of databases. Please log issues at https://issues.redhat.com/browse/DBZ.
While working on this problem, I discovered Change-Data-Capture. The idea behind CDC is to send updates to a single store and stream the diffs of the new state to the other one. To implement it by oneself is not trivial. I'd recommend using an existing product: I've used Debezium successfully in the past for my demos.
-
That's not ELK, you need a logging observability tool. For example Grafana Loki or maybe Graylog, which uses elastic, but is not ELK..
-
-
Project mention: What kind of monitoring tools are people using for their Kafka Deployment? | reddit.com/r/apachekafka | 2022-01-27
Kafdrop (https://github.com/obsidiandynamics/kafdrop) is very popular and pretty solid. I'd start there. Kafka itself also comes with some cli tools and you can also extract metrics out of it via JMX/jolokia.
-
SonarLint
Clean code begins in your IDE with SonarLint. Up your coding game and discover issues early. SonarLint is a free plugin that helps you find & fix bugs and security issues from the moment you start writing code. Install from your favorite IDE marketplace today.
-
Project mention: Kafka cluster and its maintenance, what is it like? | reddit.com/r/apachekafka | 2022-06-07
-
akhq
Kafka GUI for Apache Kafka to manage topics, topics data, consumers group, schema registry, connect and more...
Project mention: Show HN: After AKHQ (KafkaHQ), I Made Kestra Open-Source Airflow Alternative | news.ycombinator.com | 2022-03-24The project start as a side project (yet another side project I do the night and weekend) but was quickly promoted and used in a French Big Retail Company.
This one trust on the project and decide to go production with Kestra. So they decide to inject some resource in order to develop some features that need and that is missing.
But basically, not so much people for now. We are trying to start a community around the product and started to communicate around the product since few weeks only, I hope community will follow us! And I hope to succeed like on my other open source project: https://github.com/tchiotludo/akhq
-
cruise-control
Cruise-control is the first of its kind to fully automate the dynamic workload rebalance and self-healing of a Kafka cluster. It provides great value to Kafka users by simplifying the operation of Kafka clusters.
Project mention: I'm looking to add more instances in a kafka cluster, how does kafka know that there are more instances in a cluster? How do we tell kafka that there are more instances provisioned? | reddit.com/r/apachekafka | 2022-06-22I used this at my last job and it made kafka mgmt a lot easier - https://github.com/linkedin/cruise-control
-
When you say the Rest API, are you referring to the Confluent REST Proxy?
-
Since you are trying to archive this data, a simpler solution is to use something like secor to archive the data in kafka to S3. It’s much cheaper than dynamodb too.
-
Project mention: Testing a Kafka consumer with Avro schema messages in your Spring Boot application with Testcontainers | dev.to | 2022-02-22
So that means we can configure the Kafka producer and consumer with an imaginary schema registry url, that only needs to start with “mock://” and you automatically get to work with the MockSchemaRegistryClient. This way you don't need to explicitly initiate the MockSchemaRegistryClient and configure everything accordingly. That also eradicates the need for the Confluent Schema Registry Container. Running the Kafka Testcontainer with the embedded Zookeeper, we no longer need an extra Zookeeper container and we are down to one Testcontainer for the messaging. This way I ended up with only two Testcontainers: Kafka and the database.
-
-
Hazelcast Jet
-
Nakadi
A distributed event bus that implements a RESTful API abstraction on top of Kafka-like queues
-
Event sourcing is an effective architectural pattern to record changes to the application state. Event sequence is important - we need changes as they were originally applied. incoming events are first persisted into Kafka and then processed by services independently. Kafka, hence, becomes our source of truth (SOT), a data source that gives a complete picture of the data object as a whole. This, however, meant dramatic changes in our core application. Our source of truth is still the core database, but we generate events in Kafka when data gets persisted. To ensure transactional behavior, we employed Transactional Outbox pattern. Essentially, we a) created a new events table in the database, b) wrote event data in the same transaction when we update our SOT table. Kafka Connect is subsequently used to read this table and insert records in relevant Kafka topics. This ensures that we never have an inconsistent situation where data was inserted in the database but the event is not added to Kafka topic or vice versa. We evaluated a few connectors for sourcing data from Mysql (JdbcSourceConnector, and Debezium). Our scenario was supported out of the box in JdbcSourceConnector, making it possible to have one event table in Mysql where different rows could be routed to a relevant topic based on the topic field.
-
Project mention: Got fed up with AMQP, so I wrote a HTTP message broker on top of RabbitMQ | reddit.com/r/programming | 2022-03-27
Let's draw some parallels in the Kafka ecosystem, in which there are Confluent REST Proxy and Hermes by Allegro. You could say that since Kafka uses a binary protocol over TCP, both of these projects are "moot and useless". Despite that, Allegro aka the Poland's Amazon still maintains that project and uses it in production as a means of HTTP-based communication between their 700+ microservices arguing that it makes their development process easier, since it allows them to abstract away the intricacies of Kafka and use plain HTTP instead.
-
-
Project mention: Building a Scalable Live Stream Chat Service with Spring WebFlux, Redis PubSub, RSocket and Auth0 | dev.to | 2021-08-20
We will use the Playtika testcontainers-springboot library to easily spin up a Redis docker container while running the integration tests locally. This can then be used as our PubSub message broker as we test the functionality of the connection server.
-
-
zerocode
A community-developed, free, open source, microservices API automation and load testing framework built using JUnit core runners for Http REST, SOAP, Security, Database, Kafka and much more. Zerocode Open Source enables you to create, change, orchestrate and maintain your automated test cases declaratively with absolute ease.
Zerocode
-
eventeum
A resilient Ethereum event listener that bridges your smart contract events and backend microservices
Project mention: Services for listening to events on smart contracts | reddit.com/r/ethereum | 2022-01-17I am creating a system to accept erc20 payments on my website. A few years back I came across Eventeum. I used in in a prototype (for a different purpose) and it worked great.
Java Kafka related posts
- I'm looking to add more instances in a kafka cluster, how does kafka know that there are more instances in a cluster? How do we tell kafka that there are more instances provisioned?
- Bring your own REST APIs for Apache Kafka
- An open source API gateway for Kafka — thoughts?
- Kafkaer: Manage Kafka topics and configurations as code
- Kafka: Manage kafka topics and configs as code
- Apache Camel Integration with Spring-boot and Kafka
- PranaDB
Index
What are some of the best open-source Kafka projects in Java? This list will help you:
Project | Stars | |
---|---|---|
1 | Apache Kafka | 22,336 |
2 | SpringBoot-Labs | 14,510 |
3 | Thingsboard | 11,930 |
4 | debezium | 7,009 |
5 | graylog | 6,162 |
6 | ksql | 5,056 |
7 | kafdrop | 3,800 |
8 | strimzi-kafka-operator | 3,272 |
9 | akhq | 2,536 |
10 | cruise-control | 2,204 |
11 | kafka-rest | 1,954 |
12 | secor | 1,769 |
13 | schema-registry | 1,765 |
14 | PretendYoureXyzzy | 1,140 |
15 | Hazelcast Jet | 998 |
16 | Nakadi | 847 |
17 | kafka-connect-jdbc | 833 |
18 | Hermes | 720 |
19 | kafka-streams-course | 696 |
20 | testcontainers-spring-boot | 670 |
21 | kafka-connect-elasticsearch | 661 |
22 | zerocode | 613 |
23 | eventeum | 403 |
Are you hiring? Post a new remote job listing for free.