Apache Kafka
Aeron
Our great sponsors
Apache Kafka | Aeron | |
---|---|---|
26 | 20 | |
27,275 | 7,046 | |
1.3% | 1.0% | |
9.9 | 9.8 | |
6 days ago | 7 days ago | |
Java | Java | |
Apache License 2.0 | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
Apache Kafka
-
On Implementation of Distributed Protocols
Apache Kafka — a distributed event streaming platform implementing a variant of the Raft consensus protocol (written in Java, integrated with Scala);
- Implementing tagged fields for Kafka Protocol
-
Help me identify this design pattern
Spring does this during autoconfiguration. For example this and this. When the user adds a configuration then it gets to overwrite the default from the template. I am looking for something similar, perhaps simpler approach.
- Kafka Broker Config properties
- Scala DevInTraining looking to contribute to projects
- *bip*
-
What is Kafka ?
Source and documentation on GitHub
-
A simple file source/sink connector?
Code is still in trunk though. https://github.com/apache/kafka/tree/trunk/connect/file/src/main/java/org/apache/kafka/connect/file
-
Can someone please eli5 how the hierarchical timing wheel algorithm works?
I briefly described the algorithm in this article and there is a wonderful article from Kafka that goes into more depth in their general purpose implementation. My implementation is specialized and over optimized in comparison, e.g. by using bit manipulation to avoid more expensive division/modulus instructions. Tokio rewrote their timerwheel after I showed them mine, borrowing some ideas but also staying more general. Hope that helps!
-
How-to-Guide: Contributing to Open Source
Apache Kafka
Aeron
-
LMAX Disruptor – High Performance Inter-Thread Messaging Library
Semi-related is the Aeron project: https://github.com/real-logic/aeron
-
Nálatok mi a helyzet?
- ez itt most egy izgalmasabb product (trading/matching engine, low latency code, aeron alapokon)
-
How do you do UDP Flow control?
Look into Aeron for examples of high performance UDP message sending. We use it for high performance audio messaging, and I previously used it in high frequency trading https://github.com/real-logic/aeron. It is written in Java/C, but the general concepts of back pressure and reliable delivery over UDP are well documented.
- Aeron: Efficient reliable UDP unicast, UDP multicast, and IPC message transport
-
Experience taking the training offer from real-logic Aeron framework creators?
They mention their training offer on the Aeron GitHub page here: https://github.com/real-logic/aeron
-
Low Latency C++ programs for High Frequency Trading (HFT)
Yup the Disruptor paper actually shocked the industry a bit, b/c it was so out of place. BTW, Martin Thompson went on improving the Disruptor, and the result is the Aeron Protocol: https://github.com/real-logic/aeron
- What network messaging library do you recommend?
-
Possibly stupid question, is java the right language for low latency and high throughput web servers?
I was about to suggest Chronicle, but it looks like they have gone closed-source. The older version is still interesting to look through though. Aeron / Disruptor / SBE are good projects for inspiration as well.
-
Looking similar framework with Aeron ( Java) to do benchmark test
We are using this Java Aeron (https://github.com/real-logic/aeron) to build our production distributed messaging cluster. As a Rust lover, Is there any similar lib or framework in our ecosystem to test benchmark with it?
-
if you had to restart at 0 knowledge what would you do?
Java: In the past years C++ in finance has been rapidly supplanted by Java thanks to breakthrough technologies in the past decade like LMAX Disruptor, Chronicle Queue, Azul JVM, and Aeron (not the ergonomic chair, but this one, the transport protocol that breaks kafka performance records out of the park - not really a full kafka replacement, as Kafka enforces subscriber GD and aeron is more of an OSI layer 4 better than TCP; google "Best-effort delivery vs reliable delivery"). There's plenty more but thanks to these technologies, they allowed a Java based stack to perform the latency and throughput requirements needed for high frequency trading/HFT. From top trading firms like Two Sigma to the New York Stock Exchange, they're in Java. For banks, large modern western banks worth their salt and have modernized their systems are dominated by Java, especially thanks to Azul. To list a few banks, ING, Wells Fargo, Credit Suisse, and Barclays are all in Azul. Even at work Java still dominates.
What are some alternatives?
celery - Distributed Task Queue (development branch)
Embedded RabbitMQ - A JVM library to use RabbitMQ as an embedded service
Apache ActiveMQ Artemis - Mirror of Apache ActiveMQ Artemis
Apache Pulsar - Apache Pulsar - distributed pub-sub messaging system
redpanda - Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
Apache ActiveMQ - Mirror of Apache ActiveMQ
jetstream - JetStream Utilities
JeroMQ - Pure Java ZeroMQ
NATS - High-Performance server for NATS.io, the cloud and edge native messaging system.
Apache Camel - Apache Camel is an open source integration framework that empowers you to quickly and easily integrate various systems consuming or producing data.
Apache Qpid - Mirror of Apache Qpid
Disruptor-cpp - Port of LMAX Disruptor to C++