dataloader VS Aeron

When I began working with GraphQL, I had concerns about the N+1 query problem. In my research, I came across the DataLoader pattern and its implementation on GitHub. While I explored various examples of its usage, I still struggled to grasp how it operates internally. Join me in delving a bit deeper into GraphQL DataLoader! :)

DataLoader: it is the standard solution to N+1 problem.

Additionally, you can use DataLoader or similar tools to optimize data fetching and avoid over-fetching or under-fetching data. Ultimately, performance and load tests ensure that your GraphQL API delivers optimal performance, meets response time expectations, and provides a smooth experience for users, even under heavy loads.

The most common practice is to turn N+1 into 1+1 using dataloaders (https://github.com/graphql/dataloader for JS, there are equivalents for most implementations). The N resolvers invoke a single batched loader which receives a list of keys and returns a list of values.

Let's take Payload, for example. Surprise, surprise. We have a relationship field, and it can store IDs to other related documents which are seamlessly merged in when you retrieve documents from the DB. We leverage the dataloader pattern to batch together all "populations" required for a given query, returning them all super fast and with as few separate queries to the DB as possible. We actually even outperform SQL-based frameworks quite a bit. In a purely relational test, we were 3x faster than Directus and 7x faster than Strapi while both were running Postgres, and we were on MongoDB.

Also, have a look at Facebook's Dataloader[0].

[0] https://github.com/graphql/dataloader

In the next article, I'm going to implement a GraphQL server with dataloader using the tools we introduced.

The one example of using Scoped dependency that comes to my mind, it's HTTP request level caching for libs like dataloader.

you would fetch these 1000 rows via dataloader that batches all requests for this relation to a single query... solving the n+1 issue

Semi-related is the Aeron project: https://github.com/real-logic/aeron

- ez itt most egy izgalmasabb product (trading/matching engine, low latency code, aeron alapokon)

Look into Aeron for examples of high performance UDP message sending. We use it for high performance audio messaging, and I previously used it in high frequency trading https://github.com/real-logic/aeron. It is written in Java/C, but the general concepts of back pressure and reliable delivery over UDP are well documented.

They mention their training offer on the Aeron GitHub page here: https://github.com/real-logic/aeron

Yup the Disruptor paper actually shocked the industry a bit, b/c it was so out of place. BTW, Martin Thompson went on improving the Disruptor, and the result is the Aeron Protocol: https://github.com/real-logic/aeron

I was about to suggest Chronicle, but it looks like they have gone closed-source. The older version is still interesting to look through though. Aeron / Disruptor / SBE are good projects for inspiration as well.

We are using this Java Aeron (https://github.com/real-logic/aeron) to build our production distributed messaging cluster. As a Rust lover, Is there any similar lib or framework in our ecosystem to test benchmark with it?

Java: In the past years C++ in finance has been rapidly supplanted by Java thanks to breakthrough technologies in the past decade like LMAX Disruptor, Chronicle Queue, Azul JVM, and Aeron (not the ergonomic chair, but this one, the transport protocol that breaks kafka performance records out of the park - not really a full kafka replacement, as Kafka enforces subscriber GD and aeron is more of an OSI layer 4 better than TCP; google "Best-effort delivery vs reliable delivery"). There's plenty more but thanks to these technologies, they allowed a Java based stack to perform the latency and throughput requirements needed for high frequency trading/HFT. From top trading firms like Two Sigma to the New York Stock Exchange, they're in Java. For banks, large modern western banks worth their salt and have modernized their systems are dominated by Java, especially thanks to Azul. To list a few banks, ING, Wells Fargo, Credit Suisse, and Barclays are all in Azul. Even at work Java still dominates.