dataloader VS graphql-query-complexity

When I began working with GraphQL, I had concerns about the N+1 query problem. In my research, I came across the DataLoader pattern and its implementation on GitHub. While I explored various examples of its usage, I still struggled to grasp how it operates internally. Join me in delving a bit deeper into GraphQL DataLoader! :)

DataLoader: it is the standard solution to N+1 problem.

Additionally, you can use DataLoader or similar tools to optimize data fetching and avoid over-fetching or under-fetching data. Ultimately, performance and load tests ensure that your GraphQL API delivers optimal performance, meets response time expectations, and provides a smooth experience for users, even under heavy loads.

The most common practice is to turn N+1 into 1+1 using dataloaders (https://github.com/graphql/dataloader for JS, there are equivalents for most implementations). The N resolvers invoke a single batched loader which receives a list of keys and returns a list of values.

Let's take Payload, for example. Surprise, surprise. We have a relationship field, and it can store IDs to other related documents which are seamlessly merged in when you retrieve documents from the DB. We leverage the dataloader pattern to batch together all "populations" required for a given query, returning them all super fast and with as few separate queries to the DB as possible. We actually even outperform SQL-based frameworks quite a bit. In a purely relational test, we were 3x faster than Directus and 7x faster than Strapi while both were running Postgres, and we were on MongoDB.

Also, have a look at Facebook's Dataloader[0].

[0] https://github.com/graphql/dataloader

In the next article, I'm going to implement a GraphQL server with dataloader using the tools we introduced.

The one example of using Scoped dependency that comes to my mind, it's HTTP request level caching for libs like dataloader.

you would fetch these 1000 rows via dataloader that batches all requests for this relation to a single query... solving the n+1 issue

https://github.com/slicknode/graphql-query-complexity

In addition you could introduce CI tools to enforce your devs stop writing such complex queries. Also see the @skip and @include directives that can further be used to control what data is queried. In practice, however, this isn't something that comes up too much. In cases where I have seen this happen, it's usually because a developer is trying to reuse fragments without considering what data they are querying, and whether they should be reusing those fragments.

https://graphql.org/learn/queries/#fragments

very cool! I was looking at https://github.com/slicknode/graphql-query-complexity

There are a couple of techniques that can be used to prevent this kind of problem one of them is GraphQL Query Complexity Analysis which is, as the name suggests, very complex to implement correctly. It requires analysis of how the graphql API is used, and what queries and mutations are most often called. If you get this wrong, there is a danger of the server denying perfectly valid queries.

The problem is that those queries are not prevented by commonly available rate limiters. You can send a single request to a GraphQL server that completely overwhelms the servers. To prevent such queries to GraphQL APIs, I wrote graphql-query-complexity, an extensible open-source library that detects such queries and rejects pathological queries before consuming too many resources on the server. You can assign each field a complexity value, and queries that exceed a threshold will be rejected. In Slicknode this protection is added automatically based on the number of nodes that are being returned.