dataloader VS Knex

When I began working with GraphQL, I had concerns about the N+1 query problem. In my research, I came across the DataLoader pattern and its implementation on GitHub. While I explored various examples of its usage, I still struggled to grasp how it operates internally. Join me in delving a bit deeper into GraphQL DataLoader! :)

DataLoader: it is the standard solution to N+1 problem.

Additionally, you can use DataLoader or similar tools to optimize data fetching and avoid over-fetching or under-fetching data. Ultimately, performance and load tests ensure that your GraphQL API delivers optimal performance, meets response time expectations, and provides a smooth experience for users, even under heavy loads.

The most common practice is to turn N+1 into 1+1 using dataloaders (https://github.com/graphql/dataloader for JS, there are equivalents for most implementations). The N resolvers invoke a single batched loader which receives a list of keys and returns a list of values.

Let's take Payload, for example. Surprise, surprise. We have a relationship field, and it can store IDs to other related documents which are seamlessly merged in when you retrieve documents from the DB. We leverage the dataloader pattern to batch together all "populations" required for a given query, returning them all super fast and with as few separate queries to the DB as possible. We actually even outperform SQL-based frameworks quite a bit. In a purely relational test, we were 3x faster than Directus and 7x faster than Strapi while both were running Postgres, and we were on MongoDB.

Also, have a look at Facebook's Dataloader[0].

[0] https://github.com/graphql/dataloader

In the next article, I'm going to implement a GraphQL server with dataloader using the tools we introduced.

The one example of using Scoped dependency that comes to my mind, it's HTTP request level caching for libs like dataloader.

you would fetch these 1000 rows via dataloader that batches all requests for this relation to a single query... solving the n+1 issue

Knex.js

AdonisJS core team has created/maintains Lucid. It is a SQL query builder, and an Active Record ORM built on top of Knex.

Now, why not use an ORM? I've seen performance issues too many times with ORMs. I prefer writing my own SQL to avoid surprises. After all, I know the database schema and writing code for a specific purpose very often leads to better performance than generic code. ORMs have to support all kinds of database schemas. I only have to support mine. Having successfully used Knex.js in NodeJS (a popular query builder) in the past, I know writing SQL queries myself is not hard and provides very good performance.

Given the dynamic nature of the schema, we employ Knex, a query builder, for database access.

https://www.npmjs.com/package/knex - 779 issues

# related: https://github.com/knex/knex/issues/2354

This article will be focused on a security issue that I found in Knex and how to mitigate it, but I'll also talk briefly about the social aspects of this problem.

SQL is an old, irregular language to work with, but it is better known than HCL and SQL already has it's own Pulumi/CDK in the form of every ORM with introspection (like Javascript's Prisma, Python's Django, Go's XO etc) and QueryBuilder (LINQ, Knex, etc) in whatever programming language you prefer. You probably already know it.