The gist VS Moleculer

Graphile worker has been great for me because it's a library that works with Postgres that allows me to queue jobs and execute them on the server without adding too many additional layers of complexity for being able to accomplish async tasks. (I'm aware of how popular bull is, but I don't want to add another data-store only for async tasks)

Bull is currently in maintenance mode, we are only fixing bugs. For new features check BullMQ, a modern rewritten implementation in Typescript. You are still very welcome to use Bull if it suits your needs, which is a safe, battle-tested library.

I use bullmq with node

You might consider using a queue for processing the request. I've found bullMQ, which works with Redis, to be a nice developer experience.

BullMQ is a pretty solid choice: https://github.com/taskforcesh/bullmq It's the successor of Bull: https://github.com/OptimalBits/bull

BullMQ

For Node.JS you can use something like BullMQ (https://github.com/taskforcesh/bullmq) and then dispatch jobs to the message queue with your worker handling the jobs. You can read about an example for Bull MQ here (https://deadsimplechat.com/blog/best-nodejs-schedulers/#2-bull)

bullmq seems like an open issue

In order to try and mitigate the OOMs. I read the Background Jobs in Node.JS with Redis blog post and implemented Heroku Data For Redis with ioredis, BullMQ and Throng,

My goto for this kind of task is moleculer: https://moleculer.services/

Fast, battle tested, vue2-like approach, great documentation, good community. The automatic indipendent-scalability as an option is usually the main selling point of these solutions, but honestly I think the real pro is the "composition" approach, which is essential if you want to keep a clean and well-organized codebase. On this regard, I found moleculer pretty great even for large teams.

If you’re using k8s, check out https://moleculer.services and this would likely solve what you’re looking for.

Molecular – Progressive Microservices Framework for Node.js

While you’re delving into microservices, check out Moleculer https://moleculer.services

I almost can’t believe I haven’t seen it mentioned here before, but adding Moleculer into your node project (if it’s clustered/k8s’d) will literally solve many single threaded problems, not to mention tons of other scalability issues. https://moleculer.services/

You could use a framework for this. Have a look at moleculer

I haven't used this, but it seems neat: https://moleculer.services/

Look into Moleculer.

Moleculer already provides an in-house http gateway, but what if you want to use an existing API, and how to maintain decoupled code when creating your gateway? This package solves both. You can create your API, passing in any services you require as dependencies. You can then bind your API to moleculer using the moleculer-universal-gateway.

But there's more to the topic of microservices. Seems like all the conversation focuses on deployment pain. You can build a service with something like Akka or Moleculer where the modules act independently and have some message passing and resilience from each other, but they can still all live in one codebase, one process, and deployed as one unit. It works fine and isn't painful at all. And maybe down the line you decide to split the thing up into multiple processes and multiple deployment units, and that's an easy refactor because the modules are already somewhat separated.