piscina VS Bee-Queue

Workers multi-threading via Tinypool (a lightweight fork of Piscina)

For threading best to use piscina and for clustering best to use pm2 which does not even need one line of code change in Node.js code.

We used piscina at it was quite good: https://github.com/piscinajs/piscina

We used Benchmark.js for benchmarking and piscina as a pool for worker threads. benchmark.js was used to run the same code in 2 scenarios - one using a single thread and one using the piscina pool. The degree of parallelism was passed to the program via an environment variable. The test code is present in worker.js in both the cases.

piscina for managing node's worker threads

One other thing...this passed by in my feeds last week, I’m not an expert in web workers or browser threads etc, but it seems like it might be relevant, so sharing it just in case:

“A fast, efficient Node.js Worker Thread Pool implementation”

https://github.com/piscinajs/piscina

This is by no means a complete list of what piscina can do.... checkout the official docs here.

You could use a Redis based job system like https://github.com/bee-queue/bee-queue for maintenance and email. The jobs could be populated by the API service, some cron scripts, and whatever else, while you'd have dedicated worker processes (written in Node.js?) pulling the jobs and running them.

The Node.js libraries BullMQ and Bee Queue are also infrastructure and interface. However, you typically have to use their interface.

I'm a fan of bee-queue for this when working in Node, but it's just a wrapper around reliable queue patterns in Redis; you might want to investigate how it works (the readme is quite good) for your own use case, or find an equivalent for your environment. The key is it's designed for short tasks (seconds long, not minutes or hours), and you can receive a result back from the task unlike some fire-and-forget queueing solutions.

I'd also consider something like bee-queue, which would enable you to scale across processes or even across multiple server machines.

For NodeJS, there are BullMQ(successor of Bull), Bull and Bee Queue

bee-queue for offloading tasks onto non-web-server machines