poolifier VS threads.js

At work I use https://github.com/pioardi/poolifier

We have an api used for generating reports from mongo dB to csv, this let's the report process in the background leaving the report api to still handle requests.

Since that I am familiar with Java programming language, I immediately started to look for thread pool implementations just like are present in java. There was only one small and not exhaustive thread pool at that time, so I started to implement poolifier. I choosed to implement two thread pools, a fixed-size thread pool and a dynamic-size thread pool so that users could tune and choose which one fits better for their applications.

Threads.js makes it pretty easy to spawn a thread pool and handle multi-threading workloads.

We did this via the npm threads wrapper and esbuild. We first moved all of our compute code into a new file with minimal imports, called imageCompare.worker.js. We then added a pre-compilation step with esbuild to compile this file into a bundle. We then spawn the worker using this generated file as a blob, and interact with it via the threads promise interface.

That latter one is in it's own process, and I also use multiple threads with the threads library.

This app has a memory leak where your worker threads are spawning out of control. You can verify this in Chrome DevTools Source tab: after you click the Generate button, you can see the worker threads spawning out of control until you close the process.

I'd advise that you do a better job at manually spawning and terminating the threads with the TFJS process. I've had success using this library in combination with TFJS: https://threads.js.org/

You can use it for everything. Even the compute tasks you can offload to worker threads using something like https://github.com/andywer/threads.js or you could create intensive modules in Rust and integrate using something like https://github.com/neon-bindings/neon

Web worker meets worker threads – threads.js\ (3 comments)

threads.js