bull VS Commander.js

To implement the solution with a queue, I used a package called 'Bull' (https://github.com/OptimalBits/bull). It's a library that helps with distributed job control, providing some very useful solutions for this type of work, such as background job processing, queues with priorities (FIFO, LIFO, and others), among other features. 'Bull' uses Redis for queue storage, so if your application crashes for any reason, once it's back online, it will continue executing the processes that are in the queue. In our case, we'll use the FIFO (First in, first out) queue solution, meaning priority based on arrival order.

We use bull-js for our distributed queue and event-driven library.

I'm using the excellent Redis based Bull.js as a job queue on Kubernetes.

For Job Queue, NestJs provides a package named @nestjs/bull as an abstraction/wrapper on top of Bull, a popular, well-supported, high-performance Node. js-based Queue system implementation.

I'm working on a large ETL project that involves handling queues for file integration. Currently, we are using Redis with Bull (https://www.npmjs.com/package/bull) for this purpose. However, to streamline our architecture and address security concerns, we are considering migrating the queue to PostgreSQL.

BullMQ is a pretty solid choice: https://github.com/taskforcesh/bullmq It's the successor of Bull: https://github.com/OptimalBits/bull

For Node specifically, things like Streams, understanding the asynchronous model completely, and the event loop. Also, queues tend to get used a lot in Node, so understanding the basic concepts behind that and how to use something like bull would be useful.

To do this properly, you need to put the task into queue, so if there are 1000 simultaneous uploads it won't kill your server but will be processed one by one. For the queue, see bull. Image hostings like AWS have some functionality for processing on their side, as an option.

This is a memory intensive process though and Heroku is OOM'ing with R14 errors. For this they recommend migrating intensive work like this to a Background Job via Redis, implemented in Bull and Throng

Visit the Commander.js reference.

In the previous article, I looked at a Commander to configure and accept cli options and assign it to a variable called program.

You can write a command line utility using zx or commander.js. Hit a public api, spit stuff out in the console, etc.

In your index.js you can do whatever you want, even create an interactive CLI (check commander).

There is clearly a whole load of repetition, so this should be fairly easy to build and parameterise. Essentially this will all just be string building so we won't need to use any particular libraries for most of this script. We will need a way to call ffmpeg though - and ffmpeg will need to be present too, of course. To call a CLI command we can use the package commander.

Commander.js is an NPM package that makes it easier to build CLI tools. You can find its documentation over here

You can also use npm package commander to make more complex command line tool with lot of options and sub commands.

commander - npm - Required. A library that lets you define the commands and their arguments, options, help, etc.

Throughout my time writing and updating my static-site generator, I've been using npm from the very foundation I use an npm package called commander. Therefore, it is obvious that for the tool that I will be using to publish my ssg, I will do so with npm.

The command line has thousands of tools, such as awk, sed, grep, and find available at your disposal that cut development time and automate tedious tasks. Creating a command line tool in Node.js isn't very complicated, thanks to a powerful library like Commander.js.