readable-stream VS Node.CLI-Progress

If it's a readable stream, you can just directly consume it with an async iterator: https://nodejs.org/api/stream.html#streams-compatibility-wit...

Node.js Stream

Note that the older Node.js streams API hasn't been deprecated or removed. It will co-exist with the new API, and it even provides a way to convert from a web stream and vice versa using the .fromWeb() and .toWeb() methods (both added in Node 17, although both methods still retain their experimental status).

Nodejs has piepes with its stream https://nodejs.org/api/stream.html

You then want what's invoked by that event to be consumed as a Response. NodeJS has something to replicate this with stream consumers where you can do all the response.text() you normally do.

[3] - https://github.com/nodejs/readable-stream/

First, we create a NodeJS ReadStream using fs.createWriteStream to open the test.log file in "append" mode. Then we start recursively looping using setTimeout. This simulates an application that logs text in the file at a variable rate. At each iteration:

This needs a comparison with streams, which is the standard way to achieve this in Node.js land.

https://nodejs.org/api/stream.html

The docs suck to read, but all the information is there https://nodejs.org/api/stream.html

Http-nas is a Node.js based API that simply streams files to and from a directory. We chose Node because it supports file streams and is generally very lightweight (Golang would have likely been another good choice). On Kubernetes you can attach a persistent volume claim to this service, enlarge it as needed, and as a Kubernetes service it can be accessed via http and the auto-created DNS entries derived from service names. Full concurrency is supported, at least as far as we decided to test this. We were interested in http as an interface since we wanted to access this API via Linux using cURL and cURL supports sending and receiving chunked file streams. This service is intended to be run on the same LAN as where the files originate, so we felt that we didn't need authentication in our first iteration. In the future we might embrace the concepts of zero-trust networking by adding some sort of authentication/validation layer (a simple option is simply http basic authentication).

Putting it all together as well as adding a cli progress by with cli-progress the ConversionService looks like this