proposal-arraybuffer-base64 VS streams

Uint8Array to/from Base64: Uint8Array<->base64/hex.

The proposal for native base64 support for Uint8Arrays is mine. I'm glad to see people are interested in using it. (So am I!)

For a status update, for the last year or two the main blocker has been a conflict between a desire to have streaming support and a desire to keep the API small and simple. That's now resolved [1] by dropping streaming support, assuming I can demonstrate a reasonably efficient streaming implementation on top of the one-shot implementation, which won't be hard unless "reasonably efficient" means "with zero copies", in which case we'll need to keep arguing about it.

I've also been working on documenting [2] the differences between various base64 implementations in other languages and in JS libraries to ensure we have a decent picture of the landscape when designing this.

With luck, I hope to advance the proposal to stage 3 ("ready for implementations") within the next two meetings of TC39 - so either next month or January. Realistically it will probably take a little longer than that, and of course implementations take a while. But it's moving along.

[1] https://github.com/tc39/proposal-arraybuffer-base64/issues/1...

[2] https://gist.github.com/bakkot/16cae276209da91b652c2cb3f612a...

Base64 for Uint8Array:ArrayBuffer to/from Base64

ArrayBuffer to/from base64: ArrayBuffer <-> base64 string functions.

I don’t know how ReadableStream.tee() got specified to backpressure when the faster branch is not consumed, since this is the opposite of what nodejs does when multiple Writables attached via Readable.pipe() and also the opposite of what the requirements document (https://github.com/whatwg/streams/blob/e9355ce79925947e8eb49...) says: “letting the speed of the slowest output determine the speed of the tee”.

I like the idea of the more ergonomic, faster api in new-stream with no buffering except at Stream.push(). NodeJS and web streams put infinitely expandable queues at every ReadableStream and WritableStream so that you can synchronously res.write(chunk) as much as you want with abandon. This API basically forces you to use generators that yield instead of synchronously writing chunks.

StarlingMonkey is a JavaScript runtime we’ve built together with our friends at Fastly and contributed to the Bytecode Alliance. It’s built on top of SpiderMonkey in a highly modular way, making it easy to configure as needed for our use case. Crucially, it comes with an implementation of key web APIs that substantially improve compatibility with the web ecosystem, like the fetch API for handling outgoing HTTP requests, key parts of the Service Workers spec for handling incoming requests, streaming processing of request and response bodies using the web’s Streams API streamssetTimeout, and setInterval.

Every single JS future is boxed. Moreover, they aren't just boxed, they are often backed by a hashmap (which may or may not be optimised away by the JIT). Elaborate allocation-free async is not an apple-to-apples comparison, that's my point.

JS does support concurrent execution, Promise.all is an example. Without it, JS async would make little sense. The problem very much exists there, and try-catch is only a surface-level answer. As you can see here [1], the interaction of cancellation and async in JS is at least just as (or more) complex than in Rust.

By the way, multithreading has little to do with Pin. I presume you're thinking of Send bounds.

"To work at all" is very dismissive. It's complex, but very well abstracted, well defined, and robust, that complexity is essential. Again, look at [1], JS async is hardly less complex, but also much more vague and ill-defined.

[1]: https://github.com/whatwg/streams/issues/1255

Yup, this is what WHATWG's Streams spec[0] (linked in the article) says. It defines backpressure as a "process of normalizing flow from the original source according to how fast the chain can process chunks" where the reader "propagates a signal backwards through the pipe chain".

Mozilla's documentation[1] similarly defines backpressure as "the process by which a single stream or a pipe chain regulates the speed of reading/writing".

The article confuses backpressure (the signal used for regulation of the flow) with the reason backpressure is needed (producers and consumers working at different speeds). It should be fairly clear from the metaphor, I would have thought: With a pipe of unbounded size there is no pressure. The pressure builds up when consumer is slower than producer, which in turn slows down the producer. (Or the pipe explodes, or springs a leak and has to drop data on the ground.)

[0] https://streams.spec.whatwg.org/#pipe-chains

[1] https://developer.mozilla.org/en-US/docs/Web/API/Streams_API...

// https://streams.spec.whatwg.org/#example-transform-identity const { writable, readable } = new TransformStream(); fetch("...", { body: readable }).then(response => /* ... */); const writer = writable.getWriter(); writer.write(new Uint8Array([0x73, 0x74, 0x72, 0x65, 0x61, 0x6D, 0x73, 0x21])); // "streams!" writer.close();

// AudioWorkletStream // Stream audio from Worker to AudioWorklet // guest271314 2-24-2020 let port; onmessage = async e => { 'use strict'; if (!port) { [port] = e.ports; port.onmessage = event => postMessage(event.data); } const { urls } = e.data; // https://github.com/whatwg/streams/blob/master/transferable-streams-explainer.md const { readable, writable } = new TransformStream(); (async _ => { for await (const _ of (async function* stream() { while (urls.length) { yield (await fetch(urls.shift(), {cache: 'no-store'})).body.pipeTo(writable, { preventClose: !!urls.length, }); } })()); })(); port.postMessage( { readable, }, [readable] ); };