multipart-stream-rs VS tusd

The article talks about multipart/form-data in particular.

Another thing one might run across is multipart/x-mixed-replace. I wrote a crate for that. [1] I didn't see a spec for it, but someone since pointed out to me that it's probably identical to multipart/x-mixed, and now seeing an example in the multer README it clicks that I should have looked at RFC 1341, which says this:

> All subtypes of "multipart" share a common syntax, defined in this section.

...and written a crate general enough for all of them. Maybe I'll update my crate for that sometime. My crate currently assumes there's a Content-Length: for each part, which isn't specified there but makes sense in the context I use it. It wouldn't be hard to also support just the boundary delimiters. And then maybe add a form-data parser on top of that.

btw, the article also talks specifically about proxying the body. I don't get why they're parsing the multipart data at all. I presume they have a reason, but I don't see it explained. I'd expect that a body is a body is a body. You can stream it along, and perhaps also buffer it in case you want to support retrying the backhaul request, probably stopping the buffering at some byte limit at which you give up on the possibility of retries, because keeping arbitrarily large bodies around (in RAM or even spilling to SSD/disk) doesn't sound fun.

[1] https://crates.io/crates/multipart-stream

I find my implementation in parser.rs kind of gross, but at least it seems to work. If anyone happens to look, I'd appreciate tips for cleaning up this code.

If it’s one way (that wasn’t quite clear from the requirements to me).

take a look at https://tus.io/

We have an application tus - resumable file uploads we are deploying to Azure as an Azure App Service; a custom linux container. TusD exposes an endpoint /metrics that Prometheus is meant to scrape. Our App Service plan is set to rules-based scaling between 2 and 30 instances depending on load and traffic. The Azure Firewall is configured to not allow inbound connections to anything except via the load balancer (API management?).

HTTP/1 requests (uploads in this case) are also separate to some degree (though there are fairly stringent limits on connections per domain iirc which HTTP/2 resolves via the mentioned streams/multiplexing of connections).

The problem they have specifically would be that in a single request (form post for example) those uploads will be linear.

Solution really boils down to paralellizing the upload, using protocols/standards like https://tus.io/ or S3-compatible APIs to push the data up then syncronize with a record/document on the server.

hey hn, supabase ceo here

This release introduces a few new features to Supabase Storage: Resumable Uploads , Quality Filters, Next.js support, and WebP support.

As a reminder, Supabase Storage is for file storage, not to be confused with Postgres Storage. Resumable Uploads is the biggest update because it means that you can build more resilient apps: your users can continue uploading a file if their internet drops or if they accidentally close a browser tab

This implementation uses TUS[0], which is an open source protocol. We opted for this over s3's protocol to support the open source ecosystem. This means you can use several existing libraries and frameworks (like Uppy.js[1]) for multi-part uploads.

It also has some neat technical details, using Postgres Advisory Locks to solve concurrency issues.

The Storage team will be in the comments to cover any technical questions.

[0] TUS: https://tus.io/

[1] Uppy: https://uppy.io/docs/tus/

If you're going to upload semi large files (100 MB) and want resumability for that upload (i.e. it can resume if the connection breaks down) I would recommend using https://tus.io and tusdotnet . It's an open protocol, clients exist for a large range of languages and tusdotnet supports customizing the storage to send files directly to Azure blob storage using Xtensible.TusDotNet.Azure

I'm a huge fan of https://tus.io/ because of the multiple backends and frontends

I usually use https://tus.io/ as the base protocol. Not too difficult to implement.

Have a look at https://tus.io the resumable file upload protocol.

We use https://tus.io/ to upload large files, but it's pure JS + ASP.NET, not through Blazor. You can easily do this in Blazor Server app, but it will have JS.