Ask HN: How to handle user file uploads?

• Uppy

  30 28,236 9.6 TypeScript

  The next open source file uploader for web browsers :dog:

  

Look at https://uppy.io/ open source and lot of integrations. You can keep moving to different levels of abstraction as required and see some good practices of how things are done.

• magika

  5 7,413 9.8 Python

  Detect file content types with deep learning

  

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• Smithereen

  8 353 9.5 Java

  Federated, ActivityPub-compatible social network server with friends, walls, and groups.

In my project[1], I convert all user-uploaded images to high-quality webp and store them like that. I discard the original files after the conversion. I use imgproxy[2] to further resize and convert them on the fly for actual display.

I don't do videos yet, but I'm kinda terrified of the idea of putting user-uploaded files through ffmpeg if/when I'll support them.

[1] https://github.com/grishka/Smithereen

[2] https://github.com/imgproxy/imgproxy

• imgproxy

  29 8,319 9.3 Go

  Fast and secure standalone server for resizing and converting remote images

  

In my project[1], I convert all user-uploaded images to high-quality webp and store them like that. I discard the original files after the conversion. I use imgproxy[2] to further resize and convert them on the fly for actual display.

I don't do videos yet, but I'm kinda terrified of the idea of putting user-uploaded files through ffmpeg if/when I'll support them.

[1] https://github.com/grishka/Smithereen

[2] https://github.com/imgproxy/imgproxy

• libvips

  24 9,148 9.2 C

  A fast image processing library with low memory needs.

  

Read through the comments and was surprised no one mentioned libvips - https://github.com/libvips/libvips. At my current small company we were trying to allow image uploads and started with imagemagick but certain images took too long to process and we were looking for faster alternatives. It's a great tool with minimum overhead. For video thumbnails, we use ffmpeg which is really heavy. We off-load video thumbnail generation to a queue. We've had great luck with these tools.

• imagor

  56 3,216 7.4 Go

  Fast, secure image processing server and Go library, using libvips

I have used https://github.com/cshum/imagor infront of S3 before and liked it, there is many (some commercial) offerings for this

• tus-js-client

  3 1,934 7.9 JavaScript

  A pure JavaScript client for the tus resumable upload protocol

  

We map the TUS[0] protocol to S3 multipart upload operations. This lets us obscure the S3 bucket from the client. The TUS operations are handled by a dedicated micro-service. It could be done in a Lambda or anything.

Once the upload completes we kick off a workflow to virus scan, unzip, decrypt, and process the file depending on what it is.

For virus scanning, we started with ClamAV[1], but eventually bought a Trend Micro product[2] for reasons that probably don't apply to you. It is serverless based on SQS, Lambda, and SNS. Works fine.

Once scanned, we do a number of things. For images that you are going to serve back out, you for sure want to re-encode those and strip metadata. I haven't worked directly on that part.

[0] https://tus.io/

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

• SwiftVips

  1 0 - Swift

  A lightning fast image processing and resizing library for Swift

+1 to vips! It's amazingly fast and stable. I even wrote (some minimal) Swift bindings for it to be used with a Swift backend: https://github.com/gh123man/SwiftVips

• photo_id_resizer

  6 11 2.6 Go

  Resize photo ID images using face recognition technology

Slight OT...

I created a program for profile pictures. It uses face recognition technology as to not deform faces when resizing photos. This may be useful to you.

https://github.com/jftuga/photo_id_resizer

• meson

  111 5,289 9.8 Python

  The Meson Build System

  

Suggest a related project