How to achieve parallelism that retains order of tasks

• zerostash

  2 151 8.0 Rust

  Zero-metadata file storage

  

This is precisely the issue I had with file system backups: how do I maintain saturation when I have smaller and larger files? My solution was using Tokio and spawning a task for every chunk of a file and then collect the results in order. You may notice in the same file that I also process multiple files at the same time, all on the same thread pool basically waiting and spawning tasks in lockstep.

• Av1an

  65 1,356 7.5 Rust

  Cross-platform command-line AV1 / VP9 / HEVC / H264 encoding framework with per scene quality encoding

The order of encoding the video matters, but if you concatenate all the chunks at the end in the correct order, then you can do it in any order you want. You do have to concatenate the chunks at the end, but the speed of this process would actually not be affected by the order you encode the chunks, because the exact same concatenation code would run, and the same files exist at the end. Basically, I'm trying to port av1an to Rust and this is something that the original implementation does, by using concurrent.futures.ThreadPoolExecutor from the Python standard library.

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

Suggest a related project