HyperQueue: ergonomic HPC task executor written in Rust

• hyperqueue

  2 253 9.2 Rust

  Scheduler for sub-node tasks for HPC systems with batch scheduling

  

Hi! Me and my colleagues are working on an HPC (High Performance Computing)-focused tool for executing workflows/pipelines/task graphs in an ergonomic and efficient manner. It is called HyperQueue, it is open-source (https://github.com/It4innovations/hyperqueue) and it is of course written in Rust :) (previous incarnations of the tool were written in C++, which was a nightmare to maintain). It leverages async Rust heavily, not just for I/O (we use Tokio), but also for communicating between a number of independent "processes" (actors) running on a single thread.

• node_crunch

  15 78 0.0 Rust

  Discontinued Allows to distribute computations across several nodes

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• nextflow

  9 2,538 9.7 Groovy

  A DSL for data-driven computational pipelines

  

• aiida-hyperqueue

  1 4 4.9 Python

  AiiDA plugin for the HyperQueue metascheduler.

  

Suggest a related project