HPC Rocket - A tool to run Slurm jobs from CI pipelines

• hpc-rocket

  1 24 4.8 Python

And finally, a link to the GitHub repo: https://github.com/SvenMarcus/hpc-rocket

• ZnTrack

  2 41 8.0 Python

  Create, visualize, run & benchmark DVC pipelines in Python & Jupyter notebooks.

  

This looks really interesting! I have a similar scenario but haven't looked into it yet. Have you looked at dvc.org - I'm planning on using it together with slurm and what they call CML for my projects. On that context I also wrote a tool that makes DVC more pythonic https://github.com/zincware/ZnTrack altough I'm currently restructuring it a bit but having backwards compatibility in mind.

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• dvc

  109 13,116 9.7 Python

  🦉 ML Experiments and Data Management with Git

  

This looks really interesting! I have a similar scenario but haven't looked into it yet. Have you looked at dvc.org - I'm planning on using it together with slurm and what they call CML for my projects. On that context I also wrote a tool that makes DVC more pythonic https://github.com/zincware/ZnTrack altough I'm currently restructuring it a bit but having backwards compatibility in mind.

• jacamar-ci

  2 - -

If you get the chance take a look at Jacamar CI, it is a Gitlab custom executor for Slurm developed by the Exascale Computing Project. While this doesn't address your specific problem set, needs installation on a login node, maybe you can get some inspiration.

Suggest a related project