HPC Rocket - A tool to run Slurm jobs from CI pipelines

This page summarizes the projects mentioned and recommended in the original post on /r/Python

InfluxDB - Power Real-Time Data Analytics at Scale
Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
www.influxdata.com
featured
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com
featured
  • hpc-rocket

  • And finally, a link to the GitHub repo: https://github.com/SvenMarcus/hpc-rocket

  • ZnTrack

    Create, visualize, run & benchmark DVC pipelines in Python & Jupyter notebooks.

  • This looks really interesting! I have a similar scenario but haven't looked into it yet. Have you looked at dvc.org - I'm planning on using it together with slurm and what they call CML for my projects. On that context I also wrote a tool that makes DVC more pythonic https://github.com/zincware/ZnTrack altough I'm currently restructuring it a bit but having backwards compatibility in mind.

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo
  • dvc

    🦉 ML Experiments and Data Management with Git

  • This looks really interesting! I have a similar scenario but haven't looked into it yet. Have you looked at dvc.org - I'm planning on using it together with slurm and what they call CML for my projects. On that context I also wrote a tool that makes DVC more pythonic https://github.com/zincware/ZnTrack altough I'm currently restructuring it a bit but having backwards compatibility in mind.

  • If you get the chance take a look at Jacamar CI, it is a Gitlab custom executor for Slurm developed by the Exascale Computing Project. While this doesn't address your specific problem set, needs installation on a login node, maybe you can get some inspiration.

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts

  • Git Version Controlled Datasets in S3

    1 project | news.ycombinator.com | 25 Oct 2023
  • Where do I best store my test data when using github for code?

    1 project | /r/learnmachinelearning | 9 May 2023
  • Using git to version control experimental data (not code)?

    1 project | /r/git | 19 Apr 2023
  • What are some good examples of well-engineered pipelines

    3 projects | /r/ScientificComputing | 5 Apr 2023
  • Introduction to Data Version Control

    1 project | dev.to | 1 Apr 2023