ZnTrack
Create, visualize, run & benchmark DVC pipelines in Python & Jupyter notebooks. (by zincware)
jacamar-ci
By ecp-ci
| ZnTrack | jacamar-ci | |
|---|---|---|
| 2 | 2 | |
| 41 | - | |
| - | - | |
| 7.7 | - | |
| 6 days ago | - | |
| Python | ||
| Apache License 2.0 | - |
The number of mentions indicates the total number of mentions that we've tracked plus the number of user suggested alternatives.
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
ZnTrack
Posts with mentions or reviews of ZnTrack.
We have used some of these posts to build our list of alternatives
and similar projects. The last one was on 2023-04-05.
-
What are some good examples of well-engineered pipelines
I expaned a bit on them with my own package https://zntrack.readthedocs.io/ - a general framework for building DVC pipelines through python scripts (and more). This finally brings me to the project I'm actually working on https://github.com/zincware/IPSuite which brings all of this together for the specific use case of machine learned interatomic potentials.
-
HPC Rocket - A tool to run Slurm jobs from CI pipelines
This looks really interesting! I have a similar scenario but haven't looked into it yet. Have you looked at dvc.org - I'm planning on using it together with slurm and what they call CML for my projects. On that context I also wrote a tool that makes DVC more pythonic https://github.com/zincware/ZnTrack altough I'm currently restructuring it a bit but having backwards compatibility in mind.
jacamar-ci
Posts with mentions or reviews of jacamar-ci.
We have used some of these posts to build our list of alternatives
and similar projects. The last one was on 2022-01-03.
-
HPC Rocket - A tool to run Slurm jobs from CI pipelines
If you get the chance take a look at Jacamar CI, it is a Gitlab custom executor for Slurm developed by the Exascale Computing Project. While this doesn't address your specific problem set, needs installation on a login node, maybe you can get some inspiration.
-
HPC vs DataOps
We use Jacamar CI to integrate Slurm into our GitLab CI/CD pipelines. As GitLab allows for webhooks to trigger pipeline job you could use this to integrate something like Apache Airflow. There is probably a better or more direct way such as using the new Slurm Rest API (which I believe can be used to submit jobs).
What are some alternatives?
When comparing ZnTrack and jacamar-ci you can also consider the following projects:
fiftyone - The open-source tool for building high-quality datasets and computer vision models
dvc - 🦉 ML Experiments and Data Management with Git
mlem - 🐶 A tool to package, serve, and deploy any ML model on any platform. Archived to be resurrected one day🤞
hpc-rocket
Activeloop Hub - Data Lake for Deep Learning. Build, manage, query, version, & visualize datasets. Stream data real-time to PyTorch/TensorFlow. https://activeloop.ai [Moved to: https://github.com/activeloopai/deeplake]
messages - xcompute sub-module (CAE schema layer) including serialization utilities and bindings
IPSuite - Machine Learned Interatomic Potential Tools
example-get-started - Get started DVC project