Get Scout setup in minutes, and let us sweat the small stuff. A couple lines in settings.py is all you need to start monitoring your apps. Sign up for our free tier today. Learn more →
Top 23 Python HPC Projects
-
-
Scout Monitoring
Free Django app performance insights with Scout Monitoring. Get Scout setup in minutes, and let us sweat the small stuff. A couple lines in settings.py is all you need to start monitoring your apps. Sign up for our free tier today.
-
spack
A flexible package manager that supports multiple versions, configurations, platforms, and compilers.
Project mention: Spack – a multi-platform, multi-version package manager for OS X, Windows, Linux | news.ycombinator.com | 2024-08-05 -
udocker
A basic user tool to execute simple docker containers in batch or interactive systems without root privileges.
-
covalent
Pythonic tool for orchestrating machine-learning/high performance/quantum-computing workflows in heterogeneous compute environments. (by AgnostiqHQ)
Pretty interesting request, if SSH is not used, i would try using something like dask which uses tcp to connect and execute assuming your workers are in another machine.I also think something like covalent can be used to extend your own custom plugin in their ecosystem to connect how you want. We have a very custom private plugin written on top of covalent's to have a custom protocol to connect our central on-prem GPU machines to our local laptops that is rpc based, mostly for high performance as well as some mandate security from where the GPU machines are. Once done it is pretty much something like
-
prometeo
An experimental Python-to-C transpiler and domain specific language for embedded high-performance computing
Project mention: Borgo is a statically typed language that compiles to Go | news.ycombinator.com | 2024-04-30Not impossible but I guess you might end up with an extra runtime layer and some more dynamic operations will not be very fast. Or you restrict it to a subset of Python like this project does: https://github.com/zanellia/prometeo
You could of course write a bytecode VM in Golang but I guess that defeats the purpose.
-
-
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
-
-
-
easybuild-easyconfigs
A collection of easyconfig files that describe which software to build using which build options with EasyBuild.
-
-
Project mention: Show HN: Elodin – A better framework for physics simulation | news.ycombinator.com | 2024-03-06
You are completely correct; right now it is just mechanics that we have built out. But, there isn't any theoretical reason you couldn't use this framework for other types of simulation. In particular, the Monte Carlo runner is super flexible. Since we are based on JAX you can utilize a ton of the tooling that others have built in the physics space like https://github.com/tumaer/JAXFLUIDS or https://github.com/DifferentiableUniverseInitiative/jax_cosm... . The goal right now though is pretty firmly focused on controls engineers and their needs, but we envision this becoming broadly used.
-
deephyper
DeepHyper: Scalable Asynchronous Neural Architecture and Hyperparameter Search for Deep Neural Networks
-
Those interested in the intersection between Python, HPC, and data science may want to take a look at Arkouda, which is a Python package for data science at massive scales (TB of memory) at interactive rates (seconds), powered by Chapel:
* https://github.com/Bears-R-Us/arkouda
-
-
-
singularity-hpc
Local filesystem registry for containers (intended for HPC) using Lmod or Environment Modules. Works for users and admins.
-
entangle
A lightweight (serverless) native python parallel processing framework based on simple decorators and call graphs.
-
-
-
pavilion2
Pavilion is a Python 3 (3.5+) based framework for running and analyzing tests targeting HPC systems.
-
There is also `spack develop`, on which people are building hack/build/test loops. You can `spack develop` any package in spack, and easily build with a modified version of something.
See also https://github.com/sandialabs/spack-manager.
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
NOTE:
The open source projects on this list are ordered by number of github stars.
The number of mentions indicates repo mentiontions in the last 12 Months or
since we started tracking (Dec 2020).
Python HPC discussion
Python HPC related posts
-
Spack – a multi-platform, multi-version package manager for OS X, Windows, Linux
-
Making large AI models cheaper, faster and more accessible
-
Spack Package Manager v0.21.0
-
Announcing Chapel 1.32
-
Cython 3.0 Released
-
Linux server for physics simulations
-
Prefect alternatives meant for Slurm (HPC)
-
A note from our sponsor - Scout Monitoring
www.scoutapm.com | 15 Oct 2024
Index
What are some of the best open-source HPC projects in Python? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | ColossalAI | 38,736 |
| 2 | spack | 4,255 |
| 3 | udocker | 1,344 |
| 4 | covalent | 761 |
| 5 | prometeo | 620 |
| 6 | devito | 559 |
| 7 | hpc-container-maker | 452 |
| 8 | arrayfire-python | 415 |
| 9 | jug | 412 |
| 10 | easybuild-easyconfigs | 375 |
| 11 | pyccel | 349 |
| 12 | JAXFLUIDS | 317 |
| 13 | deephyper | 274 |
| 14 | arkouda | 240 |
| 15 | SmartSim | 231 |
| 16 | shenfun | 197 |
| 17 | singularity-hpc | 111 |
| 18 | entangle | 104 |
| 19 | luna | 58 |
| 20 | waldur-mastermind | 48 |
| 21 | pavilion2 | 43 |
| 22 | hpc-rocket | 26 |
| 23 | spack-manager | 25 |
Sponsored
Free Django app performance insights with Scout Monitoring
Get Scout setup in minutes, and let us sweat the small stuff. A couple lines in settings.py is all you need to start monitoring your apps. Sign up for our free tier today.
www.scoutapm.com