Python HPC

Open-source Python projects categorized as HPC
Edit details

High Performance Computing. Scientists, engineers, and researchers use high performance computing (HPC) for incredibly computationally intensive tasks, like those used in weather forecasting, oil and gas exploration, physics, quantum mechanics, and other areas.

While HPC can be run on a single system, its real power comes from connecting multiple HPC nodes into a cluster, which may also be called a supercomputer. This provides the capabilities to compute extreme-scale simulations that are not feasible or even possible on a single system.

Topics: Python Machine Learning Containers Deep Learning GPU
The modern identity platform for B2B SaaS
The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
workos.com
Sponsored

Top 23 Python HPC Projects

  • ColossalAI

    Making large AI models cheaper, faster and more accessible

    Project mention: Making large AI models cheaper, faster and more accessible | news.ycombinator.com | 2024-03-21

  • spack

    A flexible package manager that supports multiple versions, configurations, platforms, and compilers.

    Project mention: Autodafe: "freeing your freeing your project from the clammy grip of autotools." | news.ycombinator.com | 2024-04-06

    > Are we talking about the same autotools?

    Yes. Instead of figuring out how to do something particular with every single software package, I can do a --with-foo or --without-bar or --prefix=/opt/baz-1.2.3, and be fairly confident that it will work the way I want.

    Certainly with package managers or (FreeBSD) Ports a lot is taken care of behind the scenes, but the above would also help the package/port maintainers as well. Lately I've been using Spack for special-needs compiles, but maintainer ease also helps there, but there are still cases one a 'fully manual' compile is still done.

    > Suffice it to say, I prefer to work with handwritten makefiles.

    Having everyone 'roll their own' system would probably be worse, because any "mysteriously failure" then has to be debugged specially for each project.

    Have you tried Spack?

    * https://spack.io

    * https://spack.readthedocs.io/en/latest/

  • WorkOS

    The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  • udocker

    A basic user tool to execute simple docker containers in batch or interactive systems without root privileges.

  • covalent

    Pythonic tool for orchestrating machine-learning/high performance/quantum-computing workflows in heterogeneous compute environments. (by AgnostiqHQ)

    Project mention: Remote execution of code | /r/Python | 2023-12-05

    Pretty interesting request, if SSH is not used, i would try using something like dask which uses tcp to connect and execute assuming your workers are in another machine.I also think something like covalent can be used to extend your own custom plugin in their ecosystem to connect how you want. We have a very custom private plugin written on top of covalent's to have a custom protocol to connect our central on-prem GPU machines to our local laptops that is rpc based, mostly for high performance as well as some mandate security from where the GPU machines are. Once done it is pretty much something like

  • prometeo

    An experimental Python-to-C transpiler and domain specific language for embedded high-performance computing

  • devito

    DSL and compiler framework for automated finite-differences and stencil computation

  • hpc-container-maker

    HPC Container Maker

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  • arrayfire-python

    Python bindings for ArrayFire: A general purpose GPU library.

  • jug

    Parallel programming with Python

  • easybuild-easyconfigs

    A collection of easyconfig files that describe which software to build using which build options with EasyBuild.

  • pyccel

    Python extension language using accelerators

  • deephyper

    DeepHyper: Scalable Asynchronous Neural Architecture and Hyperparameter Search for Deep Neural Networks

  • JAXFLUIDS

    Differentiable Fluid Dynamics Package

    Project mention: Show HN: Elodin – A better framework for physics simulation | news.ycombinator.com | 2024-03-06

    You are completely correct; right now it is just mechanics that we have built out. But, there isn't any theoretical reason you couldn't use this framework for other types of simulation. In particular, the Monte Carlo runner is super flexible. Since we are based on JAX you can utilize a ton of the tooling that others have built in the physics space like https://github.com/tumaer/JAXFLUIDS or https://github.com/DifferentiableUniverseInitiative/jax_cosm... . The goal right now though is pretty firmly focused on controls engineers and their needs, but we envision this becoming broadly used.

  • arkouda

    Arkouda (αρκούδα): Interactive Data Analytics at Supercomputing Scale :bear:

    Project mention: Mojo is now available on Mac | news.ycombinator.com | 2023-10-19

    Those interested in the intersection between Python, HPC, and data science may want to take a look at Arkouda, which is a Python package for data science at massive scales (TB of memory) at interactive rates (seconds), powered by Chapel:

    * https://github.com/Bears-R-Us/arkouda

  • SmartSim

    SmartSim Infrastructure Library.

  • shenfun

    High performance computational platform in Python for the spectral Galerkin method

  • entangle

    A lightweight (serverless) native python parallel processing framework based on simple decorators and call graphs.

  • singularity-hpc

    Local filesystem registry for containers (intended for HPC) using Lmod or Environment Modules. Works for users and admins.

  • luna

    Provisioning tool for clusters (by dchirikov)

  • waldur-mastermind

    Waldur MasterMind is a hybrid cloud orchestrator.

  • pavilion2

    Pavilion is a Python 3 (3.5+) based framework for running and analyzing tests targeting HPC systems.

  • spack-manager

    A project and machine deployment model using Spack

    Project mention: C++ Package Managers: The Ultimate Roundup | news.ycombinator.com | 2024-02-24

    There is also `spack develop`, on which people are building hack/build/test loops. You can `spack develop` any package in spack, and easily build with a modified version of something.

    See also https://github.com/sandialabs/spack-manager.

  • hpc-rocket

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2024-04-06.

Python HPC related posts

Index

What are some of the best open-source HPC projects in Python? This list will help you:

Project Stars
1 ColossalAI 37,775
2 spack 3,938
3 udocker 1,209
4 covalent 687
5 prometeo 610
6 devito 517
7 hpc-container-maker 435
8 arrayfire-python 413
9 jug 407
10 easybuild-easyconfigs 344
11 pyccel 330
12 deephyper 261
13 JAXFLUIDS 242
14 arkouda 224
15 SmartSim 211
16 shenfun 180
17 entangle 105
18 singularity-hpc 102
19 luna 58
20 waldur-mastermind 44
21 pavilion2 41
22 spack-manager 24
23 hpc-rocket 23

Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com