HPC

Open-source projects categorized as HPC | Edit details

High Performance Computing. Scientists, engineers, and researchers use high performance computing (HPC) for incredibly computationally intensive tasks, like those used in weather forecasting, oil and gas exploration, physics, quantum mechanics, and other areas.

While HPC can be run on a single system, its real power comes from connecting multiple HPC nodes into a cluster, which may also be called a supercomputer. This provides the capabilities to compute extreme-scale simulations that are not feasible or even possible on a single system.

Top 23 HPC Open-Source Projects

  • GitHub repo julia

    The Julia Programming Language

    Project mention: Recursion absolutely necessary for distributed computing? | reddit.com/r/Julia | 2021-10-15

    Then, if everything is pure functions, instead of iteratively mutating an array, you can recursively do calls that make new arrays that change one element at a time. But wait, doesn't that sound very inefficient compared to mutation? Well yes it does! It would satisfy the purity argument to allow a compiler to know how to auto-parallelize, but it would be making so many temporary arrays that it would likely be slower than a good explicit loop. For this reason you need compiler optimizations which would remove the intermediate arrays and transform it under the hood to mutating code (see https://github.com/JuliaLang/julia/pull/42465 as an example of this in the Julia compiler). This is one optimization that is needed, another is the tail-call elimination that I mentioned earlier, etc. If you have all pure functions, and if all of these optimizations are perfect, then you can match the serial code performance of C/Fortran. But that is a big if, which is why you don't see successful BLAS's written in say Haskell (GHC is a good compiler but it's hard to make this perfect).

  • GitHub repo ArrayFire

    ArrayFire: a general purpose GPU library.

    Project mention: Arrayfire progressive performance decline? | reddit.com/r/rust | 2021-06-09

    Your Problem may be the lazy evaluation, see this issue: https://github.com/arrayfire/arrayfire/issues/1709

  • Scout APM

    Scout APM: A developer's best friend. Try free for 14-days. Scout APM uses tracing logic that ties bottlenecks to source code so you know the exact line of code causing performance issues and can get back to building a great product faster.

  • GitHub repo spack

    A flexible package manager that supports multiple versions, configurations, platforms, and compilers.

    Project mention: Finding a job related to FLOSS | reddit.com/r/freesoftware | 2021-10-16

    If you're in the US, you could look for jobs in research software engineering (we have an organization called US-RSE). You can also look into jobs at the National Laboratories - super-computing clusters run GNU/Linux and a lot of the software tends to be FLOSS. At a smaller scale, research computing at universities and industry also can have a somewhat similar environment. I was a sysadmin doing my graduate assistantship at my university's high performance computing cluster and I loved it. Probably the project I got to contribute to most was spack.

  • GitHub repo singularity

    Singularity: Application containers for Linux

    Project mention: Podman: A tool for managing OCI containers and pods | news.ycombinator.com | 2021-08-31

    This sounds exactly like Singularity http://singularity.hpcng.org/

  • GitHub repo volcano

    A Kubernetes Native Batch System (Project under CNCF)

    Project mention: My Journey With Spark On Kubernetes... In Python (1/3) | dev.to | 2021-04-12

    For our experiments, we will use Volcano which is a batch scheduler for Kubernetes, well-suited for scheduling Spark applications pods with a better efficiency than the default kube-scheduler. The main reason is that Volcano allows "group scheduling" or "gang scheduling": while the default scheduler of Kubernetes schedules containers one by one, Volcano ensures that a gang of related containers (here, the Spark driver and its executors) can be scheduled at the same time. If for any reason it is not possible to deploy all the containers in a gang, Volcano will not schedule that gang. This article explains in more detail the reasons for using Volcano.

  • GitHub repo futhark

    :boom::computer::boom: A data-parallel functional programming language

    Project mention: Futhark: Automatic wrapping of C headers in Nim | reddit.com/r/nim | 2021-09-22

    Futhark is already a name of an existing GPU programming language, you might consider changing the name of your utility.

  • GitHub repo ompi

    Open MPI main development repository

    Project mention: Distributed Training Made Easy with PyTorch-Ignite | dev.to | 2021-08-10

    backends from native torch distributed configuration: nccl, gloo, mpi.

  • Nanos

    Run Linux Software Faster and Safer than Linux with Unikernels.

  • GitHub repo Boost.Compute

    A C++ GPU Computing Library for OpenCL

  • GitHub repo udocker

    A basic user tool to execute simple docker containers in batch or interactive systems without root privileges.

    Project mention: Docker containers on termux? | reddit.com/r/termux | 2021-10-03

    What about, "udocker is a basic user tool to execute simple docker containers in user space without requiring root privileges," from https://github.com/indigo-dc/udocker?

  • GitHub repo kube-batch

    A batch scheduler of kubernetes for high performance workload, e.g. AI/ML, BigData, HPC

    Project mention: Scaling Kubernetes to 7,500 Nodes | news.ycombinator.com | 2021-01-25

    > That said, strain on the kube-scheduler is spiky. A new job may consist of many hundreds of pods all being created at once, then return to a relatively low rate of churn.

    Last I checked, the default scheduler places Pods one at a time. It might be advantageous to use a gang/batch scheduler like kube-batch[0], Poseidon[1] or DCM[2].

    [0] https://github.com/kubernetes-sigs/kube-batch

    [1] https://github.com/kubernetes-sigs/poseidon

    [2] https://github.com/vmware/declarative-cluster-management

  • GitHub repo mfem

    Lightweight, general, scalable C++ library for finite element methods

    Project mention: Intact Solutions is hiring a Software Engineer | reddit.com/r/fea | 2021-08-31

    We currently have several solvers of our own, as well as interfacing with MPI capable solvers via mfem. We are currently a pre and post processor for several NASTRAN based solvers using our meshing free method.

  • GitHub repo future

    :rocket: R package: future: Unified Parallel and Distributed Processing in R for Everyone

    Project mention: Running Code in Parallel | reddit.com/r/datascience | 2021-07-29

    Check out the future package: https://github.com/HenrikBengtsson/future

  • GitHub repo VkFFT

    Vulkan/CUDA/HIP/OpenCL Fast Fourier Transform library

    Project mention: VkFFT - GPU Fast Fourier Transform library API guide release | reddit.com/r/programming | 2021-08-22
  • GitHub repo RaftLib

    The RaftLib C++ library, streaming/dataflow concurrency via C++ iostream-like operators

  • GitHub repo ohpc

    OpenHPC Integration, Packaging, and Test Repo

    Project mention: CentOS 8.4 does not provide rdma.service and /etc/rdma/rdma.conf? | reddit.com/r/HPC | 2021-08-14
  • GitHub repo arrayfire-rust

    Rust wrapper for ArrayFire

    Project mention: Fast Linear Algebra library for Rust | reddit.com/r/rust | 2021-07-01

    I haven't tried it myself, but I believe that arrayfire-rust supports GPU.

  • GitHub repo openpbs

    An HPC workload manager and job scheduler for desktops, clusters, and clouds.

    Project mention: A PBS question. | reddit.com/r/HPC | 2021-01-23

    So the one in GitHub (https://github.com/openpbs/openpbs) is abandoned or still going on?

  • GitHub repo arrayfire-python

    Python bindings for ArrayFire: A general purpose GPU library.

    Project mention: Doubts on pyopencl | reddit.com/r/OpenCL | 2021-08-02

    You want an abstraction over OpenCL like ArrayFire

  • GitHub repo Fastor

    A lightweight high performance tensor algebra framework for modern C++

    Project mention: Use of BLAS vs direct SIMD for linear algebra library operations? | reddit.com/r/cpp | 2021-08-28

    Picking what size you are targeting is really important, though. Could the matrices you are working with realistically be bigger than say 32x32? BLAS is good for big matrices. It's not as great for small matrices. Eigen or Fastor will do better for these smaller problems. And for various common operations on sizes 2, 3, and 4, hand coded graphics-oriented libraries might outperform those.

  • GitHub repo EasyBuild

    EasyBuild - building software with ease

    Project mention: Pounding my head on the wall | reddit.com/r/SLURM | 2021-06-10

    Modules, in this context, refer to Environment Modules). The article would be better than any summary I could hastily provide, but I would recommend the functionally equivalent Lmod as an alternative. Furthermore, I would strongly recommend managing software installation via EasyBuild.

  • GitHub repo coherence

    Oracle Coherence Community Edition (by oracle)

    Project mention: A new kind of scope? | reddit.com/r/ProgrammingLanguages | 2021-10-12

    If you want to see an example, here's a Context interface from a multi-language compiler framework (compiling multiple different languages to Java byte-code) that I wrote years ago.

  • GitHub repo environment-modules Lmod

    Lmod: An Environment Module System based on Lua, Reads TCL Modules, Supports a Software Hierarchy

  • GitHub repo relion

    Image-processing software for cryo-electron microscopy

    Project mention: MOBO Tips for 2x 3090s | Ryzen 3900x CPU | reddit.com/r/buildapc | 2021-02-11

    Relion

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2021-10-16.

Index

What are some of the best open-source HPC projects? This list will help you:

Project Stars
1 julia 36,581
2 ArrayFire 3,638
3 spack 2,289
4 singularity 2,220
5 volcano 1,995
6 futhark 1,571
7 ompi 1,354
8 Boost.Compute 1,240
9 udocker 883
10 kube-batch 876
11 mfem 808
12 future 797
13 VkFFT 755
14 RaftLib 751
15 ohpc 598
16 arrayfire-rust 572
17 openpbs 480
18 arrayfire-python 380
19 Fastor 361
20 EasyBuild 349
21 coherence 341
22 environment-modules Lmod 325
23 relion 258
Find remote jobs at our new job board 99remotejobs.com. There are 34 new remote jobs listed recently.
Are you hiring? Post a new remote job listing for free.
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com