Open-source Go projects categorized as HPC | Edit details

High Performance Computing. Scientists, engineers, and researchers use high performance computing (HPC) for incredibly computationally intensive tasks, like those used in weather forecasting, oil and gas exploration, physics, quantum mechanics, and other areas.

While HPC can be run on a single system, its real power comes from connecting multiple HPC nodes into a cluster, which may also be called a supercomputer. This provides the capabilities to compute extreme-scale simulations that are not feasible or even possible on a single system.

Top 6 Go HPC Projects

  • GitHub repo singularity

    Singularity: Application containers for Linux

    Project mention: Singularity – Open Source Container System for Secure High Performance Computing | reddit.com/r/CKsTechNews | 2021-11-06
  • GitHub repo volcano

    A Cloud Native Batch System (Project under CNCF)

    Project mention: My Journey With Spark On Kubernetes... In Python (1/3) | dev.to | 2021-04-12

    For our experiments, we will use Volcano which is a batch scheduler for Kubernetes, well-suited for scheduling Spark applications pods with a better efficiency than the default kube-scheduler. The main reason is that Volcano allows "group scheduling" or "gang scheduling": while the default scheduler of Kubernetes schedules containers one by one, Volcano ensures that a gang of related containers (here, the Spark driver and its executors) can be scheduled at the same time. If for any reason it is not possible to deploy all the containers in a gang, Volcano will not schedule that gang. This article explains in more detail the reasons for using Volcano.

  • OPS

    OPS - Build and Run Open Source Unikernels. Quickly and easily build and deploy open source unikernels in tens of seconds. Deploy in any language to any cloud.

  • GitHub repo kube-batch

    A batch scheduler of kubernetes for high performance workload, e.g. AI/ML, BigData, HPC

    Project mention: Scaling Kubernetes to 7,500 Nodes | news.ycombinator.com | 2021-01-25

    > That said, strain on the kube-scheduler is spiky. A new job may consist of many hundreds of pods all being created at once, then return to a relatively low rate of churn.

    Last I checked, the default scheduler places Pods one at a time. It might be advantageous to use a gang/batch scheduler like kube-batch[0], Poseidon[1] or DCM[2].

    [0] https://github.com/kubernetes-sigs/kube-batch

    [1] https://github.com/kubernetes-sigs/poseidon

    [2] https://github.com/vmware/declarative-cluster-management

  • GitHub repo singularity

    SingularityCE is the Community Edition of Singularity, an open source container platform designed to be simple, fast, and secure. (by sylabs)

    Project mention: ELI5: Why does the FreeBSD community hate Docker and Kubernetes so much? | reddit.com/r/freebsd | 2021-11-26

    Docker (and the current generation of OCI runtimes) is rather shitty at host OS isolation too. It may have started well, but over time more capabilities were added for the convenience of developers and at the cost of maintainers. I would personally love if the ecosystem were to shift to more isolated workloads with better HPC support. Singualrity looks promising but still maintains OCI compatibility.

  • GitHub repo warewulf

    Warewulf is a stateless and diskless container operating system provisioning system for large clusters of bare metal and/or virtual systems.

    Project mention: going to replace our old cluster, which way xCat or Warewulf? | reddit.com/r/HPC | 2021-04-11
  • GitHub repo kraken

    Kraken is a distributed state engine framework for scalable automation and orchestration tools. (by kraken-hpc)

    Project mention: Building a bare-metal Kubernetes cluster on Raspberry Pi | news.ycombinator.com | 2021-11-22


    Kraken is a state engine. In the case of the Pi cluster it would work to provide network images, install and configure nodes, reboot nodes, and manipulate their state as needed to keep things running.

    Another fun piece of research on Pis out of LANL is that at their altitude, 7000ft, they estimate that on average a Raspberry Pi will crash once every 2 years due to bit flips caused by cosmic rays.

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2021-11-26.

Go HPC related posts


What are some of the best open-source HPC projects in Go? This list will help you:

Project Stars
1 singularity 2,295
2 volcano 2,148
3 kube-batch 904
4 singularity 160
5 warewulf 65
6 kraken 44
Find remote jobs at our new job board 99remotejobs.com. There are 29 new remote jobs listed recently.
Are you hiring? Post a new remote job listing for free.
Less time debugging, more time building
Scout APM allows you to find and fix performance issues with no hassle. Now with error monitoring and external services monitoring, Scout is a developer's best friend when it comes to application development.