Top 6 Go HPC Projects
Singularity: Application containers for LinuxProject mention: Singularity – Open Source Container System for Secure High Performance Computing | reddit.com/r/CKsTechNews | 2021-11-06
A Cloud Native Batch System (Project under CNCF)Project mention: My Journey With Spark On Kubernetes... In Python (1/3) | dev.to | 2021-04-12
For our experiments, we will use Volcano which is a batch scheduler for Kubernetes, well-suited for scheduling Spark applications pods with a better efficiency than the default kube-scheduler. The main reason is that Volcano allows "group scheduling" or "gang scheduling": while the default scheduler of Kubernetes schedules containers one by one, Volcano ensures that a gang of related containers (here, the Spark driver and its executors) can be scheduled at the same time. If for any reason it is not possible to deploy all the containers in a gang, Volcano will not schedule that gang. This article explains in more detail the reasons for using Volcano.
Run Linux Software Faster and Safer than Linux with Unikernels.
A batch scheduler of kubernetes for high performance workload, e.g. AI/ML, BigData, HPCProject mention: Scaling Kubernetes to 7,500 Nodes | news.ycombinator.com | 2021-01-25
> That said, strain on the kube-scheduler is spiky. A new job may consist of many hundreds of pods all being created at once, then return to a relatively low rate of churn.
Last I checked, the default scheduler places Pods one at a time. It might be advantageous to use a gang/batch scheduler like kube-batch, Poseidon or DCM.
SingularityCE is the Community Edition of Singularity, an open source container platform designed to be simple, fast, and secure. (by sylabs)Project mention: ELI5: Why does the FreeBSD community hate Docker and Kubernetes so much? | reddit.com/r/freebsd | 2021-11-26
Docker (and the current generation of OCI runtimes) is rather shitty at host OS isolation too. It may have started well, but over time more capabilities were added for the convenience of developers and at the cost of maintainers. I would personally love if the ecosystem were to shift to more isolated workloads with better HPC support. Singualrity looks promising but still maintains OCI compatibility.
Warewulf is a stateless and diskless container operating system provisioning system for large clusters of bare metal and/or virtual systems.Project mention: going to replace our old cluster, which way xCat or Warewulf? | reddit.com/r/HPC | 2021-04-11
Kraken is a distributed state engine framework for scalable automation and orchestration tools. (by kraken-hpc)Project mention: Building a bare-metal Kubernetes cluster on Raspberry Pi | news.ycombinator.com | 2021-11-22
Kraken is a state engine. In the case of the Pi cluster it would work to provide network images, install and configure nodes, reboot nodes, and manipulate their state as needed to keep things running.
Another fun piece of research on Pis out of LANL is that at their altitude, 7000ft, they estimate that on average a Raspberry Pi will crash once every 2 years due to bit flips caused by cosmic rays.
Go HPC related posts
Building a bare-metal Kubernetes cluster on Raspberry Pi
10 projects | news.ycombinator.com | 22 Nov 2021
going to replace our old cluster, which way xCat or Warewulf?
1 project | reddit.com/r/HPC | 11 Apr 2021
Scaling Kubernetes to 7,500 Nodes
3 projects | news.ycombinator.com | 25 Jan 2021
What are some of the best open-source HPC projects in Go? This list will help you:
Are you hiring? Post a new remote job listing for free.