Top 23 HPC Open-Source Projects
The Julia Programming LanguageProject mention: Recursion absolutely necessary for distributed computing? | reddit.com/r/Julia | 2021-10-15
Then, if everything is pure functions, instead of iteratively mutating an array, you can recursively do calls that make new arrays that change one element at a time. But wait, doesn't that sound very inefficient compared to mutation? Well yes it does! It would satisfy the purity argument to allow a compiler to know how to auto-parallelize, but it would be making so many temporary arrays that it would likely be slower than a good explicit loop. For this reason you need compiler optimizations which would remove the intermediate arrays and transform it under the hood to mutating code (see https://github.com/JuliaLang/julia/pull/42465 as an example of this in the Julia compiler). This is one optimization that is needed, another is the tail-call elimination that I mentioned earlier, etc. If you have all pure functions, and if all of these optimizations are perfect, then you can match the serial code performance of C/Fortran. But that is a big if, which is why you don't see successful BLAS's written in say Haskell (GHC is a good compiler but it's hard to make this perfect).
ArrayFire: a general purpose GPU library.Project mention: Arrayfire progressive performance decline? | reddit.com/r/rust | 2021-06-09
Your Problem may be the lazy evaluation, see this issue: https://github.com/arrayfire/arrayfire/issues/1709
Scout APM: A developer's best friend. Try free for 14-days. Scout APM uses tracing logic that ties bottlenecks to source code so you know the exact line of code causing performance issues and can get back to building a great product faster.
A flexible package manager that supports multiple versions, configurations, platforms, and compilers.Project mention: Finding a job related to FLOSS | reddit.com/r/freesoftware | 2021-10-16
If you're in the US, you could look for jobs in research software engineering (we have an organization called US-RSE). You can also look into jobs at the National Laboratories - super-computing clusters run GNU/Linux and a lot of the software tends to be FLOSS. At a smaller scale, research computing at universities and industry also can have a somewhat similar environment. I was a sysadmin doing my graduate assistantship at my university's high performance computing cluster and I loved it. Probably the project I got to contribute to most was spack.
Singularity: Application containers for Linux
A Kubernetes Native Batch System (Project under CNCF)Project mention: My Journey With Spark On Kubernetes... In Python (1/3) | dev.to | 2021-04-12
For our experiments, we will use Volcano which is a batch scheduler for Kubernetes, well-suited for scheduling Spark applications pods with a better efficiency than the default kube-scheduler. The main reason is that Volcano allows "group scheduling" or "gang scheduling": while the default scheduler of Kubernetes schedules containers one by one, Volcano ensures that a gang of related containers (here, the Spark driver and its executors) can be scheduled at the same time. If for any reason it is not possible to deploy all the containers in a gang, Volcano will not schedule that gang. This article explains in more detail the reasons for using Volcano.
:boom::computer::boom: A data-parallel functional programming languageProject mention: Futhark: Automatic wrapping of C headers in Nim | reddit.com/r/nim | 2021-09-22
Futhark is already a name of an existing GPU programming language, you might consider changing the name of your utility.
Open MPI main development repositoryProject mention: Distributed Training Made Easy with PyTorch-Ignite | dev.to | 2021-08-10
backends from native torch distributed configuration: nccl, gloo, mpi.
Run Linux Software Faster and Safer than Linux with Unikernels.
A C++ GPU Computing Library for OpenCL
A basic user tool to execute simple docker containers in batch or interactive systems without root privileges.Project mention: Docker containers on termux? | reddit.com/r/termux | 2021-10-03
What about, "udocker is a basic user tool to execute simple docker containers in user space without requiring root privileges," from https://github.com/indigo-dc/udocker?
A batch scheduler of kubernetes for high performance workload, e.g. AI/ML, BigData, HPCProject mention: Scaling Kubernetes to 7,500 Nodes | news.ycombinator.com | 2021-01-25
> That said, strain on the kube-scheduler is spiky. A new job may consist of many hundreds of pods all being created at once, then return to a relatively low rate of churn.
Last I checked, the default scheduler places Pods one at a time. It might be advantageous to use a gang/batch scheduler like kube-batch, Poseidon or DCM.
Lightweight, general, scalable C++ library for finite element methodsProject mention: Intact Solutions is hiring a Software Engineer | reddit.com/r/fea | 2021-08-31
We currently have several solvers of our own, as well as interfacing with MPI capable solvers via mfem. We are currently a pre and post processor for several NASTRAN based solvers using our meshing free method.
:rocket: R package: future: Unified Parallel and Distributed Processing in R for EveryoneProject mention: Running Code in Parallel | reddit.com/r/datascience | 2021-07-29
Check out the future package: https://github.com/HenrikBengtsson/future
Vulkan/CUDA/HIP/OpenCL Fast Fourier Transform libraryProject mention: VkFFT - GPU Fast Fourier Transform library API guide release | reddit.com/r/programming | 2021-08-22
The RaftLib C++ library, streaming/dataflow concurrency via C++ iostream-like operators
OpenHPC Integration, Packaging, and Test RepoProject mention: CentOS 8.4 does not provide rdma.service and /etc/rdma/rdma.conf? | reddit.com/r/HPC | 2021-08-14
Rust wrapper for ArrayFireProject mention: Fast Linear Algebra library for Rust | reddit.com/r/rust | 2021-07-01
I haven't tried it myself, but I believe that arrayfire-rust supports GPU.
An HPC workload manager and job scheduler for desktops, clusters, and clouds.Project mention: A PBS question. | reddit.com/r/HPC | 2021-01-23
So the one in GitHub (https://github.com/openpbs/openpbs) is abandoned or still going on?
Python bindings for ArrayFire: A general purpose GPU library.Project mention: Doubts on pyopencl | reddit.com/r/OpenCL | 2021-08-02
You want an abstraction over OpenCL like ArrayFire
A lightweight high performance tensor algebra framework for modern C++Project mention: Use of BLAS vs direct SIMD for linear algebra library operations? | reddit.com/r/cpp | 2021-08-28
Picking what size you are targeting is really important, though. Could the matrices you are working with realistically be bigger than say 32x32? BLAS is good for big matrices. It's not as great for small matrices. Eigen or Fastor will do better for these smaller problems. And for various common operations on sizes 2, 3, and 4, hand coded graphics-oriented libraries might outperform those.
EasyBuild - building software with easeProject mention: Pounding my head on the wall | reddit.com/r/SLURM | 2021-06-10
Modules, in this context, refer to Environment Modules). The article would be better than any summary I could hastily provide, but I would recommend the functionally equivalent Lmod as an alternative. Furthermore, I would strongly recommend managing software installation via EasyBuild.
Oracle Coherence Community Edition (by oracle)Project mention: A new kind of scope? | reddit.com/r/ProgrammingLanguages | 2021-10-12
If you want to see an example, here's a Context interface from a multi-language compiler framework (compiling multiple different languages to Java byte-code) that I wrote years ago.
Lmod: An Environment Module System based on Lua, Reads TCL Modules, Supports a Software Hierarchy
Image-processing software for cryo-electron microscopyProject mention: MOBO Tips for 2x 3090s | Ryzen 3900x CPU | reddit.com/r/buildapc | 2021-02-11
What are some of the best open-source HPC projects? This list will help you:
Are you hiring? Post a new remote job listing for free.