Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality. Learn more →
Top 23 HPC Open-Source Projects
-
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
-
spack
A flexible package manager that supports multiple versions, configurations, platforms, and compilers.
-
-
FluidX3D
The fastest and most memory efficient lattice Boltzmann CFD software, running on all GPUs via OpenCL.
-
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
singularity
Singularity has been renamed to Apptainer as part of us moving the project to the Linux Foundation. This repo has been persisted as a snapshot right before the changes.
-
-
-
-
-
-
-
udocker
A basic user tool to execute simple docker containers in batch or interactive systems without root privileges.
-
-
AdaptiveCpp
Implementation of SYCL and C++ standard parallelism for CPUs and GPUs from all vendors: The independent, community-driven compiler for C++-based heterogeneous programming models. Lets applications adapt themselves to all the hardware in the system - even at runtime!
-
cromwell
Scientific workflow engine designed for simplicity & scalability. Trivially transition between one off use cases to massive scale production environments
-
-
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
34. Julia - $74,963
Loads of people have stated why easy GPU interfaces are difficult to create, but we solve many difficult things all the time.
Ultimately I think CPUs are just satisfactory for the vast vast majority of workloads. Servers rarely come with any GPUs to speak of. The ecosystem around GPUs is unattractive. CPUs have SIMD instructions that can help. There are so many reasons not to use GPUs. By the time anyone seriously considers using GPUs they're, in my imagination, typically seriously starved for performance, and looking to control as much of the execution details as possible. GPU programmers don't want an automagic solution.
So I think the demand for easy GPU interfaces is just very weak, and therefore no effort has taken off. The amount of work needed to make it as easy to use as CPUs is massive, and the only reason anyone would even attempt to take this on is to lock you in to expensive hardware (see CUDA).
For a practical suggestion, have you taken a look at https://arrayfire.com/ ? It can run on both CUDA and OpenCL, and it has C++, Rust and Python bindings.
Project mention: Autodafe: "freeing your freeing your project from the clammy grip of autotools." | news.ycombinator.com | 2024-04-06> Are we talking about the same autotools?
Yes. Instead of figuring out how to do something particular with every single software package, I can do a --with-foo or --without-bar or --prefix=/opt/baz-1.2.3, and be fairly confident that it will work the way I want.
Certainly with package managers or (FreeBSD) Ports a lot is taken care of behind the scenes, but the above would also help the package/port maintainers as well. Lately I've been using Spack for special-needs compiles, but maintainer ease also helps there, but there are still cases one a 'fully manual' compile is still done.
> Suffice it to say, I prefer to work with handwritten makefiles.
Having everyone 'roll their own' system would probably be worse, because any "mysteriously failure" then has to be debugged specially for each project.
Have you tried Spack?
* https://spack.io
* https://spack.readthedocs.io/en/latest/
> It's been a while since you can rerun/resume Nextflow pipelines
Yes, you can resume, but you need your whole upstream DAG to be present. Snakemake can rerun a job when only the dependencies of that job are present, which allows to neatly manage the disk usage, or archive an intermediate state of a project and rerun things from there.
> and yes, you can have dry runs in Nextflow
You have stubs, which really isn't the same thing.
> I have no idea what you're referring to with the 'arbitrary limit of 1000 parallel jobs' though
I was referring to this issue: https://github.com/nextflow-io/nextflow/issues/1871. Except, the discussion doesn't give the issue a full justice. Nextflow spans each job in a separate thread, and when it tries to span 1000+ condor jobs it die with a cryptic error message. The option of -Dnxf.pool.type=sync and -Dnxf.pool.maxThreads=N prevents the ability to resume and attempts to rerun the pipeline.
> As for deleting temporary files, there are features that allow you to do a few things related to that, and other features being implemented.
There are some hacks for this - but nothing I would feel safe to integrate into a production tool. They are implementing something - you're right - and it's been the case for several years now, so we'll see.
Snakemake has all that out of the box.
Project mention: What downsides exist to Futhark? Seems almost too good to be true? | /r/CUDA | 2023-06-29Why Futhark? (futhark-lang.org)
There is a recent update to the blis alternative to BLAS that includes a number of RISC-V performance optimizations.
https://github.com/flame/blis/pull/737
The commercial Linux world (see https://github.com/open-mpi/ompi/issues/4349) and other open source OSes (eg FreeBSD) seem to have lined up behind little-endian PowerPC. IBM still has a big-endian problem with AIX, IBM i, and Linux on Z.
Thanks, @subharmonicon!
While Chapel can run on many different systems, the main goal is making HPC programming much easier. Therefore, we are currently focusing on hardware that you can find in HPC systems (NVIDIA, AMD and Intel). Metal doesn't fall into that category, unfortunately. So far, the name came up infrequently in our discussions IIRC (especially targetting SPIRV), but we haven't heard from any [potential] user who may be interested in it. I would encourage you or anybody else interested in it to create an issue asking for the feature: https://github.com/chapel-lang/chapel/issues/new. Seeing public interest in that direction can change our prioritization.
One thing that I wanted to add that's not in the blogpost is the "cpu-as-device" mode. With that mode, you can use any machine, even one without a GPU, to write applications using Chapel's GPU features. That mode is for those who want to do initial development/debugging on their personal laptops before putting their application on an HPC system. In other words, while you can't use Metal directly, you can still write GPU-enabled applications in your Mac using Chapel, if the end goal is to run it on an HPC system. More details on cpu-as-device: https://chapel-lang.org/docs/main/technotes/gpu.html#cpu-as-...
Project mention: [P] - VkFFT now supports quad precision (double-double) FFT computation on GPU | /r/MachineLearning | 2023-09-27Hello, I am the creator of the VkFFT - GPU Fast Fourier Transform library for Vulkan/CUDA/HIP/OpenCL/Level Zero and Metal. In the latest update, I have added support for quad-precision double-double emulation for FFT calculation on most modern GPUs. I understand that modern ML is going in the opposite low-precision direction, but I still think that it may be useful to have this functionality at least for some prototyping and development of concepts.
Project mention: An efficient C++17 GPU numerical computing library with Python-like syntax | /r/programming | 2023-10-05
Project mention: What Every Developer Should Know About GPU Computing | news.ycombinator.com | 2023-10-21Sapphire Rapids is a CPU.
AMD's primary focus for a GPU software ecosystem these days seems to be implementing CUDA with s/cuda/hip, so AMD directly supports and encourages running GPU software written in CUDA on AMD GPUs.
The only implementation for sycl on AMD GPUs that I can find is a hobby project that apparently is not allowed to use either the 'hip' or 'sycl' names. https://github.com/AdaptiveCpp/AdaptiveCpp
Project mention: I have decided to connect my future with Scala (if possible), need little advice | /r/scala | 2023-05-22What about https://github.com/broadinstitute/cromwell ?
Singularity is a container platform often used in HPC (high performance computing) because it doesn't require root.
From the git repo: https://github.com/apptainer/apptainer
> Apptainer was formerly known as Singularity and is now a part of the Linux Foundation.
NOTE:
The open source projects on this list are ordered by number of github stars.
The number of mentions indicates repo mentiontions in the last 12 Months or
since we started tracking (Dec 2020).
HPC related posts
- Ask HN: Recommendations for high quality, free CS books online
- A stateless and diskless container OS provisioning system for large clusters
- Making large AI models cheaper, faster and more accessible
- Algorithms for Modern Hardware
- Lisp/Scheme/Clojure and APL/K (2016)
- Ask HN: How can I learn about performance optimization?
- Apptainer (Formerly Singularity)
-
A note from our sponsor - InfluxDB
www.influxdata.com | 23 Apr 2024
Index
What are some of the best open-source HPC projects? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | julia | 44,469 |
| 2 | ColossalAI | 37,836 |
| 3 | ArrayFire | 4,395 |
| 4 | spack | 3,949 |
| 5 | volcano | 3,760 |
| 6 | FluidX3D | 3,193 |
| 7 | nextflow | 2,538 |
| 8 | singularity | 2,495 |
| 9 | futhark | 2,291 |
| 10 | blis | 2,079 |
| 11 | ompi | 2,008 |
| 12 | chapel | 1,734 |
| 13 | mfem | 1,537 |
| 14 | Boost.Compute | 1,496 |
| 15 | VkFFT | 1,441 |
| 16 | udocker | 1,209 |
| 17 | MatX | 1,115 |
| 18 | AdaptiveCpp | 1,037 |
| 19 | cromwell | 956 |
| 20 | future | 930 |
| 21 | RaftLib | 922 |
| 22 | apptainer | 904 |
| 23 | eve | 843 |
Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com