Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality. Learn more →
Top 23 Parallel Open-Source Projects
-
Ray
Ray is a unified framework for scaling AI and Python applications. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
22. Ray | Github | tutorial
-
yeah my code needs to use multiprocessing, which does not play nice with tqdm. thanks for the tip about positions though, that helped me search more effectively and came up with two promising comments. unmerged / require some workarounds, but might just work:
https://github.com/tqdm/tqdm/issues/1000#issuecomment-184208...
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
LightGBM
A fast, distributed, high performance gradient boosting (GBT, GBDT, GBRT, GBM or MART) framework based on decision tree algorithms, used for ranking, classification and many other machine learning tasks.
Project mention: SIRUS.jl: Interpretable Machine Learning via Rule Extraction | /r/Julia | 2023-06-29SIRUS.jl is a pure Julia implementation of the SIRUS algorithm by Bénard et al. (2021). The algorithm is a rule-based machine learning model meaning that it is fully interpretable. The algorithm does this by firstly fitting a random forests and then converting this forest to rules. Furthermore, the algorithm is stable and achieves a predictive performance that is comparable to LightGBM, a state-of-the-art gradient boosting model created by Microsoft. Interpretability, stability, and predictive performance are described in more detail below.
-
Documentation
-
Project mention: Optuna – A Hyperparameter Optimization Framework | news.ycombinator.com | 2024-04-06
I didn’t even know WandB did hyperparameter optimization, I figured it was a neural network visualizer based on 2 minute papers. Didn’t seem like many alternatives out there to Optuna with TPE + persistence in conditional continuous & discrete spaces.
Anyway, it’s doable to make a multi objective decide_to_prune function with Optuna, here’s an example https://github.com/optuna/optuna/issues/3450#issuecomment-19...
-
For parallel programming nowadays, personally I reach for C++ Taskflow when I really care about performance, or a mix of core.async and running multiple load balanced instances when I’m doing more traditional web backend stuff in Clojure.
-
Around the same time, I discovered Numba and was fascinated by how easily it could bring huge performance improvements to Python code.
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
Project mention: A list of JavaScript engines, runtimes, interpreters | /r/learnjavascript | 2023-12-10
Napa.js
-
Before we begin, it's essential to ensure that we have Tailwind CSS and Concurrently installed. Tailwind CSS utility classes will be used for styling our project and will not affect the functionality. Concurrently will allow us to run our React frontend and server file simultaneously on our machines. For now, knowing the purpose that Concurrently serves is enough. We will see how to make it work later in the article.
-
-
-
-
-
Project mention: parallel.js VS multithreading - a user suggested alternative | libhunt.com/r/parallel.js | 2024-01-10
-
Bluepill
Bluepill is a reliable iOS testing tool that runs UI tests using multiple simulators on a single machine
-
-
Project mention: Cypress E2E test automation breaks open-source integration for “security” | news.ycombinator.com | 2023-06-02
Cypress E2E testing framework will apparently break compatibility with very popular open-source dashboard https://sorry-cypress.dev/ which is used to watch the execution of your E2E tests.
This is from an email we received:
We recently released changes to encrypt our private Cypress API requests for improving network security and stability of the overall Cypress testing experience. We are currently rolling this out across our user base and expect this to be complete effective June 30, 2023. -
Interactive Parallel Computing with IPython
IPython Parallel: Interactive Parallel Computing in Python
-
-
singularity
Singularity has been renamed to Apptainer as part of us moving the project to the Linux Foundation. This repo has been persisted as a snapshot right before the changes.
-
Project mention: If you can't reproduce the model then it's not open-source | news.ycombinator.com | 2024-01-17
I think the process of data acquisition isn't so clear-cut. Take CERN as an example: they release loads of data from various experiments under the CC0 license [1]. This isn't just a few small datasets for classroom use; we're talking big-league data, like the entire first run data from LHCb [2].
On their portal, they don't just dump the data and leave you to it. They've got guides on analysis and the necessary tools (mostly open source stuff like ROOT [3] and even VMs). This means anyone can dive in. You could potentially discover something new or build on existing experiment analyses. This setup, with open data and tools, ticks the boxes for reproducibility. But does it mean people need to recreate the data themselves?
Ideally, yeah, but realistically, while you could theoretically rebuild the LHC (since most technical details are public), it would take an army of skilled people, billions of dollars, and years to do it.
This contrasts with open source models, where you can retrain models using data to get the weights. But getting hold of the data and the cost to reproduce the weights is usually prohibitive. I get that CERN's approach might seem to counter this, but remember, they're not releasing raw data (which is mostly noise), but a more refined version. Try downloading several petabytes of raw data if not; good luck with that. But for training something like a LLM, you might need the whole dataset, which in many cases have its own problems with copyrights…etc.
[1] https://opendata.cern.ch/docs/terms-of-use
[2] https://opendata.cern.ch/docs/lhcb-releases-entire-run1-data...
-
parallel-hashmap
A family of header-only, very fast and memory-friendly hashmap and btree containers.
Project mention: The One Billion Row Challenge in CUDA: from 17 minutes to 17 seconds | news.ycombinator.com | 2024-04-13Standard library maps/unordered_maps are themselves notoriously slow anyway. A sparse_hash_map from abseil or parallel-hashmaps[1] would be better.
-
This tutorial explains how to backup PostgreSQL database using pgBackRest and S3.
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
NOTE:
The open source projects on this list are ordered by number of github stars.
The number of mentions indicates repo mentiontions in the last 12 Months or
since we started tracking (Dec 2020).
The latest post mention was on 2024-04-13.
Parallel related posts
- Optuna – A Hyperparameter Optimization Framework
- MPL: Automatic Management of Parallelism
- Algorithms - Researchers Approach New Speed Limit for Seminal Problem
-
parallel.js VS multithreading - a user suggested alternative
2 projects | 10 Jan 2024
- How to test optimal parameters
- Optuna – A Hyperparameter Optimization Framework
- Experimental Parallel JSON Parser - vztpv/scj3, claujson (Using modified simdjson as Tokenizer, and some code of rapidjson - when ast->text, and std::thread)
-
A note from our sponsor - InfluxDB
www.influxdata.com | 18 Apr 2024
Index
What are some of the best open-source Parallel projects? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | Ray | 30,879 |
| 2 | tqdm | 27,353 |
| 3 | LightGBM | 16,025 |
| 4 | razzle | 11,082 |
| 5 | optuna | 9,615 |
| 6 | Taskflow | 9,520 |
| 7 | Numba | 9,404 |
| 8 | napajs | 9,238 |
| 9 | concurrently | 6,749 |
| 10 | npm-run-all | 5,620 |
| 11 | pandarallel | 3,486 |
| 12 | Index | 3,375 |
| 13 | qawolf | 3,273 |
| 14 | parallel.js | 3,209 |
| 15 | Bluepill | 3,178 |
| 16 | rslint | 2,661 |
| 17 | sorry-cypress | 2,635 |
| 18 | Interactive Parallel Computing with IPython | 2,548 |
| 19 | taskr | 2,527 |
| 20 | singularity | 2,495 |
| 21 | root | 2,411 |
| 22 | parallel-hashmap | 2,307 |
| 23 | pgBackRest | 2,177 |
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com