Top 14 parallel-processing Open-Source Projects

• JobRunr

  7 2,162 9.3 Java

  An extremely easy way to perform background processing in Java. Backed by persistent storage. Open and free for commercial use.

  

Project mention: Open source Job Scheduler Library in java for high throughput | /r/java | 2023-11-20

In java we have found Quartz( Quartz) , Job Runr(Jobrunr) and db-scheduler(db-scheduler) and need to evaluate these for our use case.

• envpool

  3 1,021 4.2 C++

  C++-based high-performance parallel environment execution engine (vectorized env) for general RL environments.

  

Project mention: How do I improve my SB3 PPO on an EnvPool environment | /r/reinforcementlearning | 2023-09-21

I am looking to improve the overall performance as well as optimize the wall clock time. I slightly modified the code to develop a SB3 wrapper for envpool from here.

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• future

  2 932 8.1 R

  :rocket: R package: future: Unified Parallel and Distributed Processing in R for Everyone

• fastcore

  2 901 6.2 Jupyter Notebook

  Python supercharged for the fastai library

  

Project mention: I decided not to commercialize nbdev | news.ycombinator.com | 2023-05-31

Yes, I encourage you to look at some example projects made with nbdev incase that is helpful:

- fastcore: https://github.com/fastai/fastcore - the nbs/ folder contains the source files that generate source code, docs and tests.

• parallel

  4 745 7.6 PHP

  An advanced parallelization library for PHP, enabling efficient multitasking, optimizing resource use, and application responsiveness through multiple CPU threads. (by amphp)

  

• p_tqdm

  1 438 0.0 Python

  Parallel processing with progress bars

Project mention: Need Help With Multiprocessing or Multithreading or whatever it's called | /r/learnpython | 2023-06-24

I avoid dealing directly with multiprocessing and asyncio unless absolutely necessary. I find I can parallelize a lot of tasks simply with the p_tqdm library which abstracts a lot of that away. Just be careful with it as it makes copies of anything you feed into it to get around the GIL.

• massiv

  0 382 5.9 Haskell

  Efficient Haskell Arrays featuring Parallel computation

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

• mug

  2 358 9.7 Java

  A small Java 8 library (string manipulation, stream utils)

  

Project mention: Parsing dates and time in Java | dev.to | 2024-04-28

So I started to build a library to help me parse all common dates and time strings. It should be able to infer from the input what the format specifiers should be (supports timezone, weekday, even including formats like 11/30/2024), and then use that format specifier to parse the date or time for me:

• BabelStream

  1 311 7.7 C++

  STREAM, for lots of devices written in many programming models

  

• yapp

  1 58 0.0 C++

  A parallel pipeline for stream processing

• didact-engine

  11 46 8.9 C#

  The REST API and execution engine for the Didact Platform.

  

Project mention: Anyone have experience with azure data factory for "small" data ETL? | /r/dataengineering | 2023-12-10

• pyDag

  2 24 0.0 Python

  Scheduling Big Data Workloads and Data Pipelines in the Cloud with pyDag

• bedrock-clump

  2 20 0.0 C++

  A GPU accelerated program to search the minecraft world for the largest connected clump of layer 5 bedrock

• librer

  1 12 9.3 Python

  A cross-platform GUI file cataloging program with extensive customization options to suit user preferences. Highly optimized for multi-core parallel search speed, data integrity, and repository portability.

Project mention: Librer – a highly customizable X-platform file cataloguing program | news.ycombinator.com | 2024-02-20

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

parallel-processing related posts

• Parsing dates and time in Java

  1 project | dev.to | 28 Apr 2024

• How do I improve my SB3 PPO on an EnvPool environment

  1 project | /r/reinforcementlearning | 21 Sep 2023

• Scheduling a job in Spring Boot

  2 projects | /r/SpringBoot | 14 Jun 2023

• amphp/parallel v2.0.0 has been released

  1 project | /r/PHP | 23 Feb 2023

• [D] Adding a new RL environment to envpool

  3 projects | /r/MachineLearning | 20 Dec 2022

• Is Quartz abandoned?

  1 project | /r/java | 2 Dec 2022

• Concurrency framework amphp has been installed nearly 27 million times in the last three years but I rarely hear it used in production maybe reason is my location. Do you use it for production?What kind of projects?

  1 project | /r/PHP | 11 Aug 2022
• A note from our sponsor - InfluxDB
  www.influxdata.com | 5 May 2024

  Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality. Learn more →

Index

Sponsored

SaaSHub - Software Alternatives and Reviews

SaaSHub helps you find the best software and product alternatives

www.saashub.com