GNU Parallel, where have you been all my life?

This page summarizes the projects mentioned and recommended in the original post on news.ycombinator.com

Our great sponsors
  • WorkOS - The modern API for authentication & user identity.
  • LearnThisRepo.com - Learn 300+ open source libraries for free using AI.
  • InfluxDB - Power Real-Time Data Analytics at Scale
  • bu

    B)asic|But-For U)tility Code/Programs (in Nim & Often Unix/POSIX/Linux Context)

    This sounds like a job for what standard C calls "popen". You can do `import posix; for line in popen("ls", "r"): echo line` in Nim, though you obviously need to replace `echo line` with other desired processing and learn how to do that.

    You might also want to consider `rp` which is a program generator-compiler-runner along the lines of `awk` but with all the code just Nim snippets interpolated into a program template: https://github.com/c-blake/bu/blob/main/doc/rp.md . E.g.:

        ls -l | rp -pimport\ stats -bvar\ r:RunningStat -wnf\>4 r.push\ 4.f -eecho\ r

  • pash

    PaSh: Light-touch Data-Parallel Shell Processing

  • WorkOS

    The modern API for authentication & user identity. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  • PPSS

    Parallel Processing Shell Script

  • fish-shell

    The user-friendly command line shell.

  • nushell

    A new type of shell

  • progress

    A simple-text subtask progress standard and accompanying utilities (by titzer)

    I didn't know about this, and reading through the comments, I found out that xargs can also do batching and parallelism (nice!). However, it appears that if you pipe the output of an xargs-parallel command into another utility, it jumbles the output of the multiple subprocesses, whereas GNU parallel does not.

    I was a little put off by the annoying/scary citation issue mentioned by another commenter, so I am not sure I will use parallel.

    I want to pipe the output of parallel processes into a utility that I wrote for progress printing (https://github.com/titzer/progress), but I think that neither of these solutions work; my progress utility will have to do this on its own.

  • script

    Making it easy to write shell-like scripts in Go

    I use Go. You can run scripts with go run directly, and this package makes shell tasks easy: https://github.com/bitfield/script

  • LearnThisRepo.com

    Learn 300+ open source libraries for free using AI. LearnThisRepo lets you learn 300+ open source repos including Postgres, Langchain, VS Code, and more by chatting with them using AI!

  • jo

    JSON output from a shell

    That should recursively list directories, counting only the files within each, and output² jsonl that can be further mangled within the shell². You could just as easily populate an associative array for further work, or $whatever. Unlike bash, zsh has reasonable behaviour around quoting and whitespace too.

    ¹ https://zsh.sourceforge.io/Doc/Release/User-Contributions.ht...

    ² https://github.com/jpmens/jo

    ³ https://github.com/stedolan/jq

  • jq

    Command-line JSON processor [Moved to: https://github.com/jqlang/jq] (by stedolan)

    That should recursively list directories, counting only the files within each, and output² jsonl that can be further mangled within the shell². You could just as easily populate an associative array for further work, or $whatever. Unlike bash, zsh has reasonable behaviour around quoting and whitespace too.

    ¹ https://zsh.sourceforge.io/Doc/Release/User-Contributions.ht...

    ² https://github.com/jpmens/jo

    ³ https://github.com/stedolan/jq

  • wrk2

    A constant throughput, correct latency recording variant of wrk

    > This runs a benchmark for 30 seconds, using 2 threads, keeping 100 HTTP connections open, and a constant throughput of 2000 requests per second (total, across all connections combined).

    Some distros include `ab`[2] which is also good, but wrk2 improves on it (and on wrk version 1) in multiple ways, so that's what I use myself.

    [1] https://github.com/giltene/wrk2

    [2] https://httpd.apache.org/docs/2.2/programs/ab.html

  • xe

    simple xargs and apply replacement

    I also recommend checking out `xe`: https://github.com/leahneukirchen/xe

    It’s like xargs with sane defaults and a couple tricks of its own.

  • task-spooler

    A scheduler for GPU/CPU tasks

    Wow this tool is fantastic, thank you! The UI is very nice and simple. How has this not existed in Unix for 30+ years?

    https://github.com/justanhduc/task-spooler

  • shell

    A Nim mini DSL to execute shell commands (by Vindaar)

    On Unix, you might try Nim (https://nim-lang.org) with https://github.com/Vindaar/shell and there are a slew of pipeline-y/Unix-y utilities over at https://github.com/c-blake/bu

    Nim is statically typed and (generally) native-compiled, but it has very low ceremony ergonomics and a powerful compile-time macro/template system as well as user-defined operators (e.g., you can use `+-` to make a constructor for uncertain values so that `9 +- 2` builds a typed object.

  • Nim

    Nim is a statically typed compiled systems programming language. It combines successful concepts from mature languages like Python, Ada and Modula. Its design focuses on efficiency, expressiveness, and elegance (in that order of priority).

    On Unix, you might try Nim (https://nim-lang.org) with https://github.com/Vindaar/shell and there are a slew of pipeline-y/Unix-y utilities over at https://github.com/c-blake/bu

    Nim is statically typed and (generally) native-compiled, but it has very low ceremony ergonomics and a powerful compile-time macro/template system as well as user-defined operators (e.g., you can use `+-` to make a constructor for uncertain values so that `9 +- 2` builds a typed object.

  • blender_gnu_parallel_render

    Use GNU Parallel to render blender movies distributed by a bunch of nodes

    I use GNU Parallel to render Blender videos distributed by a bunch of nodes https://github.com/tfmoraes/blender_gnu_parallel_render

  • cligen

    Nim library to infer/generate command-line-interfaces / option / argument parsing; Docs at

    Sure. No problem.

    Even Windows has popen these days. There are some tiny popenr/popenw wrappers in https://github.com/c-blake/cligen/blob/master/cligen/osUt.ni...

    Depending upon how balanced work is on either side of the pipe, you usually can even get parallel speed-up on multicore with almost no work. For example, there is no need to use quote-escaped CSV parsing libraries when you just read from a popen()d translator program producing an easier format: https://github.com/c-blake/nio/blob/main/utils/c2tsv.nim

  • nio

    Low Overhead Numerical/Native IO library & tools (by c-blake)

    Sure. No problem.

    Even Windows has popen these days. There are some tiny popenr/popenw wrappers in https://github.com/c-blake/cligen/blob/master/cligen/osUt.ni...

    Depending upon how balanced work is on either side of the pipe, you usually can even get parallel speed-up on multicore with almost no work. For example, there is no need to use quote-escaped CSV parsing libraries when you just read from a popen()d translator program producing an easier format: https://github.com/c-blake/nio/blob/main/utils/c2tsv.nim

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts