GNU Parallel, where have you been all my life?

1. bu

   1 16 62 9.1 Nim

   B)asic|But-For U)tility Code/Programs (in Nim & Often Unix/POSIX/Linux Context)

   

   This sounds like a job for what standard C calls "popen". You can do `import posix; for line in popen("ls", "r"): echo line` in Nim, though you obviously need to replace `echo line` with other desired processing and learn how to do that.

   You might also want to consider `rp` which is a program generator-compiler-runner along the lines of `awk` but with all the code just Nim snippets interpolated into a program template: https://github.com/c-blake/bu/blob/main/doc/rp.md . E.g.:

       ls -l | rp -pimport\ stats -bvar\ r:RunningStat -wnf\>4 r.push\ 4.f -eecho\ r

2. CodeRabbit

   coderabbit.ai featured

   CodeRabbit: AI Code Reviews for Developers. Revolutionize your code reviews with AI. CodeRabbit offers PR summaries, code walkthroughs, 1-click suggestions, and AST-based analysis. Boost productivity and code quality across all major languages with each PR.

   

3. pash

   2 7 567 3.5 Shell

   PaSh: Light-touch Data-Parallel Shell Processing

   

4. PPSS

   3 2 108 10.0 Shell

   Parallel Processing Shell Script

   

5. fish-shell

   4 347 29,094 9.9 Rust

   The user-friendly command line shell.

   

6. nushell

   5 228 34,271 9.9 Rust

   A new type of shell

   

7. progress

   6 2 12 6.1 C

   A simple-text subtask progress standard and accompanying utilities (by titzer)

   

   I didn't know about this, and reading through the comments, I found out that xargs can also do batching and parallelism (nice!). However, it appears that if you pipe the output of an xargs-parallel command into another utility, it jumbles the output of the multiple subprocesses, whereas GNU parallel does not.

   I was a little put off by the annoying/scary citation issue mentioned by another commenter, so I am not sure I will use parallel.

   I want to pipe the output of parallel processes into a utility that I wrote for progress printing (https://github.com/titzer/progress), but I think that neither of these solutions work; my progress utility will have to do this on its own.

8. script

   7 16 6,458 6.5 Go

   Making it easy to write shell-like scripts in Go

   

   I use Go. You can run scripts with go run directly, and this package makes shell tasks easy: https://github.com/bitfield/script

9. SaaSHub

   www.saashub.com featured

   SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

   

10. jo

    8 16 4,716 2.6 C

    JSON output from a shell

    

    That should recursively list directories, counting only the files within each, and output² jsonl that can be further mangled within the shell². You could just as easily populate an associative array for further work, or $whatever. Unlike bash, zsh has reasonable behaviour around quoting and whitespace too.

    ¹ https://zsh.sourceforge.io/Doc/Release/User-Contributions.ht...

    ² https://github.com/jpmens/jo

    ³ https://github.com/stedolan/jq

11. jq

    9 306 25,063 0.0 C

    Discontinued Command-line JSON processor [Moved to: https://github.com/jqlang/jq] (by stedolan)

    

    That should recursively list directories, counting only the files within each, and output² jsonl that can be further mangled within the shell². You could just as easily populate an associative array for further work, or $whatever. Unlike bash, zsh has reasonable behaviour around quoting and whitespace too.

    ¹ https://zsh.sourceforge.io/Doc/Release/User-Contributions.ht...

    ² https://github.com/jpmens/jo

    ³ https://github.com/stedolan/jq

12. wrk2

    10 14 4,321 0.0 C

    A constant throughput, correct latency recording variant of wrk

    

    > This runs a benchmark for 30 seconds, using 2 threads, keeping 100 HTTP connections open, and a constant throughput of 2000 requests per second (total, across all connections combined).

    Some distros include `ab`[2] which is also good, but wrk2 improves on it (and on wrk version 1) in multiple ways, so that's what I use myself.

    [1] https://github.com/giltene/wrk2

    [2] https://httpd.apache.org/docs/2.2/programs/ab.html

13. xe

    11 2 182 3.0 C

    simple xargs and apply replacement

    

    I also recommend checking out `xe`: https://github.com/leahneukirchen/xe

    It’s like xargs with sane defaults and a couple tricks of its own.

14. task-spooler

    12 4 310 4.4 C

    A scheduler for GPU/CPU tasks

    

    Wow this tool is fantastic, thank you! The UI is very nice and simple. How has this not existed in Unix for 30+ years?

    https://github.com/justanhduc/task-spooler

15. shell

    13 2 157 4.4 Nim

    A Nim mini DSL to execute shell commands (by Vindaar)

    

    On Unix, you might try Nim (https://nim-lang.org) with https://github.com/Vindaar/shell and there are a slew of pipeline-y/Unix-y utilities over at https://github.com/c-blake/bu

    Nim is statically typed and (generally) native-compiled, but it has very low ceremony ergonomics and a powerful compile-time macro/template system as well as user-defined operators (e.g., you can use `+-` to make a constructor for uncertain values so that `9 +- 2` builds a typed object.

16. Nim

    14 356 16,882 9.9 Nim

    Nim is a statically typed compiled systems programming language. It combines successful concepts from mature languages like Python, Ada and Modula. Its design focuses on efficiency, expressiveness, and elegance (in that order of priority).

    

    On Unix, you might try Nim (https://nim-lang.org) with https://github.com/Vindaar/shell and there are a slew of pipeline-y/Unix-y utilities over at https://github.com/c-blake/bu

    Nim is statically typed and (generally) native-compiled, but it has very low ceremony ergonomics and a powerful compile-time macro/template system as well as user-defined operators (e.g., you can use `+-` to make a constructor for uncertain values so that `9 +- 2` builds a typed object.

17. blender_gnu_parallel_render

    15 2 2 0.0 Shell

    Use GNU Parallel to render blender movies distributed by a bunch of nodes

    

    I use GNU Parallel to render Blender videos distributed by a bunch of nodes https://github.com/tfmoraes/blender_gnu_parallel_render

18. cligen

    16 32 522 8.5 Nim

    Nim library to infer/generate command-line-interfaces / option / argument parsing; Docs at

    

    Sure. No problem.

    Even Windows has popen these days. There are some tiny popenr/popenw wrappers in https://github.com/c-blake/cligen/blob/master/cligen/osUt.ni...

    Depending upon how balanced work is on either side of the pipe, you usually can even get parallel speed-up on multicore with almost no work. For example, there is no need to use quote-escaped CSV parsing libraries when you just read from a popen()d translator program producing an easier format: https://github.com/c-blake/nio/blob/main/utils/c2tsv.nim

19. nio

    17 7 35 6.9 Nim

    Low Overhead Numerical/Native IO library & tools (by c-blake)

    

    Sure. No problem.

    Even Windows has popen these days. There are some tiny popenr/popenw wrappers in https://github.com/c-blake/cligen/blob/master/cligen/osUt.ni...

    Depending upon how balanced work is on either side of the pipe, you usually can even get parallel speed-up on multicore with almost no work. For example, there is no need to use quote-escaped CSV parsing libraries when you just read from a popen()d translator program producing an easier format: https://github.com/c-blake/nio/blob/main/utils/c2tsv.nim

20. SaaSHub

    www.saashub.com featured

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    

Suggest a related project