GNU Parallel, where have you been all my life?

• bu

  16 51 9.2 Nim

  B)asic|But-For U)tility Code/Programs (in Nim & Often Unix/POSIX/Linux Context)

This sounds like a job for what standard C calls "popen". You can do `import posix; for line in popen("ls", "r"): echo line` in Nim, though you obviously need to replace `echo line` with other desired processing and learn how to do that.

You might also want to consider `rp` which is a program generator-compiler-runner along the lines of `awk` but with all the code just Nim snippets interpolated into a program template: https://github.com/c-blake/bu/blob/main/doc/rp.md . E.g.:

    ls -l | rp -pimport\ stats -bvar\ r:RunningStat -wnf\>4 r.push\ 4.f -eecho\ r

• pash

  7 529 4.7 Shell

  PaSh: Light-touch Data-Parallel Shell Processing

  

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• PPSS

  2 104 10.0 Shell

  Parallel Processing Shell Script

• fish-shell

  320 24,551 9.9 Rust

  The user-friendly command line shell.

  

• nushell

  212 29,963 9.9 Rust

  A new type of shell

  

• progress

  2 10 6.1 C

  A simple-text subtask progress standard and accompanying utilities (by titzer)

I didn't know about this, and reading through the comments, I found out that xargs can also do batching and parallelism (nice!). However, it appears that if you pipe the output of an xargs-parallel command into another utility, it jumbles the output of the multiple subprocesses, whereas GNU parallel does not.

I was a little put off by the annoying/scary citation issue mentioned by another commenter, so I am not sure I will use parallel.

I want to pipe the output of parallel processes into a utility that I wrote for progress printing (https://github.com/titzer/progress), but I think that neither of these solutions work; my progress utility will have to do this on its own.

• script

  11 5,070 5.2 Go

  Making it easy to write shell-like scripts in Go

I use Go. You can run scripts with go run directly, and this package makes shell tasks easy: https://github.com/bitfield/script

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• jo

  15 4,582 4.3 C

  JSON output from a shell

That should recursively list directories, counting only the files within each, and output² jsonl that can be further mangled within the shell². You could just as easily populate an associative array for further work, or $whatever. Unlike bash, zsh has reasonable behaviour around quoting and whitespace too.

¹ https://zsh.sourceforge.io/Doc/Release/User-Contributions.ht...

² https://github.com/jpmens/jo

³ https://github.com/stedolan/jq

• jq

  306 25,063 0.0 C

  Discontinued Command-line JSON processor [Moved to: https://github.com/jqlang/jq] (by stedolan)

  

That should recursively list directories, counting only the files within each, and output² jsonl that can be further mangled within the shell². You could just as easily populate an associative array for further work, or $whatever. Unlike bash, zsh has reasonable behaviour around quoting and whitespace too.

¹ https://zsh.sourceforge.io/Doc/Release/User-Contributions.ht...

² https://github.com/jpmens/jo

³ https://github.com/stedolan/jq

• wrk2

  13 4,155 0.0 C

  A constant throughput, correct latency recording variant of wrk

> This runs a benchmark for 30 seconds, using 2 threads, keeping 100 HTTP connections open, and a constant throughput of 2000 requests per second (total, across all connections combined).

Some distros include `ab`[2] which is also good, but wrk2 improves on it (and on wrk version 1) in multiple ways, so that's what I use myself.

[1] https://github.com/giltene/wrk2

[2] https://httpd.apache.org/docs/2.2/programs/ab.html

• xe

  2 170 4.1 C

  simple xargs and apply replacement

I also recommend checking out `xe`: https://github.com/leahneukirchen/xe

It’s like xargs with sane defaults and a couple tricks of its own.

• task-spooler

  4 226 4.4 C

  A scheduler for GPU/CPU tasks

Wow this tool is fantastic, thank you! The UI is very nice and simple. How has this not existed in Unix for 30+ years?

https://github.com/justanhduc/task-spooler

• shell

  2 151 4.4 Nim

  A Nim mini DSL to execute shell commands (by Vindaar)

On Unix, you might try Nim (https://nim-lang.org) with https://github.com/Vindaar/shell and there are a slew of pipeline-y/Unix-y utilities over at https://github.com/c-blake/bu

Nim is statically typed and (generally) native-compiled, but it has very low ceremony ergonomics and a powerful compile-time macro/template system as well as user-defined operators (e.g., you can use `+-` to make a constructor for uncertain values so that `9 +- 2` builds a typed object.

• Nim

  347 16,060 9.9 Nim

  Nim is a statically typed compiled systems programming language. It combines successful concepts from mature languages like Python, Ada and Modula. Its design focuses on efficiency, expressiveness, and elegance (in that order of priority).

  

On Unix, you might try Nim (https://nim-lang.org) with https://github.com/Vindaar/shell and there are a slew of pipeline-y/Unix-y utilities over at https://github.com/c-blake/bu

Nim is statically typed and (generally) native-compiled, but it has very low ceremony ergonomics and a powerful compile-time macro/template system as well as user-defined operators (e.g., you can use `+-` to make a constructor for uncertain values so that `9 +- 2` builds a typed object.

• blender_gnu_parallel_render

  2 2 0.0 Shell

  Use GNU Parallel to render blender movies distributed by a bunch of nodes

I use GNU Parallel to render Blender videos distributed by a bunch of nodes https://github.com/tfmoraes/blender_gnu_parallel_render

• cligen

  32 489 8.4 Nim

  Nim library to infer/generate command-line-interfaces / option / argument parsing; Docs at

Sure. No problem.

Even Windows has popen these days. There are some tiny popenr/popenw wrappers in https://github.com/c-blake/cligen/blob/master/cligen/osUt.ni...

Depending upon how balanced work is on either side of the pipe, you usually can even get parallel speed-up on multicore with almost no work. For example, there is no need to use quote-escaped CSV parsing libraries when you just read from a popen()d translator program producing an easier format: https://github.com/c-blake/nio/blob/main/utils/c2tsv.nim

• nio

  7 32 6.3 Nim

  Low Overhead Numerical/Native IO library & tools (by c-blake)

Sure. No problem.

Even Windows has popen these days. There are some tiny popenr/popenw wrappers in https://github.com/c-blake/cligen/blob/master/cligen/osUt.ni...

Depending upon how balanced work is on either side of the pipe, you usually can even get parallel speed-up on multicore with almost no work. For example, there is no need to use quote-escaped CSV parsing libraries when you just read from a popen()d translator program producing an easier format: https://github.com/c-blake/nio/blob/main/utils/c2tsv.nim

• SaaSHub

  www.saashub.com sponsored

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

Suggest a related project