scripts-to-rule-them-all VS doit

Personally I like https://github.blog/2015-06-30-scripts-to-rule-them-all/ as a pattern and then let the authors do whatever crazy thing they want from there. In my experience, 99% of repos never move past using simple shell scripts with a few common functions with that pattern, and things are kept fairly simple. A select few repositories tend to mature enough that they are able to invest in swapping towards something more testable than shell scripts, and then you just have a couple people who stick to invoking `make` from the scripts but it's fine and nobody has to think about it except them. We don't stick to that exact set of scripts, but find that as long as you don't use more than like 10ish entrypoints in `script/*`, and have at least `script/bootstrap` it's fine.

I dig the general idea, but question the value add over a directory of `scripts` that follow sane conventions (ie `script/test`, `script/build` etc). Is the main thing that you can do `just -l` to see available commands? I have never really reached for `make` when I've had a choice, as I've done mostly ruby, JS, or java where you have more sane, humane tools (i.e. Rake, Yarn, Maven though that one is never fun).

My general approach is every repo should have something that follows https://github.com/github/scripts-to-rule-them-all, written in sh (maybe bash, its 2023), linted with shellcheck. When you need something fancy Rake is great or grab some nice bash command line helper and source it from all your scripts. Is a command listing really worth another dependency over what you get from `ls script` or `cat script/README` ?

Afaik AzDo cannot run tasks concurrently. From having had to work with azure pipelines I would highly suggest to use the github approach of Scripts to rule them all and avoiding predefined tasks unless absolutely necessary(Things that are complicated to implement and solutions already existing.

GitHub have a pattern for this called "scripts to rule them all" - https://github.com/github/scripts-to-rule-them-all - I've not fully adopted it yet but I probably should, it looks very well thought-out.

People at Github made an attempt to fix this situation: scripts to rule them all. The idea is to have common set of executable scripts for common developer tasks in a script/ directory in the root of every project:

https://github.com/github/scripts-to-rule-them-all

I like mage for Go and doit for Python.

Some competitors - Rake (ruby) - Bake - Earthly - SCons - doit

An alternative to Scons could be Doit (<https://pydoit.org/>), which if I remember correctly was built as a faster alternative to Scons. See also reasons of some users to prefer the later to other mentioned here: <https://pydoit.org/stories.html>.

If you code in Python, your probably should use the language as much as possible and avoid calling shell commands.

E.G:

- manipulate the file system with pathlib

- do hashes with hashlib

- zip with zipfile

- set error code with sys.exit

- use os.environ for env vars

- print to stderr with print(..., file=...)

- sometimes you'll need to install lib. Like, if you want to manipulate a git repo, instead of calling the git command, use gitpython (https://gitpython.readthedocs.io/en/stable/)

But if you don't feel like installing a too many libs, or just really want to call commands because you know them well, then the "sh" lib is going to make things smoother:

https://pypi.org/project/sh/

Also, enjoy the fact Python comes with argparse to parse script arguments (or if you feel like installing stuff, use typer). It sucks to do it in bash .

If what you need is more build oriented, like something to replace "make", then I would instead recommend "doit":

https://pydoit.org/

It's the only task runner that I haven't run away from yet.

Remember to always to everything in a venv. But you can have a giant venv for all the scripts, and just she-bang the venv python executable so that it's transparent. Things don't have to be difficult.

I've been using Doit for a project which involves gathering together documents made up of multiple Markdown files and converting to multiple formats. It's really cool but has some irritations. It didn't end up being much simpler than Make for me. I'm interested in trying some of the alternatives people have posted.

The code for all of this is available here, and described in detail in my article. I'm particularly fan of doit for this type of project, and highly encourage everyone to check it out!

Instead, I use pydoit (which is basically a Python version of make). It's simple, flexible, and quite extensible. So, here's what I do with it: