Our great sponsors
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
KeenWrite
Discontinued Free, open-source, cross-platform desktop Markdown text editor with live preview, string interpolation, and math.
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
-
-
oil
Oils is our upgrade path from bash to a better language and runtime. It's also for Python and JavaScript users who avoid shell!
-
-
-
murex
A smarter shell and scripting environment with advanced features designed for usability, safety and productivity (eg smarter DevOps tooling)
-
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
> the mystery behind function overloading
In NGS the common case for multiple dispatch would be to define your own type and then add methods to existing multimethod to handle that type; additionally I've simplified the dispatch algorithm exactly for this reason - unclear answer to "what's going to be called actually?".
> function overloading is supported in the same way that its also supported in Bash too where you might have a function but if there is a alias of the same name that will take priority.
I don't think it is called overloading. That's very different from having two methods with the same name and the "right one" is called based on types of passed arguments.
> interpreting an indexed argument into an named arguments.
In NGS it happens in exactly one place, when main() is called. At that point command line arguments are automatically parsed based on main() parameters and passed to main().
I don't think I understand your reasoning behind not including named parameters.
> NGS likes to expose its smart features whereas in murex they're abstracted away a little
Sounds about right. Power to the people!
> Having different behaviours for different executables hard coded into the shell is one behaviour I purposely avoided.
I can see why. NGS prefers to do the right thing in most cases and have shorter and cleaner code. Yes adding some risk, which I see as not that big - exception when an exit code was "ok" for unknown program (unknown external programs default to exception on non-zero exit code).
I also dream about moving the hard coded information about programs into separate schema, which could be re-used between different shells. That would be similar to externally provided typescript definitions for existing code, described at https://www.typescriptlang.org/docs/handbook/declaration-fil...
> differ between Linux, BSD, Windows,
"detect program variant" feature in the schema I mentioned above
> hard coding executable behaviour
Yep, smells a bit
> causes more potential issues than it solves.
I think in particular situation with exit codes the risk is low.
> users expect the shell to understand all external executables but there are some you haven't anticipated thus breaking expectation / assumptions
Yes. Downside. That's the kind of surprises that suck. Need to make sure at least the docs are clearly warning about this.
> So instead I've relied on having foundational logic that is consistent.
Totally understandable.
> STDERR contains a failed message (eg "false", "failed", etc) or STDERR is > STDOUT[6]. This has covered every use case I've come across.
Sounds also like potentially surprising, somewhat similar to handling exit codes. Yes, it's not per program and that's why it's better but still.
> Additionally STDERR is highlighted red by default
As it should! Yes!
> mini-stack trace
Yes, please!
> testing framework baked into the shell language.
Same in NGS but https://github.com/ngs-lang/ngs/issues/494 :)
> So its fair to say I've spent a significant amount of time designing smarter ways of handling failures than your average shell scripting language. As I'm sure you have too.
I guess. Differently of course :)
> Personally I just describe it as "error handling" because in my view this is how peoples expectations are rather than the reality of handling forked executables.
Sounds like I was not clear enough as if something is not uniform in this regard in NGS. Exceptions are not only for external programs. It just happens that external program can be called inside "if" condition; an exception might occur there.
I write a LOT of bash/shell scripts. And I don't like it, it's just part of what I have to do.
Learning a handful of bash idioms and best-practices has made a massive impact for me, and life much easier. The shell is something you cannot avoid if you're a programmer or other sort of code-wrangler.
You can interact with it + be (mostly) clueless and still get things done, but it's a huge return-on-investment to set up "shellcheck" and lookup "bash'isms", etc.
----
(Off-topic: I am convinced Ruby cannot be beaten for shell-scripting purposes. If I had a wish, it would be that every machine had a tiny Ruby interpreter on it so I could just use Ruby. I'm not even "a Ruby guy", it's just unreasonably good/easy for this sort of thing. And I keep my mind open for better alternatives constantly.)
Example of near-identical script in bash vs Ruby:
https://github.com/GavinRay97/hasura-ci-cd-action/blob/maste...
https://github.com/GavinRay97/hasura-ci-cd-action/blob/maste...
Oh yeah, bash functions are great and absolutely abusable. Sometimes you need some grand hacks to get it to work well, but when it works well, it can do some magic. You can even export functions over ssh!
I wrote this a few years back which ran on bunches of hosts and fed into a infrastructure network mapper based on each hosts' open network sockets to other known hosts. It wasn't really feasible to install a set of tools on random hosts.. but I still had root ssh access across the board. So I needed something tool agnostic, short, auditable, and effectively guaranteed to work:
https://github.com/red-bin/lsofer/blob/master/lsofer.sh
Feel free to replace "time" with "eval" too if you don't want your command timed.
This is a really useful pattern because it means you can create a "run" script with a bunch of sub-commands (private or public), render help menus and create project specific scripts without any boilerplate.
I have an example of that here: https://github.com/nickjj/docker-flask-example/blob/main/run
I've written about this pattern in more detail here https://nickjanetakis.com/blog/replacing-make-with-a-shell-s.... It's basically a less limited Makefile for when you want to make project specific aliases.
If "-V" isn't supplied, then every invocation of $log simply returns without printing a message. The upshot is a reduction in conditional statements. Function pointers FTW. I wrote about this technique in my Typesetting Markdown series:
https://dave.autonoma.ca/blog/2019/05/22/typesetting-markdow...
Here's a rudimentary implementation and example usage to wet your whistle:
https://github.com/DaveJarvis/keenwrite/blob/master/scripts/...
https://github.com/DaveJarvis/keenwrite/blob/master/installe...
While I understand the sentiment, I'm not sure how bash could ever be as maintainable as a something written in e.g. Python (or even better, a strongly-typed language).
The thing with bash is, it's great for tying things together and quick bits and pieces, but it's not set up for writing maintainable code. Arrays, functions, even if statements comparisons can all be done in bash (as first-class features), but are just... easier in other languages. And then think about the refactoring, linting, testing tools available in bash vs other languages. And then on top of that, there's the issue of handling non-zero return codes from programs you call; do you `set -e`, and exit on any non-zero return code even if you wanted to continue, or not `set -e`, ignoring any errors as your script just continues.
Personally, when I feel I want to use a function (or array, or other similar, non-trivial thing), in bash, it's time to reach for another language.
Having said that, there are some nice programs written in bash. https://www.passwordstore.org/ being one that comes to mind.
I do the same thing, but slightly differently. https://github.com/francislavoie/laravel-websockets-example/...
At one time, I did learn myself to write shell scripts. I even wrote this 3Kl line monstrosity [0]
However, I would strongly advice to master a proper programming language. I respect the article and the efforts of the author, but I feel that it is the past.
I mastered Python a bit and the ability to just use things like dictionaries, proper parsing libraries and such, instead of kilometers of fragile pipes, it is so much better.
I understand something like Python may feel total overkill, but that 10 line shell script suddenly needs quite a bit of error handling and some other features and before you know it, you wish you started out with python or something similar.
[0]: https://github.com/louwrentius/ppss
I enjoyed how my bash 2048 came out: https://github.com/dlthomas/bash2048/blob/master/2048.sh
I've seen some pretty interesting things like the use of Rust for front end development, like yew[1] and Seed[2].
There aren't many languages that are practical for WASM output. Scripting and managed languages need to ship their interpreters and runtimes with their WASM blobs, and can end up relatively large. JavaScript's interpreter and runtime are baked into every browser already.
That leaves only compiled and unmanaged languages for potentially good WASM targets. As mentioned before, Rust is seeing a lot development in that space. If LLVM can compile it, then Emscripten can output it to WASM.
[1] https://github.com/yewstack/yew
[2] https://github.com/seed-rs/seed
I've seen some pretty interesting things like the use of Rust for front end development, like yew[1] and Seed[2].
There aren't many languages that are practical for WASM output. Scripting and managed languages need to ship their interpreters and runtimes with their WASM blobs, and can end up relatively large. JavaScript's interpreter and runtime are baked into every browser already.
That leaves only compiled and unmanaged languages for potentially good WASM targets. As mentioned before, Rust is seeing a lot development in that space. If LLVM can compile it, then Emscripten can output it to WASM.
[1] https://github.com/yewstack/yew
[2] https://github.com/seed-rs/seed
I agree with your broader point that we seem stuck with old shells and no contenders seems in a position to replace them. However some are certainly trying, like osh [1] that takes backwards compatibility with bash seriously, which makes adoption easier.
Maybe it'll be like the C++ vs Rust situation?
https://www.oilshell.org/
I'm quite happy to see that something Bash-related is on Hacker News! Unfortunately it seems that I don't really agree with much the author...
While I do agree that it would be nice to be able to have 'local' functions and have inter-function cleanup work better, the logical conclusion for me was not to use function subshells. Since the use case is for larger programs (where different functions may want to have their own cleanup mechanisms), I'm opting to go for more of a library route. For example, I'm working on a Bash library that includes a function to allow different sources to add (and remove) functions to the same `TRAP`. A similar function may be useful, possibly involving the `RETURN` trap and the `-T` flag. Obviously, using a package manager for _Bash_ of all languages brings in a lot of overhead, but I think it can be quite powerful, especially with a potential "Bundle" feature that makes scripts work without the package manager.
Concerning specifically the use of subshells, (as other commenters have pointed out) it significantly reduces performance. I also disagree that dynamic scoping is necessarily bad for Bash. I find it quite useful when I need to use various common functions to manipulate a variable - since modifying and 'returning' variables from a function is usually either slow or verbose with Bash. Admittedly though, this feature is quite annoying at times - for example, most public functions in my Bash package manager[2] all have their variables prefixed with two underscores - because they `source` all the shell scripts of all package dependencies - so I want to be extra certain nothing weird happens
[1] https://github.com/hyperupcall/bash-core/blob/a17ab0a8b6070f...
Woah, this is very cool. I may try to adopt this.
I recently discovered, similar to the author of the post for this thread, that local variables are dynamically scoped.
I have been writing a lot more shell scripts lately, using a "library" [1] of sorts I've been writing. When I was debugging one of my scripts that uses mycmd, I discovered that I had failed to declare some of my variables local and they were leaking out to the global scope.
I had recently added functionality to call a set of functions on script exit, so I added something that would output the defined variables, in hopes that I could write something that will output them at the beginning and then the end and show the difference. I was surprised when variables defined in my dispatch function [2] for those at exit functions were showing up, even though they were definitely defined as local. It was then that I dug around and discovered the dynamic scope of variables.
I've been trying to figure out how to accomplish what I desire but exclude those variables from calling functions. I haven't been able to find an obvious way to see if the variable is coming from a calling function. I might be able to use techniques like you've pointed out in your linked post to add the tracing that I want. Still need to think more on this.
---
[1] https://github.com/travisbhartwell/mycmd
> Is there a reason we aren’t using a shell with a proper programming language for scripting?
Mostly because the people who want to introduce a "programming language" into the shell don't prioritize being a shell.
Check out the "Oh" shell for contrast. This is what a programming language looks like when you force it to conform to being a shell first priority.
https://github.com/michaelmacinnis/oh
https://www.youtube.com/watch?v=v1m-WEZz46U
This is "Scheme-like" but has FEXPRs so things can be redefined and evaluation can be controlled.
murex author here. Hopefully I can answer some questions:
> I do scan alternative shells from time to time. I could have missed but I didn't see multiple dispatch in any of them. Which ones have it? Just to clarify: In which shell you can define several methods with the same name and when called, the method to invoke is selected based on the types of the arguments?
I've not heard of the term "multiple dispatch" but reading the thread it sounds like you're describing function overloading. Powershell does support this with classes[0].
murex does it's overloading at the API level[1]. The reason behind that decision is to keep the methods simple (eg a pipeline might contain JSON or CSV data but you as a shell user don't want to run into situations where you've written a function that supports one data type but not another. So murex automatically abstracts that part away for you. In addition to the point laumars mentioned about consistent `jq`-like methods that are data type agnostic, it allows for easy iteration (eg `open somedata -> foreach { do stuff }` -- where you don't need to think about data types, murex does the heavy lifting for you).
There are some specific builtins that support "overloading" of sorts but instead of "overloading" they have handlers[2][3]
It's also worth adding that function overloading is supported in the same way that its also supported in Bash too where you might have a function but if there is a alias of the same name that will take priority. There are also private[4] functions (which are namespaced) so you have additional controls to avoid accidental overloading when writing modules too.
> I did not even see how to define named parameters - https://murex.rocks/docs/commands/function.html
Named parameters are optional because neither Windows nor POSIX have any understanding of named arguments in their processes. I did consider abstracting that into murex's functions regardless like Python et al would but I couldn't design a way that wasn't jarring nor cumbersome to write in a hurry nor worked transparently with Windows and POSIX ARGS[]. So I've come up with an optional builtin called `args`[5] which allows you to define flags, which are effectively the same thing as named parameters except they're supported natively by Windows and POSIX ARGS[], so you can write a murex script as a native shell script without leaving the user to write another abstraction layer themselves interpreting an indexed argument into an named arguments. You can write really quick Bash-like functions or more verbose scripting style functions too.
> Examples of things in NGS that I have not seen anywhere else: syntax for run-command-and-parse-output, proper handling of exit codes.
Both of these are baked into murex as well. The run-command-and-parse-output is the API stuff mentioned above. It's where the overloading happens.
As for error handling: any command is considered a failure if there is either a non-zero exit code, STDERR contains a failed message (eg "false", "failed", etc) or STDERR is > STDOUT[6]. This has covered every use case I've come across.
Additionally STDERR is highlighted red by default and when a process fails you're given a mini-stack trace showing where in the script the error happened. You have try/catch blocks, saner if syntax etc that all use the same API for detecting if a process has failed too.
The other thing murex does is have a testing framework baked into the shell. The docs for test[7] need expanding though. But in essence:
- you can write proper unit tests
- you can intercept STDOUT (even if it's mid pipeline) and test that output. This works around instances where (1) you can't have full on unit tests due to side effects in a function that can't be easily mocked (2) you want to add your own debugging routines (rather than just printing values to STDOUT)
- you can add state watches (except in murex, the watches are Turing complete so you're not just adding noise to your terminal output but can put meaningful debug messages).
And all of the debugging stuff can be written straight into your normal shell routines and cause no additional execution overhead unless you've purposely enabled test mode.
So yeah, I've spent a significant amount of time designing smarter ways of handling failures than your average shell scripting language.
> Regarding exit codes. Typical approach to exit codes varies. Python - "I don't care, the programmer should handle it". Some Python libraries and other places - "Non-zero is an error". That's simplistic and does not reflect the reality in which some utilities return 1 for "false". bash (and probably other shells too) is unable to handle in a straightforward manner situation where external command can return exit codes for "true", "false" and "error". It just doesn't fit in the "if" with two branches. NGS does handle it with "if" with two branches + possible exception thrown for "error" exit code.
If you're describing that as a "two branch" approach then technically murex could be argued as having "three branches" because it checks exit code, STDERR contents, and payload size too. Personally I just describe it as "error handling" because in my view this is how peoples expectations are rather than the reality of handling forked executables.
I guess where NGS and murex really differ is NGS likes to expose its smart features whereas in murex they're abstracted away a little (they can still be altered, customised, etc) to keep the daily mundane shell usage as KISS (keep it simple stupid) as possible. eg you can overload function calls if you really wanted but that can often cause unforeseen complications or other unexpected annoyances right when you least want it to. So murex keeps that stuff around for when you need it but finds ways to avoid people needing to rely on it.
> NGS knows that some external programs have exit code 1 which does not signify an error.
Having different behaviours for different executables hard coded into the shell is one behaviour I purposely avoided. I do completely understand why incentive behind wanting to do this and wouldn't criticise others for doing that but given external programs can change without the shell being aware, they can be overloaded with aliases, functions, etc, and they might even just differ between Linux, BSD, Windows, etc -- well it just seemed like it causes more issues than it solves. You also then run into problems where users expect the shell to understand all external executables but there are some you haven't anticipated. Ultimately it creates a kind of special magic that is unpredictable and outside the control of the shell itself. So instead I've relied on having foundational logic that is consistent. It's the one side of shell programming where I've placed the responsibility onto the developer to get right rather than automagically doing what I think they are expecting.
Links:
[0] https://stackoverflow.com/a/57294942
[1] https://murex.rocks/docs/GUIDE.apis.html
[2] https://murex.rocks/docs/commands/openagent.html && https://murex.rocks/docs/commands/open.html
[3] https://murex.rocks/docs/commands/event.html (I need to write a lot more documentation around event driving programming in murex)
[4] https://murex.rocks/docs/commands/private.html
[5] https://murex.rocks/docs/commands/args.html
[6] https://github.com/lmorg/murex/blob/19d2aeb27e814a4c6fec77d2...
[7] https://murex.rocks/docs/commands/test.html
I would rather use TypeScript and the oclif framework (what heroku-cli uses under the hood) to build any cli-tools I might need...
The other day I just made a sample stripe-jobs cli app with it and playwright in case you want to see how that looks
https://github.com/agustif/stripe-jobs-cli
> there isn't really an entry point in murex scripts
I have a nice trick in NGS for that. Under the idea that "small scripts should not suffer", script is running top to bottom without "entry point". However, if the script has defined main() function, it is invoked (with command line arguments passed).
Example - https://github.com/ngs-lang/nsd/blob/afe0cad5e506ec4ee2fa924...
> `args` still contains more boilerplate code than I'm happy with
Is there anything preventing you to have exactly the same functionality but with syntactic sugar that it looks like parameters declaration? (Just to be clear, keeping all the ARGV machinery).
Something like (assuming local variables are supported; if not, it could still be $args[Flags] etc):
function hippo(name:str, hungry:bool) {
Related posts
- ThePrimeagen builds a complete Neovim config from scratch
- numToStr/dotfiles: 🏡 /.dotfiles | Includes configs for neovim, tmux, zsh, alacrity, kitty, and more | Managed by GNU stow
- How to Make ZSH Start as Fast as Bash?
- Workaround for multiple Monitors with different refresh rates
- Slow zsh startup time