nushell VS oil

I've been using nushell as my daily driver shell for a while. It's still very much in development, but it's got some nice features, like Windows cross compatibility, proper data types, and syntax highlighting

Now, I guess the deeper question is ‶is the text-based, line-delimited, pipe-oriented, interaction model between programs that don't know each other still relevant″; and we have seen new, more structured, approaches being tested, for instance with PowerShell or NuShell.

You can install PowerShell on Linux now, if you like - or there's nushell: https://www.nushell.sh/

I'll add the long awaited support for nushell in broot.

But that and xplr are both programs used within the shell and don't address that the shell is still outputting plain-dumb-dataless-text. https://www.nushell.sh/ promotes right on the homepage Everything is data so it's exciting to see and hope for a better future!

I don't understand what you mean by Graphics integration or graphical frontend. Do you have an example? Instead of du - I recommend dust. The data features - are basically the features of nushell, are you using it? Regarding the variables - excellent idea, I will implement this. What do you mean by menu creation? What syntax would you use for "skip next line"?

> Different components can integrate and coordinate through standards.

We've tried that and for the most part, it doesn't actually work in practice. Hell, it doesn't even work between standard Unix programs. They don't "integrate and coordinate" at all. They just push unstructured text around, and in some cases that's sufficient to perform some very basic compound tasks.

There is an upcoming project though that tries to make Unix pipelines work the way they are supposed to: Nushell[1]. It has real composability with strongly typed data that flows between commands.

Guess how they're doing it? Yup, that's right: They're building it all as a monolithic system where the entire pipeline is supervised and interpreted by the shell. Unix is too fragmented and standards are too difficult to develop and implement for any other approach to be realistically feasible. "Small programs" are an evolutionary dead end.

[1] https://www.nushell.sh/

Finally, I really do recommend looking into Nushell. It should give you a much better idea of how a shell could work if it weren't tied to ASCII blobs as its core data type.

If you're discontent with the state of shell scripting, you should give nushell a try: nushell.sh

If structured data is what excites you about pwsh, you should check out nushell: https://www.nushell.sh/. It is cross-platform like pwsh, and takes a similar approach to structured data, but has a much simpler syntax that I find more readable and much nicer to type than pwsh.

Nice! I also use inotifywait for hot reload when writing https://www.oilshell.org/

I never heard of the xdotool method ... although it sounds like it has some drawbacks?

I agree with 90%-95% of the article. I would phrase it more pithily as:

1. Use recursive descent for recognizing existing languages, where you have a good test corpus. (And very good point about testing INVALID inputs too!)

2. Use grammars to design new languages

I follow this with https://www.oilshell.org/ -- the compatible OSH is a huge recursive descent parser, and Oil is designed with a grammar.

Guy Steele and Russ Cox also mentioned this methodology -- design the language with a grammar, and then implement it a second time by hand. Then you REALLY know it! :)

It is definitely possible to not know the language you're implementing / creating!

---

As an undergrad, I never tried to design a language, or even thought about it honestly. But I agree that undergrads should know of both approaches, without necessarily getting into the details of CFGs.

It would be a mistake to just teach recursive descent.

https://news.ycombinator.com/item?id=34291191 )

I would just use shell scripts to saturate 64-128 cores, rather than dealing with slow schedulers and distributed file systems.

But garbage collectors and memory management are a main reason you can't use all of a machine from one process. There's just so much room for contention. Also the hardware was trending toward NUMA at the time, and probably is even more now, so processes make even more sense.

All of that is to say that I'm a little scared of multi-threaded GC ... especially when linking in lots of third party libraries.

---

But of course there are many domains where you don't have embarrassingly parallel problems, and writing tight single- or multi-threaded code is the best solution!

Some more color here: https://old.reddit.com/r/oilshell/comments/109t7os/pictures_...

I wonder if Clasp has any support for multi-process programming? Beyond Unix pipes, you could also use shared memory and maybe some semaphores to synchronize access, and avoid copying. I think of that as sort of "inverting" the problem. Certain kinds of data like pointer-rich data is probably annoying to deal with in shared memory, but there are lots of representations for data and I imagine Lisps could take advantage of some of them, e.g. https://github.com/oilshell/oil/wiki/Compact-AST-Representat...

Hm yes now I remember that point about how the data is anonymous Python objects that you can pass around to functions.

Are there any open source examples? I looked around the github account, but I mostly remember this tool

https://github.com/stb-tester/apt2ostree

I'd be interested in seeing the Python config and Ninja output, to see how it works. Right now it looks to me like the dependencies are more implicit than explicit, e.g. with your copen example

---

The system I ended up with is more like Bazel, but it's not building containers, so it's a slightly different problem. But I'm interested in building containers incrementally without 'docker build'.

I like the apt lockfile idea definitely ... However I also have a bunch of other blobs and tarballs, that I might not want to check into git. I guess you just put those in OSTree?

Our config looks like this

https://github.com/oilshell/oil/blob/master/core/NINJA_subgr...

And all the code is in build/ninja* of the same repo

I watched this a few weeks ago, great talk and exciting tool!

Can you expand on this answer about whether the generated code is run in production or not?

https://youtu.be/5YjsSDDWFDY?t=2200

I would expect that generating USB drivers is just as demanding from a performance perspective as cloud services? You have generally have better and more control over hardware in the latter case?

Do you know if the P-generated code is still used for USB drivers, or if they moved away from that approach and just use the modeling process?

The state machines seem pretty low level, so I don't understand in principle why efficient code couldn't be generated from them?

FWIW I'm using 3 or 4 C/C++ code generators for https://www.oilshell.org, and I've the layer of indirection useful in many ways (although it also can make it harder for people to understand)

Oil Shell | C++ / Python Language Engineer | Remote (e.g. Canada, Europe), Flexible, Open Source | https://oilshell.org

I have a slightly unusual job funded by a grant under https://nlnet.nl ! Everything is flexible -- think of it as an open source project where you happen to get paid.

We started in June, and there are currently 2 part-time contributors. We can use more help!

The job is to work on a C++/Python shell interpreter with a garbage-collected runtime. Depending on your interest, you can work on:

(1) Data-structure heavy, performance-sensitive C++ code -- e.g. mark bitmaps for the GC, small value optimization, lazy sweeping, hash tables, and more. We have an objective goal that enables creativity: make the shell as fast as bash. Currently it's 1.6x slower on a long batch workload.

This runtime will let us add real data structures and JSON to a shell! ( http://www.oilshell.org/blog/2021/01/why-a-new-shell.html )

(2) Semi-automatically translating code from typed Python to C++. This can be simple (adding MyPy types) or involved (compiling Python's yield to state machines)

(3) Benchmarks, builds, testing -- I like automating everything, and shell works great here :)

We've made great progress lately -- here's a screencast of our custom ~6000 line garbage-collected runtime working on real code (note GC debug output):

https://asciinema.org/a/547312?t=30 (this worked as of 11 days ago, detailed blog post upcoming)

Some background: A Garbage-Collected Heap in C++, Shaped Like Typed Python

https://www.oilshell.org/blog/2022/05/gc-heap.html (and more on the blog)

Everything about the project is open, so please take a look and contact me at [email protected] if interested. We can chat online or do a video call (I'm based in the USA). We can also arrange a code walkthrough or PAID "trial period". We want people who will have fun working on these problems.

Compensation: First grant was for 50K euros, and a potential second grant would be for 50K, to be split up among contributors.

https://github.com/oilshell/oil/wiki/Where-Contributors-Have... -- We now have a fast and flexible Ninja build, as well as a comprehensive CI that keeps the master branch green

Contact: [email protected]

Thanks! There's definitely room for people with a wide variety of skills, e.g. just shell or just Python. Feel free to chime in on Github, Zulip, reddit, etc.! (all links on the home page https://www.oilshell.org)

FWIW here is a list of other such projects: https://github.com/oilshell/oil/wiki/Alternative-Regex-Synta...

The Oil language is a very interesting project whose author clearly recognizes both the benefits and the weaknesses of POSIX shells. This post describes major improvements that can only be made by breaking POSIX compliance (again, like Bash, via modes/options); but more relevant to my criticism of existing POSIX-compliant shells is this in-depth post about arrays, which argues that the core data type for shells should have been array-of-strings rather than string. I completely agree with this premise, and I think that the need for tools like find to include an option (-print0) for splitting output using null characters rather than newlines just shows how inadequate ASCII is as a global datatype for a glue language like the shell.

FWIW I recently tried this on https://www.oilshell.org, thinking the code would be faster due to better cache locality. There are many pointer rich data structures and they should be packed much more tightly when pointers are 32 bit.

It definitely saved memory, but it was slightly slower or the same speed. My guess is that 32-bit performance isn't prioritized on x86-64 by either compiler writers or chip designers, i.e. they optimize the common case of 64 bit pointers.

Definitely interested in other comments / opinions / data on this.

I think if it actually sped programs up, you would see many more distros compiling packages for 32-bit. It does save memory, but somehow that's not big enough motivation. Probably because desktops/laptops have a lot of memory, and Linux systems with constrained memory are probably already 32-bit.

And because real memory hogs like browsers probably want to address more than 4GB. Though to be honest I think a multi-process browser with a 4GB-per-origin limit sounds like a very good idea!