libxo VS oils

> In FreeBSD, this problem was solved with libxo[0]:

Libxo happens to be in the base system, but it is generally available:

* https://github.com/Juniper/libxo

* https://libxo.readthedocs.io/en/latest/

Or make it flexible:

> libxo - A Library for Generating Text, XML, JSON, and HTML Output

* https://github.com/Juniper/libxo/

* https://wiki.freebsd.org/LibXo

Want structure? Ask for JSON or XML and parse. Otherwise it's the regular text output.

Can you trust it? Cli tool output is not exactly stable. I thought that's why libxo exists?

https://github.com/Juniper/libxo

Please consider https://github.com/Juniper/libxo or something even better than that.

That would look a lot like FreeBSD. Many of the FreeBSD userland tools are set up to use the excellent libxo (https://github.com/Juniper/libxo) to allow the user's choice of how things are output.

I'll keep an eye out for the blog post!

We're still working (intermittently) on a better shell <-> terminal interface. We actually did the first demo back in 2021, before we became aware of Warp. And here is an update from a month ago, which mentions Warp / IntelliJ / VSCode:

https://lobste.rs/s/viwe6p/open_source_can_t_coordinate#c_px...

That is, we did some demos with a nushell contributor, to see if there is a protocol that Oils and nushell can agree on. The big constraints seems to be bash and zsh. (I still believe in graceful evolution, not building a "inner platform")

I think these 2025 demos went pretty well, but it's still in the demo/research phase.

After working in this area for many years, I think the bottleneck is coordination, mentioned here - https://lobste.rs/s/ejcoud/dreaming_smel_shell

i.e. there are many separate projects doing similar things, and most of them get to 80% or so. I like that you mention a lot of prior art!

More links: https://github.com/oils-for-unix/oils/wiki/Headless-Shell-Pr...

Feel free to join any of the Zulip discussions ... there are a bunch of prototypes floating around, but nothing set in stone!

JSON is not a friendly format to the Unix shell — it’s hierarchical, and cannot be reasonably split on any character

Yes, shell is definitely too weak for JSON parsing!

I would also argue that Awk is too weak

The following code assumes that it will be fed valid JSON. It has some basic validation as a function of the parsing and will most likely throw an error if it encounters something strange, but there are no guarantees beyond that.

Yeah I don't like that! If you don't reject invalid input, you're not really parsing

---

OSH and YSH (https://oils.pub) both have JSON built-in, and they have the hierarchical/recursive data structures you need for the common Python/JS-like API:

    osh-0.33$ var d = { date: $(date --iso-8601) }

Wow great bug!

> Bash forgot to reset errno before the call. For about 30 years, no one noticed

I have to say, this part of the POSIX API is maddening!

99% of the time, you don't need to set errno = 0 before making a call. You check for a non-zero return, and only then look at errno.

But SOMETIMES you need to set errno = 0, because in this case readdir() returns NULL on both error and EOF.

I actually didn't realize this before working on https://oils.pub/

---

And it should go without saying: Oils simply uses libc getcwd() - we don't need to support system with a broken getcwd()!

Although a funny thing is that I just fixed a bug related to KSH that AT&T ksh (the original shell, that bash is based on!) hasn't fixed for 30+ years too!

https://www.illumos.org/issues/17442

https://github.com/oils-for-unix/oils/issues/2058

There is subtle issue that with respect to:

1) "trusting" the $PWD value you inherit from another process

2) Respecting symlinks - why you can't just call getcwd() !

    if (*p != '/' || stat(p, &st1) || stat(".", &st2) ||

Not sure if others had that same experience. I think it can't really be accurate because it doesn't have a lot of build time info. So I think they could have embraced "coarseness" more, to make it faster

Although maybe I am confusing the UI speed with the analysis speed. (One reason that this was originally a Codeberg link is that Codeberg/Forejo's UI is faster, without all the nav stuff)

There are some related links here, like How To Build Static Analyzers in Orders of Magnitude Less Code:

https://github.com/oils-for-unix/oils/wiki/Polyglot-Language...

https://oils.pub/

There's probably more.

The shell has a ____wide____ userbase with many kinds of users. Depending on your goal, the rabbit hole can go very deep (how portable across interpreters, how dependant on other binaries, how early can it work in a bootstrap scenario, etc).

These are mine:

https://github.com/alganet/coral

I am not sure if this helps or not, but if you want a bash-compatible shell, translating Oils to Ada or porting it to Windows could be easier than adding bash arrays.

It is extremely hairy, and it differs from bash version to version

https://news.ycombinator.com/item?id=43910883

I mentioned in another comment that Koichi Murase, who is a bash contributor, and wrote the largest shell program in the world, just overhauled the bash array support in OSH

Relevant tests -- he added a huge number recently, and made them work:

https://oils.pub/release/0.29.0/test/spec.wwz/osh-py/array-s...

https://oils.pub/release/0.29.0/test/spec.wwz/osh-py/array-a...

https://oils.pub/release/0.29.0/test/spec.wwz/osh-py/ble-idi...

Our tests are thorough enough that we ROUTINELY find bugs in bash, like integer overflow bugs.

Koichi also knows about the differences between say bash 4.3, 4.4, 5.0, 5.1, etc. Because he has a very large program that uses bash arrays all over the place.

---

Feel free to use our tests in any case (other shells like the Scheme shell used to bootstrap Guix have)

And feel free to post a message to https://github.com/oils-for-unix/oils if you're interested or have questions

Hm I also made a taxonomy of 5 categories of config languages, which is a bit different

Survey of Config Languages https://github.com/oils-for-unix/oils/wiki/Survey-of-Config-...

    Languages for String Data

- and we have built-in JSON, etc.

This is a hard design challenge, but I just made a release with an overhaul of Hay - https://oils.pub/release/0.28.0/

Hay version 1 was hard-coded in the interpreter - https://oils.pub/release/0.28.0/doc/hay.html

But we realized it's actually better to *self-host* it in YSH, using YSH reflection. We will be testing this by rewriting Hay in YSH

---

So that's our language design response to https://news.ycombinator.com/item?id=43386115

> It's madness that languages are effectively siloed from each other.

Instead of tiers 4, 5, 6 being silo'd, we have them all under YSH and the Oils runtime (which is tiny, 2.3 MB of pure native code).

(As a bonus, OSH also runs on the Oils runtime, and it's the most bash-compatible shell!)

----

[1] Garbage Collection Makes YSH Different - https://www.oilshell.org/blog/2024/09/gc.html

Shell, Awk, and Make Should Be Combined - https://www.oilshell.org/blog/2016/11/13.html - all these languages lack GC!

[2] Survey of Config Languages - https://github.com/oils-for-unix/oils/wiki/Survey-of-Config-... - divides this category into 5 tiers:

    1. Languages for String Data

Looking at the original posting, the maintainer knows what the bug is. The stopper is more along the lines of "This looks synthetic, so it's not high priority. If it's a priority for you, then you should fix it yourself or make an argument for its priority." followed by oguz saying "It's not actually my problem either, I'm not invested in osh.". (not actual quotes: https://github.com/oils-for-unix/oils/issues/1881#issuecomme...)

The bug is that osh leaves the script fd open for the child process when executing a command, allowing the child command to mess with the parent script. I don't see this as a huge issue for everyday interactive use, but I'd prefer my shell to not do that. If I were executing programs with limited capabilities, I'd consider this a possible vector for security bugs. That is to say, I wouldn't use osh scrpts if I cared about security.

Congrats to the fish team! Great writeup with lots of interesting detail.

I wonder if this is the biggest project that has moved from C++ entirely to Rust (or maybe even C to Rust?) It probably has useful lessons for other projects.

If I'm reading this right, it looks like fish was not released as a hybrid C++ / Rust program, with the autocxx-generated bindings. There was a release during that time, but it says "fish 3.7 remains a C++ program" [1]

It sounds like they could have released if they wanted to, but there was a last stage of testing that didn't happen until the end.

Some people didn't quite get the motivation for adding C++ features to Rust [2], and vice versa, to enable inter-op. But perhaps this is a good case study. It would be nice if you could could just write new Rust code in a C++ codebase, without writing/generating bindings, and then throwing them away, which is mentioned in this post.

--

Also the #1 gripe with Rust seems to be that it supports version detection, not feature detection.

But feature detection is better for distros, web browsers, and compilers:

https://github.com/oils-for-unix/oils/wiki/Feature-Detection...

Version detection is why Chrome and IE pretend to be Mozilla, and why Clang pretends to be GCC. Feature detection doesn't cause this problem!

[1] https://github.com/fish-shell/fish-shell/releases

[2] e.g. https://news.ycombinator.com/from?site=safecpp.org