design VS oil

First of all, a quick rundown of what WASM is. It stands for Web Assembly. In essence, similar to how Java compiles down to a bytecode that is interpreted by a Java Virtual Machine, Web Assembly is a different bytecode interpreted by the browser. Many different languages can compile into WASM, and Javascript can interface with it like a module. In my case, I wrote a lot of the source code in Rust and compiled it down to a WASM module, then called into it from Javascript.

WebAssembly (abbreviated Wasm) is a binary instruction format for a stack-based virtual machine. Wasm is designed as a portable compilation target for programming languages, enabling deployment on the web for client and server applications. - https://webassembly.org

With WebAssembly we can compile our C++ codebase into a wasm module for the browser. So when you look at a SciChart.js chart you're actually seeing our C++ graphics engine wrapped for JavaScript.

I should add, however, that the unmentioned elephant in the room is V8 JIT (TurboFan), which simply doesn't handle irreducible control flow. While there are some valid theoretical arguments around the current arrangement in Wasm, looking at the history of the associated discussions makes it pretty obvious that having V8 support Wasm and generate fast code similar to what it can do for asm.js was an overriding concern in many cases. And Google straight up said that if Wasm has ICF, they will not bother supporting such cases, so it will be done by a much slower fallback:

https://github.com/WebAssembly/design/issues/796#issuecommen...

AFAIK no other Wasm implementation has the same constraint - the rest generally tend to desugar everything to jumps and then proceed from there. So this is, at least to some extent, yet another case of a large company effectively forcing an open standard to be more convenient for them specifically.

https://github.com/WebAssembly/design/issues/1397

> Currently allocating more than ~300MB of memory is not reliable on Chrome on Android without resorting to Chrome-specific workarounds, nor in Safari on iOS.

That's about allocating CPU memory but the GPU memory situation is similar.

As far as I can tell (5 minutes of internet research) this was to allow easier compilation to JavaScript as a fallback in the days when WASM wasn't widely supported.

"Please add goto" issue has been open since 2016:

https://github.com/WebAssembly/design/issues/796

Most interesting comment:

> The upcoming Go 1.11 release will have experimental support for WebAssembly. This will include full support for all of Go's features, including goroutines, channels, etc. However, the performance of the generated WebAssembly is currently not that good.

> This is mainly because of the missing goto instruction. Without the goto instruction we had to resort to using a toplevel loop and jump table in every function. Using the relooper algorithm is not an option for us, because when switching between goroutines we need to be able to resume execution at different points of a function. The relooper can not help with this, only a goto instruction can.

> It is awesome that WebAssembly got to the point where it can support a language like Go. But to be truly the assembly of the web, WebAssembly should be equally powerful as other assembly languages. Go has an advanced compiler which is able to emit very efficient assembly for a number of other platforms. This is why I would like to argue that it is mainly a limitation of WebAssembly and not of the Go compiler that it is not possible to also use this compiler to emit efficient assembly for the web.

^ https://github.com/WebAssembly/design/issues/796#issuecommen...

When I implemented a WASM compiler, the only source of float-based non-determinism I found was in the exact byte representation of NaN. Floating point math is deterministic. See https://webassembly.org/docs/faq/#why-is-there-no-fast-math-... and https://github.com/WebAssembly/design/blob/main/Nondetermini....

> To work with GC, you need some way to track if the GC'd object is accessible in WASM itself.

I've never heard of a GC with that kind of API. Usually any native code that holds a GC reference would either mark that reference as a root explicitly (eg. https://github.com/WebAssembly/design/issues/1459) or ensure that it can be traced from a parent object. Either way, this should prevent collection of the object. I agree that explicitly checking whether a GC'd object has been freed would not make any sense.

> The reason why you probably need a custom string type is so you can actually embed string literals without relying on interop with the environment.

WASM already has ways of embedding flat string data. This can be materialized into GC/heap objects at module startup. This must happen in some form anyway, as all GC-able objects must be registered with the GC upon creation, for them to be discoverable as candidates for collection.

Overall I still don't understand the issue. There is so much prior art for these patterns in native extensions for Python, PHP, Ruby, etc.

Giving you a buffer that grows is the allocation approach I am talking about. This is not how your OS works. Your OS itself works with an allocator that does a pretty good job making sure that your memory ends up not fragmented. Because WASM is in between, the OS is not in control of the memory, and instead the browser is. The browser implementation of "bring your own allocator" is cute but realistically just a waste of time for everybody who wants to deploy a wasm app because whatever allocator you bring is crippled by the overarching allocator of the browser messing everything up.

It seems like the vendors are recognizing this though, with firefox now having a discard function aparently!

https://github.com/WebAssembly/design/issues/1397

will prevent almost all of the "silent footguns".

YSH has strict:all and then a bunch of NEW features.

There's been good feedback recently, which has led to many concrete changes. So your experience can definitely influence the language! https://github.com/oilshell/oil/wiki/Where-To-Send-Feedback

Regarding writing tools/tests/benchmarks in bash+Python, vs. writing tools in your main language:

I think we might eventually concede that something Debian-like is the “standard development environment” (at least for server side stuff, i.e. not iOS apps)

In this case, bash+Python is a non-issue. It works extremely reliably. That’s actually why I use it! Everything else seems to break, or it’s really slow (node.js is a very common alternative).

- Microsoft conceded this back in ~2017, by building Linux into their kernel with WSL, and providing Ubuntu on top

Yes bash + Python is a disaster on Windows (I have scars from it), but Microsoft agrees that the right place to solve that is in Windows :-)

- Every CI system runs Debian/Ubuntu

- Every hosting provider runs Debian/Ubuntu

- Every online dev env like gitpod.io provides Debian/Ubuntu

This is somewhat related to remote dev envs: https://lobste.rs/s/ucirlx/lapdev_self_hosted_remote_dev

One vision for https://www.oilshell.org/ is that the CI environment is the dev environment is the hosting environment.

Everything is just an equal node in a distributed system. BUT it’s more git like, in that you explicitly sync and work “locally”, wherever that is. You don’t have the network chatter and flakiness of “the cloud”.

Oils has a very large set of monotonically increasing properties too - https://www.oilshell.org/release/0.21.0/quality.html

All that is bash+Python that is run on every commit, and it’s extremely good at catching bugs and perf regressions.

I’m skeptical that any project has that level of quality automation written in pure Rust or Zig. More likely it’s a bunch of cloud services with YAML.

Also a bunch of “hard-coded” toolchains that you can’t script with bespoke code. Like some shell commands in your package.json, which is just a worse way of writing a shell script.

Our quality process is all self-hosted, in the repo, and runs on both Github Actions and sourcehut - https://www.oilshell.org/release/0.21.0/pub/metrics.wwz/line...

bash and Python runs perfectly on Github Actions and sourcehut, with zero change. Containers also do.

(Although we need to unify the CI and release, because the release runs on 2 different real hardware machines, while CI is cloud only.)

Also, a main point Oils is that bash now has another highly compatible, spec-driven implementation – OSH. Having 2 independent implementations is something newer languages don’t have.

(copy of lobste.rs comment)

in your bashrc to enable it. I've used it for probably ~18 years now.

It also works with https://www.oilshell.org/ since we use GNU readline. Just 'set -o vi' in ~/.config/oils/oshrc

No other shell does that.

But I didn't know it was called MULTIOS until now. (I guess that's read "mult I/O's"? I have a hard time not reading it was multi-OS :) )

It seems a bit niche to be honest, but it's possible to support in Oils.

---

Oils also uses Unix domain sockets already for the headless shell protocol

https://github.com/oilshell/oil/wiki/Headless-Mode

We could do something like dgsh, but so far I haven't seen a lot of uptake / demand. Every time it's mentioned, somebody kinda wants it, and then it kinda peters out again ... still possible though.

I think flat files work fine for a lot of use cases, and once you add streaming, you also want monitoring, more control over backpressure/queue sizes, etc.

which works well. You don't have to clean when rebuilding variants. IMO this is 100% essential for writing C++ these days. You need a bunch of test binaries, and all tests should be run with ASAN and UBSAN.

---

I wrote a mini-bazel on top of Ninja with these features:

https://www.oilshell.org/blog/2022/10/garbage-collector.html...

So it's ~1700 lines, but for that you get the build macros like asdl_library() generating C++ and Python (the same as proto_library(), a schema language that generates code)

And it also correctly finds dependencies of code generators. So if you change a .py file that is imported by another .py file that is used to generated a C++ header, everything will work. That was one of the trickier bits, with Ninja implicit dependencies.

I also use the Bazel-target syntax like //core/process

This build file example mixes low level Ninja n.rule() and n.build() with high level r.cc_library() and so forth. I find this layering really does make it scale better for bigger projects

https://github.com/oilshell/oil/blob/master/asdl/NINJA_subgr...

Some more description - https://lobste.rs/s/qnb7xt/ninja_is_enough_build_system#c_tu...

This is sort of a category error...

re2c is a lexer generator, and YAML and Python are recursive/nested formats.

You can definitely use re2c to lex them, but it's not the whole solution.

I use it for everything possible in https://www.oilshell.org, and it's amazing. It really reduces the amount of fiddly C code you need to parse languages, and it drops in anywhere.

SEEKING VOLUNTEERS - https://www.oilshell.org/ - https://github.com/oilshell/oil/

I'm looking for people to help fill out the "standard library" for Oils/YSH. We're implementing a shell for Python and JavaScript programmers who avoid shell!

On the surface, this is writing some very simple functions in typed Python. But I've realized that the hardest parts are specifying, TESTING, and documenting what the functions do.

---

The most recent release announcement also asks for help - https://www.oilshell.org/blog/2024/01/release-0.19.0.html (long)

If you find all those details interesting (if maybe overwhelming), you might have a mind for language design, and could be a good person to help.

Surveying what Python and JavaScript do is very helpful, e.g. for the recent Str.replace() function, which is nontrivial (takes a regex or string, replacement template or string)

But there are also very simple methods to get started, like Dict.values() and List.indexOf(). Other people have already contributed code. Examples:

https://github.com/oilshell/oil/commit/58d847008427dba2e60fe...

https://github.com/oilshell/oil/commit/8f38ee36d01162593e935...

This can also be useful to tell if you'll have fun working on the project - https://github.com/oilshell/oil/wiki/Where-Contributors-Have...

More on #help-wanted on Zulip (requires login) - https://oilshell.zulipchat.com/#narrow/stream/417617-help-wa...

Please send a message on Github or Zulip! Or e-mail me andy at oilshell dot org.

This is true, but then the corrolary is that new PRs need to come with this higher and rigorous level of test coverage.

And then that becomes a bit of a barrier to contribution -- that's a harness

I often write entirely new test harnesses for features, e.g. for https://www.oilshell.org, many of them linked here . All of these run in the CI - https://www.oilshell.org/release/latest/quality.html

The good thing is that it definitely helps me accept PRs faster. Current contributors are good at this kind of exhaustive testing, but many PRs aren't