esy VS domainslib

As someone who wrote a fair amount of Rust and OCaml code, I have to agree with the author.

While working at Routine (YC W21), I was tasked with porting our core library to iOS to minimize code duplication. This was a lucky opportunity to write something resembling a compiler: it took in schemas described with our in-house data exchange library and generated C (for FFI) and Swift code (for the end-users, i.e., iOS developers).

Since Routine uses OCaml for everything (which was a big motivator for joining the company—I wanted to see how that would work out), I wrote it in OCaml. The end result is a 3-5k LOC project. It's by no means a full compiler, but it was lots of fun to write. The language got in the way incredibly rarely. On average, it made my life a lot easier.

We did encounter our fair share of issues, mostly due to the cross-compilation tooling (we initially used esy [1], flirted with Nix, and eventually switched to opam-cross-ios [2]), third-party libraries, and intricacies of FFI. Those do take their toll on sanity.

[1]: https://github.com/esy/esy/

What's the current status of Esy? https://github.com/esy/esy

Any plans to backport its design back to Opam?

It's great to hear that Opam is making progress! I just wished that it would be more deeply integrated with Dune. A package manager that doesn't build is not very useful to be honest. Currently the only way to not have to care about switches and be able to clearly specify dependencies is to use the esy package manager[1] (which had lock files a while ago).

[1]: https://github.com/esy/esy/

If you start a project today I would really try to use esy (https://esy.sh/)

I actually don’t use it myself but it seems to bring the modern programming language experience to OCaml

Here is link number 1 - Previous text "esy"

Alternatively, you can use esy.sh for a simpler setup/build process (it does not require running in a Cygwin shell).

Running esy does not work. Apparently, my environment does not know that it is there. Anyone know what is going on here? I have posted this in the discussion for esy@next here. I will get back to you all when I figure this out.

For nested parallel computations (think Scientific Programming, where one would use OpenMP, Rust Rayon, etc), we have domainslib [1]. Eio, a direct-style, effect-based IO library is pretty competitive against Rust Tokio [2]. The performance will only get better as we get closer to the 5.0 release.

[1] https://github.com/ocaml-multicore/domainslib

[2] See the http server performance graphs at https://tarides.com/blog/2022-03-01-segfault-systems-joins-t...

1. Domains are the unit of parallelism. A domain is essentially an OS thread with a bunch of extra runtime book-keeping data. You can use Domain.spawn (https://github.com/ocaml-multicore/ocaml-multicore/blob/5.00...) to spawn off a new domain which will run the supplied function and terminate when it finishes. This is heavyweight though, domains are expected to be long-running.

2. Domainslib is the library developed alongside multicore to aid users in exploiting parallelism. It supports nested parallelism and is pretty highly optimised (https://github.com/ocaml-multicore/domainslib/pull/29 for some graphs/numbers). The domainslib repo has some good examples: https://github.com/ocaml-multicore/domainslib/tree/master/te...

3. We've not tested against other forms of parallelism. There isn't anything stopping you exploiting SIMD in addition to parallelism from domains.

4. No, we've not compared performance by OS.

5. No plans for the multicore team to look at accelerator integration at the moment.

[3] Domainslib -- Parallel Programming over Multicore OCaml, https://github.com/ocaml-multicore/domainslib