Rustler VS Cap'n Proto

For performance intensive tasks, you could rely on Rust NIFs, there is this great project: https://github.com/rusterlium/rustler

My last project with Elixir was using Elixir merely as an orchestrator of static binaries (developed in golang) which were talking in JSON via stdin/stdout.

From the moment we discovered Tauri, we really felt like this was the perfect fit. The API is really solid, the configuration files are minimal and easy to understand, and the usage of Rust makes it way easier to add new functionalities and think about interesting ways of interoperating with Elixir via the Rustler library.

Elixir/Rust is the new Python/C++, and Rustler makes the communicating between the 2 languages super easy: https://github.com/rusterlium/rustler

Rustler is so awesome for this. Write Elixir NIFs in Rust? Yes, please!

But really, I would suggest thinking about what you want to build before "how" or "with which tool" - one of the signs of a person becoming a good engineer is having an array of tools at their disposal and being able to choose a correct tool for the correct task. Rust also excels in integrating with other languages - with JS via WebAssembly (a bit of self-promotion, for example), with Elixir via Rustler, with Python via PyO3 and PyOxidizer, etc. So you absolutely can start writing a frontend app with JS, or a distributed system with Elixir, or a data processing/ML app with Python and use Rust to speed up critical parts of those. Or, in reverse, you can start with Rust & add new capabilities to whatever you're building, that being a frontend, a resilient chat interface, or an ML model.

I created a elixir (wrapper) library to generate password hashes. Other Elixir libraries use a C NIF to generate password hashes. This libary uses a Rust NIF (using Rustler) and the Rust libraries the generate the different hashes. Additionally this library uses RustlerPrecompiled so you don't need to have a Rust compiler installed to use this library. It supports argon2, scrypt, brypt and pbkdf2.

> I guess, why not use Rust entirely instead of as a FFI into Elixir or other backend language?

Because Rust brings none of the benefits of the BEAM ecosystem to the table.

I was an early Elixir adopter, not working currently as an Elixir developer, but I have deployed one of the largest Elixir applications for a private company in my country.

I know it has limits, but the language itself is only a small part of the whole.

Take ML, Jose Valim and Sean Moriarity have studied the problem, made a plan to tackle it and started solving it piece by piece [1] in a tightly integrated manner, it feels natural, as if Elixir always had those capabilities in a way that no other language does and to put the icing on the cake the community released Livebook [2] to interactively explore code and use the new tools in the simplest way possible, something that Python notebooks only dream of being capable of, after a decade of progress

That's not to say that Elixir is superior as a language, but that the ecosystem is flourishing and the community is able to extract the 100% of the benefits from the tools and create new marvellously crafted ones, that push the limits forward every time, in such a simple manner, that it looks like magic.

And going back to Rust, you can write Rust if you need speed or for whatever reason you feel it's the right tool for the job, it's totally integrated [3][4], again in a way that many other languages can only dream of, and it's in fact the reason I've learned Rust in the first place.

The opposite is not true, if you write Rust, you write Rust, and that's it. You can't take advantage of the many features the BEAM offers, OTP, hot code reloading, full inspection of running systems, distribution, scalability, fault tolerance, soft real time etc. etc. etc.

But of course if you don't see any advantage in them, it means you probably don't need them (one other option is that you still don't know you want them :] ). In that case Rust is as good as any other language, but for a backend, even though I gently despise it, Java (or Kotlin) might be a better option.

[1] https://github.com/elixir-nx/nx https://github.com/elixir-nx/axon

[2] https://livebook.dev/

[3] https://github.com/rusterlium/rustler

[4] https://dashbit.co/blog/rustler-precompiled

unfortunately as of the time of this writing, rustler does not support generic type intefaces so I guess this is impossible?

One thing that drew me to Rust was actually Elixir/Erlang calling out to it for certain specialized needs. Within Elixir/Erlang you get best of breed concurrency but exiting the BEAM to run other code is unsafe. Calling out to Rust, however, comes with great safety guarantees.

Managing concurrency outside of Rust and then calling Rust for the more focused and specialized work is a good combination IMO.

https://github.com/rusterlium/rustler

Yeah I pretty much only use my own alternate container implementations (from KJ[0]), which avoid these footguns, but the result is everyone complains our project is written in Kenton-Language rather than C++ and there's no Stack Overflow for it and we can't hire engineers who know how to write it... oops.

[0] https://github.com/capnproto/capnproto/blob/v2/kjdoc/tour.md

- may massively reduce the latency involved.

Those sharing Cap'n Proto-encoded data may have particular interest. Cap'n Proto (https://capnproto.org) is fantastic at its core task - in-place serialization with zero-copy - and we wanted to make the IPC (inter-process communication) involving capnp-serialized messages be zero-copy, end-to-end.

That said, we paid equal attention to other varieties of payload; it's not limited to capnp-encoded messages. For example there is painless (<-- I hope!) zero-copy transmission of arbitrary combinations of STL-compliant native C++ data structures.

To help determine whether Flow-IPC is relevant to you we wrote an intro blog post. It works through an example, summarizes the available features, and has some performance results. https://www.linode.com/blog/open-source/flow-ipc-introductio...

Of course there's nothing wrong with going straight to the GitHub link and getting into the README and docs.

Currently Flow-IPC is for Linux. (macOS/ARM64 and Windows support could follow soon, depending on demand/contributions.)

FWIW, my C++ toolkit library, KJ, does the same thing.[0]

But presumably you could still write a condition predicate which looks at things which aren't actually part of the mutex-wrapped structure? Or does is the Rust type system able to enforce that the callback can only consider the mutex-wrapped value and values that are constant over the lifetime of the wait? (You need the latter e.g. if you are waiting for the mutex-wrapped value to compare equal to some local variable...)

[0] https://github.com/capnproto/capnproto/blob/e6ad6f919aeb381b...

The second one is to encode data in such a way that you can read it and operate on it directly from the buffer. You write data in a layout that is the same, or easily transformed as types in memory. To do that you usually need to encode with a known schema, only Sized types to efficiently compute fields locations as offsets in the buffer, and you usually represent pointers as offset into the encode. You can look at capnproto protocol for instance https://capnproto.org/

Worked well for Cap'n Proto (the cerealization protocol)! https://capnproto.org/

With all due respect, you read completely wrong.

* The very first use case for which Cap'n Proto was designed was to be the protocol that Sandstorm.io used to talk between sandbox and supervisor -- an explicitly adversarial security scenario.

* The documentation explicitly calls out how implementations should manage resource exhaustion problems like deep recursion depth (stack overflow risk).

* The implementation has been fuzz-tested multiple ways, including as part of Google's oss-fuzz.

* When there are security bugs, I issue advisories like this:

https://github.com/capnproto/capnproto/tree/v2/security-advi...

* The primary aim of the entire project is to be a Capability-Based Security RPC protocol.

I like how they use capability-based security [0] and use Cap'n Proto protocol. This is another technology that is slow to get broad adoption, but has many things going for when compared to e.g. Protocol Buffers (Cap'n Proto is created by the primary author of Protobuf v2, Kenton Varda).

[0] https://sandstorm.io/how-it-works#capabilities

[1] https://capnproto.org

Related but not Rust-specific: FlatBuffers, Cap'n Proto.