ice VS eRPC

We still don't have anything that can do what CORBA could do. gRPC doesn't even come close.

If you are interested in this type of technology, I recommend looking at ZeroC's Ice. https://zeroc.com

It's CORBA with all the warts removed, and a lot of other useful stuff added.

Are you saying: “This solution does not look particularly great”? Well, there's more. The system was not just your average, boring product, it was a modern and fancy distributed system, so it used a framework called Zeroc Ice. This is an RPC framework that also provides deployment configuration, service discovery, SSL encryption and more. By itself it’s rather interesting, but in our case it was used almost everywhere. Majority of the functions in the system were exposed as RPCs, which made them effectively globally visible. And they could call other RPCs and they called even more. And every call produced some side effects that were quite often used in condition expressions of “Algorithms” defined with DSL. These conditions were of course quite heavily nested. As a result, even though we could see in the monitoring interface which steps were already passed, it was almost impossible to tell what would happen next.

What is ICE? In-circuit emulation? RPC framework?

The most common example is DPDK [1]. It's a framework for building bespoke networking stacks that are usable from userspace, without involving the kernel.

You'll find DPDK mentioned a lot in the networking/HPC/data center literature. An example of a backend framework that uses DPDK is the seastar framework [2]. Also, I recently stumbled upon a paper for efficient RPC networks in data centers [3].

If you want to learn more, the p99 conference by ScyllaDB has tons of speakers talking about some interesting challenges.

[1] https://www.dpdk.org/.

[2] https://github.com/scylladb/seastar

[3] https://github.com/erpc-io/eRPC

My side project has been to rewrite https://github.com/erpc-io/eRPC which does RPCs over UDP with some congestion control supposedly quite fast. Paper: https://www.usenix.org/system/files/nsdi19-kalia.pdf. I never got the code to work in AWS; I believe the author focused on Mellanox NICS but that's not really commodity H/W, which is where my interests lay.

So I dug into it .. and well I'll have my own library soon. I should be able to send UDP w/o congestion control sometime this week.

eRPC uses DPDK (100% user space NIC TX/RX control) plus the author's own other ideas to get performance. Since I'm getting into NICs + DPDK (in a serious way i.e this much more involved than vanilla sys/socket.h I/O) way, way late in the game, I hope and believe DPDK is, in the medium term, the better way to go than to turn to kernel improvements in for I/O.

Like others, getting the kernel out of the way, with pinned threads seems cleaner if one can develop from scratch.

This library will be a part of something bigger, however, a key architecture point for many people is: I got a RPC/packet/message. Now should I:

* process it in-place e.g. on the thread that was doing RX?

* delegate it to another core?

* if I delegate .. to whom?

* If I delegate how do I get a response back?

In DPDK I believe these are easier to decide and well-manage in code.