amzn-drivers VS glommio

Amazon (AWS) cloud EC2 instance specific role (Kernel and User space networking, linux OS related). Amazon has it's own network card with it's own linux driver (open source), for user space they use DPDK (open source). https://github.com/amzn/amzn-drivers I've measured the time between calling tcp send in software, and packet leaving the NIC (network card), it is around ~50 microseconds latency, aws also stated in a paper it is around that number. Goals:- Figure out the way to build from source code and load the kernel.- Reduce latency

It means, for example, writing a FreeBSD kernel driver for Elastic Network Adapter (ENA). Both Linux kernel driver and FreeBSD kernel driver is available at https://github.com/amzn/amzn-drivers

Of course, there are.

I was mostly running on AWS. In terms of hardware, for small packets loadtests most systems are constrained on throughput, i.e. number of packets per second. Some systems saturate on interrupts reaching 100% CPU on all cores and some can not even saturate the CPU and you will see that CPU is at 60% but you can not go beyond some limit. Best systems networkwise are c6gn family types. They are also better than other cloud provide. btw, you mentioned hypervisors... About 8 months ago I opened a bug on AWS Graviton team https://github.com/amzn/amzn-drivers/issues/195 - about performance issue they had on their instances at high throughput. Recently they issued the fix. I suspect it was in their hypervisor.

In terms of my software I found many performance bugs at those speeds. For example, using a default allocator is a big no. I use mimalloc for uncontended allocations. In general, you can not use mutexes and spinlocks at those speeds. Those will just cripple the system. Sometimes it can be very annoying since you can not rely on a 3rd party library without carefully analyzing its design. For example, I could not use openmetrics c++ library because it was not performant enough. Even to implement a simple counter, say to gather statistics for INFO command becomes an interesting engineering problem:

https://doc.dpdk.org/guides/nics/ena.html

and:

https://github.com/amzn/amzn-drivers/tree/master/userspace/dpdk/enav2-vfio-patch

Enabling IOMMU on i3 or c5 metal instances is as easy as adding "iommu=1 intel_iommu=on" to /etc/default/grub followed by update-grub, reboot.

I can't get this to work. Everything I update grub and reboot I cannot re-connected via ssh. Also EC2 console fails to get good status.

My config:

Ubuntu 20.04 stock AWS AMI x86 64-bit

I used glommio as the async executor (instead of something like tokio), and it is wonderful. For people wondering whether it's "good enough" or to use C++ and seastar (as I have thought about a lot before starting this project), take the leap of faith, it's fast - both in terms of run time and to code.

My understanding is you always need a runtime, somethings needs to drive the async flow. But there are others on the market, just not without the.. market domination... of tokio.

https://github.com/smol-rs/smol looks promising simply for being minimal

https://github.com/bytedance/monoio looks potentially easier to work with than tokio

https://github.com/DataDog/glommio is built around linux io_uring and seems somewhat promising for performance reasons.

I haven't played with any of these yet, because Tokio is unfortunately the path of least resistance. And a bit viral in how it's infected tings.

I think you missed one which is based on io_uring [1].

In my benchmarks with a slightly tweaked version it was 2x faster than Nginx and and 30x faster than Python's SimpleHttpServer.

[1] https://github.com/DataDog/glommio/blob/master/examples/hype...

b) It's proven now e.g Seastar, Glommio that the fastest way to run a multi-threaded application is to have one instance with one thread pinned per CPU core. Then to have fibers/lightweight threads on top handling all of the asynchronous code. Your approach of lots of instances is the slowest so there will be a ton of unnecessary thread context-switching.

I assume Tokio itself, see e.g monoio or glommio, but also Seastar for C++.

https://github.com/DataDog/glommio Rust thread per core library.

> Few of us have really figured out io_uring. But that doesn't mean it is slower.

seastar.io is a high level framework that I believe has "figured out" io_uring, with additional caveats the framework imposes (which is honestly freeing).

Additionally the rust equivalent: https://github.com/DataDog/glommio

This use case is perfect for https://github.com/DataDog/glommio which is a thread-per-core runtime that is appropriate for latency sensitive code.

It's worth mentioning: Under "Async Executors", for "io_uring" there is only "Glommio"

I recently found out that ByteDance has a competitor library which supposedly has better performance:

https://github.com/bytedance/monoio

https://github.com/DataDog/glommio/issues/554

FYI, Datadog has a Rust library for scheduling things to run thread-per-core with io_uring

It'd be really useful for DB use cases:

https://github.com/DataDog/glommio