pgfe VS Seastar

I like working in C++, after a decade of working in Java, Python, Javascript and Clojure, I find working in C++ (which I learned before these other languages) to be quite fun and pleasant, at least with relatively modern C++.

I've been, on and off, working on a little toy game engine, for a few years. Its a mix of keeping up with C++ advancements, learning various concepts like physically based rendering, and just the fun of crafting a big project, with no constraints other than my time and ability, no deadlines, no expectation of releasing anything. Its cathartic and enjoyable. I really do enjoy it.

Last September, I got frustrated with something I was working on in a more serious capacity. It was some server software, it responded to HTTP requests, it accessed third party services over HTTP and Websockets, it talked to a Postgres database. Overall it was an event driven system that transformed data and generated actions that would be applied by talking to third party services. The "real" version was written in Clojure and it worked pretty well. I really like Clojure, so all good.

But because I was frustrated with some things about how it ran and the resources it took up, I wondered what it would be like if I developed a little lean-and-mean version in C++. So I gave it a try as a side project for a few weeks. I used doctest[1] for testing, immer[2] for Clojure-like immutable data structures, [3] lager for Elm-like application state and logic management, Crow[4] for my HTTP server, ASIO[5] and websocketpp[6] for Websockets, cpp-httplib[7] as a HTTP client and PGFE[8] for Postgres, amongst some other little utility libraries. I also wrote it in a Literate Programming style using Entangled[9], which helped me keep everything well documented and explained.

For the most part, it worked pretty well. Using immer and lager helped keep the logic safe and to the point. The application started and ran very quickly and used very little cpu or memory. However, as the complexity grew, especially when using template heavy libraries like lager, or dealing with complex things like ASIO, it became very frustrating to deal with errors. Template errors even on clang became incomprehensible and segmentation faults when something wasn't quite right became pretty hard to diagnose. I had neither of these problems working on my game engine, but both became issues on this experiment. After a few weeks, I gave up on it. I do think I could have made it work and definitely could go back and simplify some of the decisions I made to make it more manageable, but ultimately, it was more work than I had free time to dedicate to it.

So my experience was that, yes, you can write high level application logic for HTTP web backends in C++. You can even use tools like immer or lager to make it feel very functional-programming in style and make the application logic really clean. Its not hard to make it run efficiently both in terms of running time and memory usage, certainly when comparing to Clojure or Python. However, I found that over all, it just wasn't as easy or productive as either of those languages and I spent more time fighting the language deficiencies, even with modern C++, than I do when using Clojure or Python.

I think I would think very long and hard before seriously considering writing a web backend in C++. If I had the time, I'd love to retry the experiment but using Rust, to see how it compares.

[1] https://github.com/doctest/doctest

[2] https://github.com/arximboldi/immer

[3] https://github.com/arximboldi/lager

[4] https://github.com/CrowCpp/crow

[5] https://think-async.com/Asio/

[6] https://www.zaphoyd.com/projects/websocketpp/

[7] https://github.com/yhirose/cpp-httplib

[8] https://github.com/dmitigr/pgfe

[9] https://entangled.github.io/

I used glommio as the async executor (instead of something like tokio), and it is wonderful. For people wondering whether it's "good enough" or to use C++ and seastar (as I have thought about a lot before starting this project), take the leap of faith, it's fast - both in terms of run time and to code.

b) It's proven now e.g Seastar, Glommio that the fastest way to run a multi-threaded application is to have one instance with one thread pinned per CPU core. Then to have fibers/lightweight threads on top handling all of the asynchronous code. Your approach of lots of instances is the slowest so there will be a ton of unnecessary thread context-switching.

The most common example is DPDK [1]. It's a framework for building bespoke networking stacks that are usable from userspace, without involving the kernel.

You'll find DPDK mentioned a lot in the networking/HPC/data center literature. An example of a backend framework that uses DPDK is the seastar framework [2]. Also, I recently stumbled upon a paper for efficient RPC networks in data centers [3].

If you want to learn more, the p99 conference by ScyllaDB has tons of speakers talking about some interesting challenges.

[1] https://www.dpdk.org/.

[2] https://github.com/scylladb/seastar

[3] https://github.com/erpc-io/eRPC

I assume Tokio itself, see e.g monoio or glommio, but also Seastar for C++.

https://core.dpdk.org/supported/ lists supported nics. You're best just reading material from the dpdk website for figuring out roughly what it is. It is used for a lot of different goals. For most web C++ stuff it's mainly used because you can avoid round trips of data passing through the kernel and can reference network data without tons of copying. For an example check out the SeaStar framework, https://seastar.io/, which is under the hood of ScyllaDB.

As this is a Kafka sub, this may be a good opportunity to mention that Redpanda is based on the same framework (seastar) as Scylla. The idea of sharding work to CPU cores turns out to apply very well to the Kafka data model, too!

Seastar which is a thread per core runtime written by the Scylla devs thats used in both Redpanda and Scylla as the underlying runtime. https://github.com/scylladb/seastar

ScyllaDB is, ironically, maybe one of the worst examples the author could have come up with for "abstraction" in the article.

If folks aren't familiar with their work/internal tech, go check out some of their repos like Seastar. They have some of the most talented systems programmers on the planet writing thin veneers over kernel and hardware API's to squeeze every ounce out of performance.

https://github.com/scylladb/seastar

I know it's beside the point, but I just had to share because I thought that was funny

I’ve seen frameworks for c++ (https://seastar.io/) and rust (https://github.com/actix/actix) which support what you’re describing out of the box.

If you're interested in scaling and asynchronous programming in c++ I highly recommend you investigate the SeaStar application framework. You wouldn't build a web service with SeaStar, rather you would build the infrastructure that you would use to build the web service on top of. https://github.com/scylladb/seastar