ghz VS grpc_bench

I once built a quick and dirty load testing tool for a public facing service we built. The tool was pretty simple - something like https://github.com/bojand/ghz but with traffic and data patterns closer to what we expected to see in the real world. We used argo-workflows to generate scale.

One thing which we noticed was that there was a considerable difference in performance characteristics based on how we parallelized the load testing tool (multiple threads, multiple processes, multiple kubernetes pods, pods forced to be distributed across nodes).

I think that when you run non-distrubuted load tests you benefit from bunch of cool things which happen with http2 and Linux (multiplexing, resource sharing etc) which might make applications seem much faster than they would be in the real world.

I'm not sure. Maybe you can write to the discussion section of the repo https://github.com/bojand/ghz/discussions

In part 1 we looked at ghz for load testing gRPC services, and now I want to cover k6, which claims to be a modern load testing tool built for developer happiness. After only a brief experience with it I can see why is that and why Grafana moved to acquire k6 earlier this year.

It should be unbound, in this particular benchmark we set ghz concurrency to 50 and connections to 5 and we don't set the rps flag of ghz (e.g. --rps=2000, from this tool)

a second container running ghz makes unary requests to the server

SayHello, GetUser, and Sum differ only by payload size. Sum is the simplest one - (int, int) -> int, GetUser is (long) -> User (medium payload), and SayHello uses exactly the same payload as this test: https://github.com/LesnyRumcajs/grpc_bench/tree/master/dotnet_grpc_bench

This is correct. The problem is not with the benchmark itself but with the implementation. If you look at the result, you can see that even with 6 "allowed" CPUs, the vertx server utilizes less than 100%. Apparently, the current vertx implementation (the one implemented in https://github.com/LesnyRumcajs/grpc_bench/tree/master/java_vertx_grpc_bench) is single-threaded or has some other limitation.

Another iteration of grpc_bench!

Because it's fast and runs on Windows, Linux, and MacOS

In general though, akka/pekko-streams are known to be one of the fastest implementations out there. Their grpc client for example even beats languages like Rust (see https://www.lightbend.com/blog/akka-grpc-update-delivers-1200-percent-performance-improvement and https://github.com/LesnyRumcajs/grpc_bench/wiki/2022-03-15-bench-results).

The whole point I was making is at least up until 8 months ago (at best, I can't commend on the stability/maturity/performance of shardcake) Akka was the only mature library/ecosystem solving this problem with also a very strong focus on performance (for example still to this day, akka/pekko-grpc is generally one of the fastest grpc implementations I am aware of, its even beating rust if you have at least 2 cores (see https://github.com/LesnyRumcajs/grpc_bench/wiki/2022-03-15-bench-results)

It would be interesting to create a new java benchmark with your implementation.

Also worth checking out the gRPC benchmarks: https://github.com/LesnyRumcajs/grpc_bench/discussions/284

dotnet is up there with Rust.