runtimelab VS FrameworkBenchmarks

For everyone reading this blog post I caution that the conclusions there are at best creative interpretations of the notes written down here: https://github.com/dotnet/runtimelab/blob/feature/async2-exp...

It is quite literally impossible to draw conclusions on e.g. memory consumption until the work on this, which is underway, makes it into mainline runtime. It's important to understand that the experiment was first and foremost a research to look into modernizing async implementation, and was a massive success. Now once that is proven, the tuned and polished implementation will be made.

Once it is done and makes into a release (it could even be as early as .NET 10), then further review will be possible.

This FAQ is a bit outdated in places, and is not something most users should worry about in practice.

JVM Green Threads here serve predominantly back-end scenarios, where most of the items on the list are not of concern. This list also exists to address bad habits that carried over from before the tasks were introduced, many years ago.

In general, the perceived want of green threads is in part caused by misunderstanding of that one bad article about function coloring. And that one bad article about function coloring also does not talk about the way you do async in C#.

Async/await in C# is just a better model with explicit understanding where a method returns an operation that promises to complete in the future or not, and composting tasks for easy (massive) concurrency is significantly more idiomatic than doing so with green threads or completable futures that existed in Java before these.

Also one change to look for is "Runtime Handled Tasks" project in .NET that will replace Roslyn-generated state machine code with runtime-provided suspension mechanism which will only ever suspend at true suspension points where task's execution actually yields asynchronously. So far numbers show at least 5x decrease in overhead, which is massive and will bring performance of computation heavy async paths in line with sync ones: https://github.com/dotnet/runtimelab/blob/feature/async2-exp...

Async/await is not a tight corner as showcased by a multitude of languages adopting the pattern: Rust, Python, JavaScript and Swift.

In fact, it is a clean abstraction where future progress is possible while retaining the convenience of its concurrency syntax and task composition.

Green threads experiment proved net negative in terms of benefit but its the follow-up work on modernizing the implementation detail was very successful: https://github.com/dotnet/runtime/issues/94620 / https://github.com/dotnet/runtimelab/blob/feature/async2-exp...

It also seems that common practices in Java indicate that properties are not a mistake as showcased by popularity of Lombok and dozens of other libraries to generate builders and property-like methods (or, worse, Java developers having to write them by hand).

There was a "green thread" experiment for dotnet a while ago, here is the conclusion: https://github.com/dotnet/runtimelab/issues/2398

Experiment result write-up: https://github.com/dotnet/runtimelab/blob/e69dda51c7d796b812...

TLDR: The green threads experiment was a failure as it found (expected and obvious) issues that the Java applications are now getting to enjoy, joining their Go colleagues, while also requiring breaking changes. It, however, gave inspiration to subsequent re-examination of current async/await implementation and whether it can be improved by moving state machine generation and execution away from IL completely to runtime. It was a massive success as evidenced by preliminary overhead estimations in the results.

Yeah, it kind of is. There are quite a few of experiments that are conducted to see if they show promise in the prototype form and then are taken further for proper integration if they do.

Unfortunately, object stack allocation was not one of them even though DOTNET_JitObjectStackAllocation configuration knob exists today, enabling it makes zero impact as it almost never kicks in. By the end of the experiment[0], it was concluded that before investing effort in this kind of feature becomes profitable given how a lot of C# code is written, there are many other lower hanging fruits.

To contrast this, in continuation to green threads experiment, a runtime handled tasks experiment[1] which moves async state machine handling from IL emitted by Roslyn to special-cased methods and then handling purely in runtime code has been a massive success and is now being worked on to be integrated in one of the future version of .NET (hopefully 10?)

[0] https://github.com/dotnet/runtime/issues/11192

[1] https://github.com/dotnet/runtimelab/blob/feature/async2-exp...

Unlike these folks from dotnet, which tested directly on ASP for real workload

  https://github.com/dotnet/runtimelab/issues/2398?darkschemeovr=1

[1] https://github.com/dotnet/runtimelab/issues/2398

The only way to avoid it is to not build on top of Java or not adding any features on top of Java.

> To give another example with C#, there has been a lot of recent discussion about finding potential alternatives to their async-await concurrency model. They cite the level of effort it takes to maintain the async await style code and the costs that come from this.

I had a very different take-away. They did PoC with virtual threads and decided it's not worth the switch now and async-await that they have is good enough.

https://github.com/dotnet/runtimelab/issues/2398

> Some of the languages it gets compared too aren't even that old yet.

C# is old enough to drink and Scala just had its 20th birthday this week :)

I share your sentiment but Swoole is an async event loop implementation for PHP.

It's relatively fast. Like 300k req/s on an old intel i5 with 4 cores fast.

https://openswoole.com/benchmark

Or 40% of top Tech Empower speed in composite score: https://www.techempower.com/benchmarks/#hw=ph&test=composite...

We have been using ruby extensively through our org for years.

We tried java, go, node, elixir, python.

Pretty much it comes down to ruby on rails have a very opitionated way of doing things.

We mix some DDD and recently High Performance PostgreSQL for Ruby on Rails https://www.amazon.com/dp/B0CX876RLY

It is slow. How much? According to tech empowered https://www.techempower.com/benchmarks/#hw=ph&test=query&sec... something like 6 times slower than java-vertx-postgres

Me moved to rails and flutter to have a quick coding hot-reloading loop

And we determined it was easier for our devs to all master a tool, instead of jumping between frameworks and DBS (we use rails, Kamal, postgres, redis and clickhouse, datadog)

The result has been awesome. Our web pages pass Google pagespeed in green, we serve millions of requests a day without issues, datadog APM helps a lot and they move between projects easily. The have hotspots like query autocompletion in one of our sites, which we do in memory with a true library which runs a C library underneath for under 100ms response times.

If your org is competing in a really really competitive field and you need to reduce costs to zero (as whatsapp had to do with Freebsd and erlang) go ahead and do your tradeoffs. But if you don't, rails is the one man army framework. Even more with hotwire.

Spring shines when it comes to having a bunch of built-in modules for everything: authentication, configuration, dependency injection, monitoring, various template engines and databases - whatever you want. The advantage is that you don't need to spend time investigating and arguing dependencies - they're all right there. It's also easy to structure your projects and to hire developers who claim to know Spring (whether they actually do understand how Spring works is another story).

But Spring has a lot of issues too:

- Performance: This is the main reason we avoid it for most projects. Spring is slow. This appears to be the main reason OP has created a Rust version of the Spring framework. Of course, Rust has less overhead than Java, but there are many Java frameworks that are faster than Spring. Spring is just proudly underoptimized. Almost anything else I could say about Spring or you could say about other frameworks may be subjective or anecdotal — but speed is easy to quantify. If you look at the TechEmpower benchmarks, there is an order of magnitude of difference between Spring and lightweight frameworks like Vert.x and Jooby and even some optimized heavyweight frameworks like Quarkus[1]. If you care about performance you just cannot use Spring.

- Inscrutable Magic: Spring has a lot of annotation-based magic. A lot of Spring enthusiasts like it, since it gets stuff done and reduces boilerplate. But it also makes your framework behavior hard to understand and scrutinize. If you want to know what an annotation does, you can't just click "go to definition" in your editor and look at its source code. You need to find out where all the possible annotation processors are and then read all the relevant code until you find how that particular annotation is processed into generated code or wrapper classes or whatever.

- Security: I beg to differ here. Spring Security can save you from the bugs that you would have if you wrote your own authentication code, but the code that Spring itself brings to the table does not have a very good track record. The sheer amount of CVEs found in Spring[2] is staggering. A lot of is due to popularity and exposure, but this is also due to Spring's desire to include everything under the sun and do as much as possible behind the scenes, with automagic. A great example of this approach is how Spring Actuator used to expose a lot of sensitive endpoints (including a full heapdump endpoint) by default, on a standard path. This needed you to add the actuator module, but a lot of servers included it because this is the standard way to enable health checks in Spring, and almost every cloud infrastructure nowadays requires health checks. The end result is that if you wanted your Spring Boot 1.5 web server to be secure, you'd had to explicitly disable these endpoints[3]. Even with modern Spring versions, the sensitive "/actuator/info" endpoint is still exposed by default.

[1] https://www.techempower.com/benchmarks/#hw=ph&test=fortune&s...

[2] https://spring.io/security

[3] https://docs.stackhawk.com/vulnerabilities/40042/

Blows most popular TS frameworks out of the water.

https://www.techempower.com/benchmarks/#section=data-r22&hw=...

For you: https://gprivate.com/6chku

Installation: Follow the instructions on the TechEmpower benchmarks site.

Since 2013, TechEmpower has established a backend framework benchmark. They meticulously define benchmark specifications and maintain an open-source approach that encourages contributions from the community. This benchmark has become a respected standard in the tech industry, serving as a reliable yardstick for technology competitors to assess the performance of their solutions (exemple Go Fiber, C# Asp.net, JS Just). So I can trust the Techempower benchmark.

Interested to see how this fares on Tech Empower's benchmarks: https://www.techempower.com/benchmarks/#hw=ph&test=fortune&s...

At the moment it is at the bottom as a "did not complete"

It seems like Rust is doing a pretty good job of applying to web apps and APIs:

https://www.techempower.com/benchmarks