wasm32-wasi-benchmark VS cpp_weekly

On other benchmarks I'm seeing numbers closer to 20% slower, e.g. https://github.com/wasm3/wasm3/blob/main/docs/Performance.md and https://github.com/second-state/wasm32-wasi-benchmark. It's numerical code, which is the best case scenario for a native binary. It's much closer on an average web app or server workload, e.g. https://krausest.github.io/js-framework-benchmark/current.html - you can find WASM frameworks that beat most JS frameworks on there, but that is not as impressive considering the state of the JS ecosystem. Overall, it's already under 50%, and there is still plenty of room for improvement.

Source code of test cases: https://github.com/second-state/wasm32-wasi-benchmark

We did benchmarks last year but not recently (Will update if there are new ones).

Paper: A Lightweight Design for High-performance Serverless Computing, published on IEEE Software, Jan 2021. https://arxiv.org/abs/2010.07115

Article: Performance Analysis for Arm vs. x86 CPUs in the Cloud, published on infoQ.com, Jan 2021. https://www.infoq.com/articles/arm-vs-x86-cloud-performance/

Source code of test cases: https://github.com/second-state/wasm32-wasi-benchmark

This response is a mess, I apologize.

One can do lots of things. Yes all of those things are still issues, but in enumerating all the things as you have, you make a compelling argument not to use C++. If one picks something else like SML, Go or even Rust, it allows them to focus on those common areas of correctness and not also, all that other stuff.

I don't think you are apologizing for C++, but I hear this a lot, whatabout all the other parts that are fixed by a more sound system? My response is that we only have so many decisions we can make in the day, using a system that removes whole classes of problems allows me to focus on the non-accidental (forced) complexity.

> Rust as Wasmtime is, that alone wouldn't actually give me much more confidence that it's safe

I never made that argument. But let's say I did. Given two systems, one made in Rust and one made in C++. How do I audit the C++ code to ensure that it has the same level of safety as basically _anything_ else?

WasmEdge is using code coverage, good! https://app.codecov.io/gh/WasmEdge/WasmEdge

Only 80% coverage, not great.

I see mention of fuzzing being used, but I only get 69 hits in the codebase for "fuzz" and no documentation about how it actually works.

Wasmtime includes documentation on how to use their fuzzer, https://github.com/bytecodealliance/wasmtime/tree/main/fuzz

In my cursory look at the WasmEdge codebase I see

No application UBSan [1]

No application of control flow guard [2], either in the build or in the generated LLVM output.

Infact the only sanitization flags they are passing are -fsanitize=fuzzer,address

0 mentions of 'control flow' in the WasmEdge codebase, 65 mentions in Wasmtime. Although it doesn't appear that Wasmtime is itself being compiled with cfguard.

309 mention of spectre mitigations in Wasmtime, 1 link to a cloudflare blogpost in WasmEdge.

In my cursor look at Wasmtime, I see

Much better fuzzing support, 85 uses of unsafe in cranelift itself, 1400+ uses of unsafe in crates/*

The changelog and git messages for WasmEdge talk about type errors (those type errors look impossible in Rust), bus errors, out of bounds, etc.

It does not inspire confidence. C++ is already starting from behind. This software is targeted to be run on bare metal, embedded and widely distributed. It just doesn't have the level of detail around testing and correctness that I would expect for such a critical piece of infrastructure.

I am not saying that Wasmtime does not also have issues, that would be preposterous. But on the face of it, WasmEdge has a lot of work to do to catch up to where Wasmtime already is.

Wasmtime appears to care about the security properties of their dependencies with https://mozilla.github.io/cargo-vet/

It does not look like the WasmEdge project applies this basic list of practices as outlined by Jason Turner [n]

[1] https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html

[2] https://www.zdnet.com/article/microsofts-control-flow-guard-... https://clang.llvm.org/docs/ControlFlowIntegrity.html

[n] https://www.youtube.com/watch?v=q7Gv4J3FyYE https://github.com/lefticus/cpp_weekly/issues/175