Debunking Cloudflare’s recent performance tests

This page summarizes the projects mentioned and recommended in the original post on news.ycombinator.com

Our great sponsors
  • OPS - Build and Run Open Source Unikernels
  • SonarQube - Static code analysis for 29 languages.
  • Scout APM - Less time debugging, more time building
  • GitHub repo js-compute-runtime

    JavaScript runtime for Fastly [email protected]

    btw. for what it's worth their javascript to wasm is opensource:

    - https://github.com/fastly/js-compute-runtime

    - https://github.com/tschneidereit/spidermonkey-wasi-embedding

    and besides that it is slower than nodejs it is still plenty fast (no matter that it is not as fast as they want) btw. it's startup is faster than node. (maybe better pgo might help)

  • GitHub repo spidermonkey-wasi-embedding

    btw. for what it's worth their javascript to wasm is opensource:

    - https://github.com/fastly/js-compute-runtime

    - https://github.com/tschneidereit/spidermonkey-wasi-embedding

    and besides that it is slower than nodejs it is still plenty fast (no matter that it is not as fast as they want) btw. it's startup is faster than node. (maybe better pgo might help)

  • OPS

    OPS - Build and Run Open Source Unikernels. Quickly and easily build and deploy open source unikernels in tens of seconds. Deploy in any language to any cloud.

  • GitHub repo wasmtime

    Standalone JIT-style runtime for WebAssembly, using Cranelift

    > I can't really square the idea that the 50-150ms time-delays in question comes down to the actual programming language, but it is absolutely believable that a longer test reduces the median latency rather than a high load test for a shorter duration.

    It seems plausible to me: in Fastly's case, they're using WebAssembly via wasmtime[1] which does support AOT compilation but most JavaScript code is dynamic enough that they still need a runtime JIT engine. I believe the the current approach Fastly is using is to compile Mozilla's SpiderMonkey JIT engine itself into WebAssembly and they've done some really nice work making that load as quickly as possible:

    https://bytecodealliance.org/articles/making-javascript-run-...

    The catch, of course, is that this still leaves a fair amount work which a JavaScript program has to deal with at runtime compared to a Rust program which the compiler can spend minutes optimizing long before deployment. This is a classic tradeoff for dynamic languages and a lot of people are satisfied with the approach of defaulting to faster developer turnaround and later converting hot spots to something like C or Rust, but I think it's definitely dodgy to use a single example of something you know to be this complex and present the results as generally representative of the entire platform.

    I have no knowledge of how Cloudflare ran their tests or reason to suspect malice but not disclosing test methodology and a “no benchmarks” license clause is going to make accusations about the results inevitable. Veterans of the benchmarketing wars have stories about comparisons where someone used a configuration option which happened to disable key performance optimizations for their competitors or used examples which favored their design decisions. Since nobody can read someone's mind to tell their real intent, the best way to avoid acrimony is to have full public disclosure of the benchmarks and their methodology — and to allow other people to conduct benchmarks so they can reproduce results or fill gaps.

    1. https://github.com/bytecodealliance/wasmtime

  • GitHub repo nativejson-benchmark

    C/C++ JSON parser/generator benchmark

    I like your ideas, but they seem difficult to enforce. It assumes good faith on all sides. One of the biggest complaints about AI/ML research results: It is frequently hard/impossible to replicate the results.

    One idea: The edge competitors can create a public (SourceHut?) project that runs various daily tests against themselves. This would similar to JSON library benchmarks. [1] Then allow each competitors to continuously tweak there settings to accomplish the task in the shortest amount of time.

    Also: It would be nice to see a cost analysis. For years, IBM's DB2 was insanely fast if you could afford to pay outrageous hardware, software license, and consulting costs. I'm not in the edge business, but I guess there are some operators where you can just pay a lot more and get better performance -- if you really need it.

    [1] https://github.com/miloyip/nativejson-benchmark

  • GitHub repo FrameworkBenchmarks

    Source for the TechEmpower Framework Benchmarks project

    > For such a small program ... the performance difference can be reduced significantly.

    I think you're wrong. Small programs don't necessarily minimize the difference between languages. Small programs often exacerbate them. Often for weird, idiosyncratic reasons. Especially if you're measuring time-to-first-byte, and not letting the VM warm up.

    For example, in this case I wouldn't be surprised if most of the CPU time was going into node + v8's startup, rather than executing the user's javascript at all.

    Look at the plaintext techempower benchmarks[1]. These benchmarks test tiny amounts of code - the programs only have to respond with "hello world" HTTP responses. The best performers are hitting hardware limits. If your theory that small program = small relative cost of using javascript was true, nodejs's performance should be similar to rust / C++. It is not - node is only 13% of the performance of the top performers. Weirdly "justjs" (another v8 based runtime) is ~5x faster than nodejs on this test. The reason probably has nothing to do with javascript at all, and is because justjs has less overhead and talks to the OS in a more efficient way. (It probably uses io_ring.)

    But maybe this is evidence you're technically correct. The performance differences can be reduced. But we have every reason to assume nodejs will have way more overhead than rust at executing the same code. So I agree with other posters. A benchmark showing rust on fastly is faster than javascript on cloudflare tells us nothing about the underlying hosting providers.

    https://www.techempower.com/benchmarks/#section=data-r20&hw=...

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts