ferros VS pprof

My company, https://www.auxon.io. We created https://github.com/auxoncorp/ferros originally to enable a customer project early in the company's life cycle.

Some time later we had another customer interested in using it and having us add some features to it (e.g. some device drivers and a persistence layer utilizing https://docs.rs/tickv/latest/tickv/). It was becoming a massive pain in the neck to work out source code sharing agreements with them, so we decided to just open source it.

There are quite a number of things that we would do differently if we had to build it again, and at some point will likely do that work to revise it. The biggest one of those is root task synthesis. The other is to build and bring in facilities for running tasks that are compiled to WASM.

When we add WASM support to https://github.com/auxoncorp/ferros it'll sorta be like what you're angling at there in your description.

We've built things on seL4 (https://github.com/auxoncorp/ferros). We like to joke that it's the most perfect piece of nearly featureless software ever made.

There's... A LOT... of work to do before seL4 is going to be anywhere near usability parity with something like Linux, unfortunately.

Rather than make a general purpose OS, we decided to use it more like a unikernel or "library OS" where you're trying to make a well defined kind of "appliance" image to deploy to specific hardware rather than try to fake being a POSIX-y shaped OS.

For what it's worth, here's FerrOS's repo as well as the underlying selfe repo

That's basically what we did with https://github.com/auxoncorp/ferros, Bundle Rust programs together as tasks to run atop the formally verified seL4 microkernel.

We also built a Rust framework called FerrOS (https://github.com/auxoncorp/ferros) atop the formally-verified seL4 microkernel.

It has a similar set of usage idioms to Hubris it looks like in terms of trying to setup as much as possible ahead of time to assemble what's kind of an application specific operating system where everything your use case needs is assembled at build-time as a bunch of communicating tasks running on seL4.

We recently added a concise little persistence interface that pulls in TicKV (https://docs.tockos.org/tickv/index.html) from the Tock project you referenced above, and some provisions are being added for some more dynamic task handling based on some asks from an automotive OEM.

We built a thing to enable combining Rust applications together to be hosted on the seL4 microkernel. The developer experience is more akin to that of something like an RTOS where the OS and your applications are built and deployed together. The whole premise of it is decidedly non-POSIX-like. The current point is for assembling software for use-case-specific/appliance computing, not general purpose computing. (https://github.com/auxoncorp/ferros)

We're looking both for contributors and also actively hiring for a couple engineering positions for the above and for or mainline product.

I tried and failed to bring unikernels to my former work when I was at Visa. Specifically, LING.

At my current company, Auxon, we recently open sourced[1] some work we did a couple years back which is more or less an attempt at the basic foundations for blending the seL4 microkernel with fairly normal no_std Rust application development and assembling them all together to make a purpose built OS/application to deploy directly to hardware or within a VM. We have some work to do to keep building it up as a foundation for broader use, but we're looking into partnering with the seL4 Foundation (now under the Linux Foundation) to iterate on it further with some of our other mutual partners. The developer experience is much closer to that of developing for an RTOS than it is like typical general purpose computing development.

I'm of course biased, but I think there's a lot of room to innovate in the space of use case specific software stacks where the domain and constraints are well understood and too many degrees of freedom are actually a hindrance and a liability, not an advantage.

[1] https://github.com/auxoncorp/ferros

On top of that is ferros (no relation to to ferrous-systems), a higher-level userland of unreasonably strong types for compile-time resource tracking. No more discovering you need more memory, or capability slots or IPC rights at runtime. These types help you fit the right seL4 screw to the right seL4 screwdriver.

The pprof format is not tied to Go. From my understanding, it's used within Google across multiple languages. The format is defined in the pprof repository[0], and the visualization tool is source-language agnostic. I've seen libraries in numerous languages (e.g. Python, Java) to publish profiles in pprof format. This is an indicator the pprof format has become de-facto. Grafana Pyroscope[1] is a tool that's capable of parsing the pprof format, agnostic to the source programming language, and has instructions for Go, Java, Python, Ruby, node.js, Rust, and .NET.

My understanding is that you're searching for a combination of the profiles, metrics, and tracing. Caddy supports all 3.

[0] https://github.com/google/pprof/blob/main/doc/README.md

[1] https://grafana.com/docs/pyroscope/latest/

metrics and tracing need to be manually enabled (for now, perhaps)

Because we couldn't identify the issue using the results we got from Callgrind, we reached for another profiler, gperftools. It's a sampling profiler and therefor it has a smaller impact on the application's performance in exchange for less accurate call statistics. After filtering out the unimportant parts and visualizing the rest with pprof, it was evident that something strange was happening with the send function. It took only 71 milliseconds with the previous implementation and more than 900 milliseconds with the new implementation of our Bolt server. It was very suspicious, but based on Callgrind, its cost was almost the same as before. We were confused as the two results seemed to conflict with each other.

github.com - google/pprof

The application performances in a specific code path (e.g. gdb, pprof, …).

pprof is https://github.com/google/pprof, it's a very useful tool in golang , and really really really convenient

Go also has pretty good out of the box profiling (pprof[0]) and third-party runtime debugging (delv[1]) that can be used both remotely and local.

These tools also have decent editor integration and can be use hand in hand:

https://blog.jetbrains.com/go/2019/04/03/profiling-go-applic...

https://blog.jetbrains.com/go/2020/03/03/how-to-find-gorouti...

[0] https://github.com/google/pprof

[1] https://github.com/go-delve/delve

And what we do in Google Cloud is that we still use the pprof. But it's a kind of forked version of the pprof because the visualization part is totally different. So we give that tool as the Cloud Profiler. So that is the product name. And then, the difference between the pprof and a Cloud Profiler is that Cloud Profiler provides the agent library for each famous programming language such as Java, Python, Node.js, and Go. And then what you need to do is to just write 5 to 10 lines of code in a new application. That launches the profile agent in your application as a subsidiary thread of the main thread. And then, that thread periodically collects the profile data of the application and then sends that data back to Google Cloud and the Cloud Profiler.

gprftools (https://github.com/gperftools/gperftools) can be easily plugged in using LD_PRELOAD and signal, and has nice go implemented visualization tool https://github.com/google/pprof.