macschema VS qmux

With the macschema toolchain, you can generate API schemas about any Apple framework, class, function, etc based on their documentation and header declarations. This is useful for us in generating framework bindings in macdriver, but it would be useful for any project doing similar work, such as bindings for other languages. This sort of approach will play a big part in how Tractor will integrate with "whatever we want" down the line.

If this is interesting, take a look at the tool I built, macschema, which is open source on GitHub. Right now it's just the toolchain, but in theory we could generate schemas for all of Apple's APIs and put them in a repo in such a way that can be updated by a combination of macschema and user contributed patches. I'll let somebody else do that though.

The connection multiplexing layer is based on qmux, a subset of SSH that I've written about previously. It was designed to optionally be swapped out with QUIC as needed. Either way, everything in qtalk happens over flow-controlled channels, which can be used like embedded TCP streams. Whatever you do with qtalk, you can also tunnel other connections and protocols on the same connection.

Unrelated to macdriver, I released a protocol called qmux with a post explaining why this (but really any) muxing protocol, including and especially QUIC, is such a great primitive for network programming. The project came with two implementations, Go and TypeScript, with more on the way. The post about it was the first explainer article I'd done in a while and people liked it. This was important because the idea behind it was really more valuable than the protocol itself, but regardless we still need implementations of it to exist. To show the idea in action, I built a 130 line version of Ngrok, which turned into a great post walking through how it works.

What qmux does is give us a subset of the SSH protocol to multiplex many connections over a single connection. It was the missing piece of my original Twisted prototype. You can read more about how there aren't a lot of these protocols (but perhaps with QUIC are also the future of the Internet) in my previous post.

If nothing else, I hope people use this story as inspiration to think about using multiplexing to design better, simpler application protocols. Even if for some reason QUIC isn't the future, you now have qmux!