foam3 VS remote-apis

Thank you for sharing that video! Your foam project looks fascinating too: https://github.com/kgrgreer/foam3

I wrote a BASIC to JS compiler/transpiler that includes all of the programs from "BASIC Computer Games" as examples.

You can try it out in your browser at: https://codepen.io/kgr/full/yLQyLjR

Just select the game you want to to run from the top-left list box, then press the "Compile" button and you'll see the translated JS source in the right text-area. Then press the "Run" button to run it.

The source code for the compiler is available at: https://github.com/kgrgreer/foam3/tree/429f2fd2b4cef0e37996a...

Sort of related but an interesting idea is take it one step further with the Feature Oriented Active Modeler (FOAM) [1,2] paradigm and use code to model your whole system, which generates diagrams model, and runnable code in whatever language needed. The project is still young and it may not be practical today with the available tooling but it seems like a cool idea and project. It is influenced by the unix principle of “coding the perimeter not the area” which is essentially factoring your dev tasks into building NM capabilities, but instead of building NM things individually build N+M tools that can be composed into N*M capabilities [2].

So with FOAM the idea is if we want to maintain a model of our software, and build it as well, what if we can use one composable tool to generate both, rather than model everything and code it separately.

[1] https://github.com/kgrgreer/foam3

[2] https://foam-framework.github.io/foam/

[3] https://m.youtube.com/watch?v=3Ea3pkTCYx4

https://github.com/kgrgreer/foam3#videos

FOAM is a modelling framework that generates cross-language boilerplate for you, but it takes a much broader view of what constitutes boilerplate than most systems. Typically, it can generate between 95-98% of a working cross-language cross-tier system.

FOAM helps you create features for modelled data. Features include things like a Java/Javascript/Swift classes to hold your modelled data, code to marshall to/from JSON/XML/CSV/etc., various GUI Views, and support for storing your data in various databases or file formats. However, FOAM models are themselves modelled, meaning they're afforded all of the above benefits as well. This lets you apply the MVC technique of having multiple views work against the same underlying data-model concurrently (say a grid and a pie-chart in a spreadsheet), so that you can choose the best view or views for your current need. When treated this way, your code is no longer text (but it can be, if that's one of your views), and you can easily view and store it in many different ways and more easily programmatically manipulate it.

In the case of the Remote Execution/Cache API used by Bazel among others[1] at least, it's a bit more detailed. There's an "ActionCache" and an actual content-addressed cache that just stores blobs ("ContentAddressableStorage"). When you run a `gcc -O2 foo.c -o foo.o` command (locally or remotely; doesn't matter), you upload an "Action" into the action cache, which basically said "This command was run. As a result it had this stderr, stdout, error code, and these input files read and output files written." The input and output files are then referenced by the hash of their contents, in this case.

Most importantly you can look up an action in the ActionCache without actually running it. So now when another person comes by and runs the same build command, they say "Has this Action, with these inputs, been run before?" and the server can say "Yes, and the output is a file identified by hash XYZ" where XYZ is the hash of foo.o

So realistically you always some mix of "input content hashing" and "output content hashing" (the second being the definition of 'content addressable'.)

[1] https://github.com/bazelbuild/remote-apis/blob/main/build/ba...

Not only it's distributed like distcc, Bazel also provide sandboxing to ensure that environment factors does not affect the build/test results. This might not mean much for smaller use cases, but at scale with different compiler toolchains targeting different OS and CPU Architecture, the sandbox helps a ton in keeping your cache accurate.

On top of it, the APIs Bazel uses to communicate with the remote execution environment is standardized and adopted by other build tools with multiple server implementation to match it. Looking into https://github.com/bazelbuild/remote-apis/#clients, you could see big players are involved: Meta, Twitter, Chromium project, Bloomberg while there are commercial supports for some server implementations.

Finally, on top of C/C++, Bazel also supports these remote compilation / remote test execution for Go, Java, Rust, JS/TS etc... Which matters a lot for many enterprise users.

Disclaimer: I work for https://www.buildbuddy.io/ which provides one of the remote execution server implementation and I am a contributor to Bazel.

Regardless of whether you should use Bazel or not, my hope is that any future build systems attempt to adopt Bazel's remote execution protocol (or at least a protocol that is similar in spirit):

https://github.com/bazelbuild/remote-apis

In my opinion the protocol is fairly well designed.

> The thing I really would like to see is a smarter CI system. Caching of build outputs, so you don't have to rebuild the world from scratch every time. Distributed execution of tests and compilation, so you are not bottle-necked by one machine.

This is already achievable nowadays using Bazel (https://bazel.build) as a build system. It uses a gRPC based protocol for offloading/caching the actual build on a build cluster (https://github.com/bazelbuild/remote-apis). I am the author of one of the Open Source build cluster implementations (Buildbarn).

Very nice! I really like the ease-of-use of this, as well as the scale-to-zero costs. That's a tricky thing to achieve. Seems like it could become a standard path to ease the migration from local to remote builds.

If the author is interested in standardizing the same, I'd suggest implementing the REAPI protocol (https://github.com/bazelbuild/remote-apis). It should be amenable to implementing on a Lambda-esque back-end, and is already standard amongst most tools doing Remote Execution (including Bazel! Bazel+llama could be fun). And equally, it's totally usable by a distcc-esque distribution tool (recc[1] is one example) - that's also what Android is doing before they finish migrating to Bazel ([2], sadly not yet oss'd).

The main interesting challenge I expect this project to hit is going to be worker-local caching: for compilation actions it's not too bad to skip assuming the compiler is built into the container environment, but if branching out into either hermetic toolchains or data-heavy action types (like linking), fetching all bytes to the ephemeral worker anew each time may prove to be prohibitive. On the other hand, that might be a nice transition point to switch to persistent workers: use a lambda backed solution for the scale-to-0 case, and switch execution stacks under the hood to something based on reused VMs when hitting sufficient scale that persistent executors start to win out.

(Disclaimer: I TL'd the creation of this API, and Google implementation of the same).

[1] https://gitlab.com/BuildGrid/recc

[2] https://opensource.googleblog.com/2020/11/welcome-android-op...