remote-apis
pbrt-v3
remote-apis | pbrt-v3 | |
---|---|---|
5 | 17 | |
300 | 4,826 | |
0.0% | - | |
5.8 | 2.3 | |
15 days ago | 8 months ago | |
Starlark | C++ | |
Apache License 2.0 | BSD 2-clause "Simplified" License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
remote-apis
-
Mozilla sccache: cache with cloud storage
In the case of the Remote Execution/Cache API used by Bazel among others[1] at least, it's a bit more detailed. There's an "ActionCache" and an actual content-addressed cache that just stores blobs ("ContentAddressableStorage"). When you run a `gcc -O2 foo.c -o foo.o` command (locally or remotely; doesn't matter), you upload an "Action" into the action cache, which basically said "This command was run. As a result it had this stderr, stdout, error code, and these input files read and output files written." The input and output files are then referenced by the hash of their contents, in this case.
Most importantly you can look up an action in the ActionCache without actually running it. So now when another person comes by and runs the same build command, they say "Has this Action, with these inputs, been run before?" and the server can say "Yes, and the output is a file identified by hash XYZ" where XYZ is the hash of foo.o
So realistically you always some mix of "input content hashing" and "output content hashing" (the second being the definition of 'content addressable'.)
[1] https://github.com/bazelbuild/remote-apis/blob/main/build/ba...
-
Distcc: A fast, free distributed C/C++ compiler
Not only it's distributed like distcc, Bazel also provide sandboxing to ensure that environment factors does not affect the build/test results. This might not mean much for smaller use cases, but at scale with different compiler toolchains targeting different OS and CPU Architecture, the sandbox helps a ton in keeping your cache accurate.
On top of it, the APIs Bazel uses to communicate with the remote execution environment is standardized and adopted by other build tools with multiple server implementation to match it. Looking into https://github.com/bazelbuild/remote-apis/#clients, you could see big players are involved: Meta, Twitter, Chromium project, Bloomberg while there are commercial supports for some server implementations.
Finally, on top of C/C++, Bazel also supports these remote compilation / remote test execution for Go, Java, Rust, JS/TS etc... Which matters a lot for many enterprise users.
Disclaimer: I work for https://www.buildbuddy.io/ which provides one of the remote execution server implementation and I am a contributor to Bazel.
-
When to Use Bazel?
Regardless of whether you should use Bazel or not, my hope is that any future build systems attempt to adopt Bazel's remote execution protocol (or at least a protocol that is similar in spirit):
https://github.com/bazelbuild/remote-apis
In my opinion the protocol is fairly well designed.
-
Programming Breakthroughs We Need
> The thing I really would like to see is a smarter CI system. Caching of build outputs, so you don't have to rebuild the world from scratch every time. Distributed execution of tests and compilation, so you are not bottle-necked by one machine.
This is already achievable nowadays using Bazel (https://bazel.build) as a build system. It uses a gRPC based protocol for offloading/caching the actual build on a build cluster (https://github.com/bazelbuild/remote-apis). I am the author of one of the Open Source build cluster implementations (Buildbarn).
-
Distributed Cloud Builds for Everyone
Very nice! I really like the ease-of-use of this, as well as the scale-to-zero costs. That's a tricky thing to achieve. Seems like it could become a standard path to ease the migration from local to remote builds.
If the author is interested in standardizing the same, I'd suggest implementing the REAPI protocol (https://github.com/bazelbuild/remote-apis). It should be amenable to implementing on a Lambda-esque back-end, and is already standard amongst most tools doing Remote Execution (including Bazel! Bazel+llama could be fun). And equally, it's totally usable by a distcc-esque distribution tool (recc[1] is one example) - that's also what Android is doing before they finish migrating to Bazel ([2], sadly not yet oss'd).
The main interesting challenge I expect this project to hit is going to be worker-local caching: for compilation actions it's not too bad to skip assuming the compiler is built into the container environment, but if branching out into either hermetic toolchains or data-heavy action types (like linking), fetching all bytes to the ephemeral worker anew each time may prove to be prohibitive. On the other hand, that might be a nice transition point to switch to persistent workers: use a lambda backed solution for the scale-to-0 case, and switch execution stacks under the hood to something based on reused VMs when hitting sufficient scale that persistent executors start to win out.
(Disclaimer: I TL'd the creation of this API, and Google implementation of the same).
[1] https://gitlab.com/BuildGrid/recc
[2] https://opensource.googleblog.com/2020/11/welcome-android-op...
pbrt-v3
-
Ask HN: Any good books on ray tracing?
Physically Based Rendering[0] was an excellent textbook when I read it ages ago and conveniently enough it looks to have been updated with a new edition last year.
[0]: https://pbrt.org/
- Spectral Ray Tracing
-
Is it possible and realistic to learn independent of an API?
Physically Based Raytracing
-
C++ Project to Put On Resume
Both of these books are free, and both are written in C++, but they can be done in any language. The first book, a raytracer in a weekend, is part of a series, you can find it here: https://raytracing.github.io/ And, if you get to the third book in that series, or you need a reference book, the PBRT book covers the math in more depth and discusses the latest theory, you can get the last edition of the book (5 years out of date) for free though: https://pbrt.org/
-
(Why) is a toy password manager a too complex summer project?
Making a “complete” one is a never-ending rabbit hole you can spend a lifetime on and is a very active area of research covering more advanced geometry, probability, optics, machine learning etc etc. A great introduction to that is https://pbrt.org
-
Suggestions for some best books on computer vision
This isn't the highest priority but if you haven't already, learn how computer graphics works. Get a working knowledge of the camera matrix, real time graphics (say, OpenGL but threeJS is an option), and photorealistic graphics. PBRT is the go-to for photorealistic graphics. The first two books of Foundations of Game Engine Development are way more useful than they have any right to be (and my favorite textbooks I've ever read, 10/10).
-
Org Mode Gripes
Org-mode strength though is in working with different languages in a same source file, which I am not sure if Knuths version does. Anyway, to see how the original idea looks like, check the Wikipedia article, or to see it in real-life see some of books that are written in the literate style, like Physically Based Rendering, which seems to be available for free nowadays or C Interfaces and Implementations.
- Ask HN: What is the coding exercise you use to explore a new language?
- Path Tracer Project
-
Physically Based: A Database of PBR Values for Real-World Materials
I contributed a tiny bit to pbrt[1], and one of the things I loved was that if you just plugged in physical values you almost always got great results with minimal tweaking.
The Octane data seems most complete at first glance (with complex IOR etc), but for things like milk and blood I expected at the very least some absorption coefficient for the translucency or similar.
[1]: https://pbrt.org/
What are some alternatives?
dylint - Run Rust lints from dynamic libraries
the_raytracer_challenge_repl - A WebAssembly (WASM) based REPL interface for my Raytracer Challenge in Rust project
bazel-gba-example - Bazel GBA (Game Boy Advance) Example
mitsuba3 - Mitsuba 3: A Retargetable Forward and Inverse Renderer
llama
odin_rosettacode - Odin examples for Rosetta Code
MyDef - Programming in the next paradigm -- your way
RiftRay - Step into the worlds of Shadertoy with an Oculus Rift.
bazel-buildfarm - Bazel remote caching and execution service
tray_rust - A toy ray tracer in Rust
embedded-postgres-binaries - Lightweight bundles of PostgreSQL binaries with reduced size intended for testing purposes.
RustCrypto - Authenticated Encryption with Associated Data Algorithms: high-level encryption ciphers