remote-apis VS jscodeshift

In the case of the Remote Execution/Cache API used by Bazel among others[1] at least, it's a bit more detailed. There's an "ActionCache" and an actual content-addressed cache that just stores blobs ("ContentAddressableStorage"). When you run a `gcc -O2 foo.c -o foo.o` command (locally or remotely; doesn't matter), you upload an "Action" into the action cache, which basically said "This command was run. As a result it had this stderr, stdout, error code, and these input files read and output files written." The input and output files are then referenced by the hash of their contents, in this case.

Most importantly you can look up an action in the ActionCache without actually running it. So now when another person comes by and runs the same build command, they say "Has this Action, with these inputs, been run before?" and the server can say "Yes, and the output is a file identified by hash XYZ" where XYZ is the hash of foo.o

So realistically you always some mix of "input content hashing" and "output content hashing" (the second being the definition of 'content addressable'.)

[1] https://github.com/bazelbuild/remote-apis/blob/main/build/ba...

Not only it's distributed like distcc, Bazel also provide sandboxing to ensure that environment factors does not affect the build/test results. This might not mean much for smaller use cases, but at scale with different compiler toolchains targeting different OS and CPU Architecture, the sandbox helps a ton in keeping your cache accurate.

On top of it, the APIs Bazel uses to communicate with the remote execution environment is standardized and adopted by other build tools with multiple server implementation to match it. Looking into https://github.com/bazelbuild/remote-apis/#clients, you could see big players are involved: Meta, Twitter, Chromium project, Bloomberg while there are commercial supports for some server implementations.

Finally, on top of C/C++, Bazel also supports these remote compilation / remote test execution for Go, Java, Rust, JS/TS etc... Which matters a lot for many enterprise users.

Disclaimer: I work for https://www.buildbuddy.io/ which provides one of the remote execution server implementation and I am a contributor to Bazel.

Regardless of whether you should use Bazel or not, my hope is that any future build systems attempt to adopt Bazel's remote execution protocol (or at least a protocol that is similar in spirit):

https://github.com/bazelbuild/remote-apis

In my opinion the protocol is fairly well designed.

> The thing I really would like to see is a smarter CI system. Caching of build outputs, so you don't have to rebuild the world from scratch every time. Distributed execution of tests and compilation, so you are not bottle-necked by one machine.

This is already achievable nowadays using Bazel (https://bazel.build) as a build system. It uses a gRPC based protocol for offloading/caching the actual build on a build cluster (https://github.com/bazelbuild/remote-apis). I am the author of one of the Open Source build cluster implementations (Buildbarn).

Very nice! I really like the ease-of-use of this, as well as the scale-to-zero costs. That's a tricky thing to achieve. Seems like it could become a standard path to ease the migration from local to remote builds.

If the author is interested in standardizing the same, I'd suggest implementing the REAPI protocol (https://github.com/bazelbuild/remote-apis). It should be amenable to implementing on a Lambda-esque back-end, and is already standard amongst most tools doing Remote Execution (including Bazel! Bazel+llama could be fun). And equally, it's totally usable by a distcc-esque distribution tool (recc[1] is one example) - that's also what Android is doing before they finish migrating to Bazel ([2], sadly not yet oss'd).

The main interesting challenge I expect this project to hit is going to be worker-local caching: for compilation actions it's not too bad to skip assuming the compiler is built into the container environment, but if branching out into either hermetic toolchains or data-heavy action types (like linking), fetching all bytes to the ephemeral worker anew each time may prove to be prohibitive. On the other hand, that might be a nice transition point to switch to persistent workers: use a lambda backed solution for the scale-to-0 case, and switch execution stacks under the hood to something based on reused VMs when hitting sufficient scale that persistent executors start to win out.

(Disclaimer: I TL'd the creation of this API, and Google implementation of the same).

[1] https://gitlab.com/BuildGrid/recc

[2] https://opensource.googleblog.com/2020/11/welcome-android-op...

A parser can have various applications in everyday life, and you probably use some parser daily. Babel, webpack, eslint, prettier, and jscodeshift. All of them, behind the scenes, run a parser that manipulates an Abstract Syntax Tree (AST) to do what you need - we'll talk about that later, don't worry.

OP: If this is an avenue you feel like entertaining, here are some nice codemod tools that could ease the transition for you: CodeQue, Subsecond, and the old standard jscodeshift.

If that isn't enough or you find issues with it, the current de-facto standard for codemods is jscodeshift. You'll write more code, but at least you only have to write it once.

Here is the pull request https://github.com/facebook/jscodeshift/pull/549

Those are important but tough problems!

1. I feel like codemods and a setup shell prompt are the best solution to this, but this looks different for each language (eg, in JS there is https://github.com/facebook/jscodeshift). I could make UI around that for listers to provide parts of the app that should replaced and buyers to provide values for it, but it could be quite a rabbit hole. This is actually why I show the README for the repos, so users can see if the setup steps are comprehensive before buying (for paid ones).

2. I agree atomic commits can help someone navigate a new codebase. In the same spirit as my response to #1, I don't feel confident I could enforce this, but I could surface the commits on the starter repo page so potential buyers could see if the author laid things out well.

Thanks for the ideas!

That's why I've been working on a tool that automatically upgrades major versions of libraries with breaking changes, the idea is to simplify the process and save developers time and effort by having a bank of transformers (using codemod & jscodeshift) and open source them:

JSCodeshift: A toolkit for running and writing codemods.

Hi HN community, we are Jake, Tzachi and Etai, co-founders of FlyCode (https://www.flycode.com/). FlyCode makes it easy for product, UX, and marketing teams to edit web apps without coding, so they don’t have to wait on (or consume) developer time, and can iterate, test, and release faster. See a quick example here: https://www.youtube.com/watch?v=jDL5oa2nEHo

Non-technical teams frequently need to edit the copy (text), images, and links that appear in a web app. How to manage these has long been a pain on software projects. You can keep them separate from the code, in some form that non-programmers can edit, but this adds a lot of complexity and is usually brittle, as it can bypass the regular development workflows (as CI, staging envs & deploy previews). It’s simpler to keep them in the code—but then only programmers can easily edit them. Everyone else has to wait to get their changes in, plus the devs have to do a lot of edits that aren’t their main work. This slows projects down and is expensive. It also means that product/marketing/UX teams can’t do things that require rapid iteration, such as sophisticated forms of A/B or usability testing. This limits their work and ultimately is bad for both quality and revenue.

There have been many approaches to solving this dilemma, including custom built admin tools that are limited in functionality and require maintenance, offloading to CMS that require heavy integration, are normally used for simple static apps, and bind your stack to their SDKs. Or wasting a developer’s time to do it for you…

We took a new approach by automatically analyzing a codebase’s structure, similar to a compiler. This allows us to automatically populate our platform in which product/UX/marketing teams can easily use to edit their text and images. We programmatically turn those edits into code changes. Our GitHub bot then takes these code changes and creates a pull request just like a developer would—but without the latency (and boredom!). Developers retain codebase ownership, while non-developers become individual contributors to the dev process, just like the others.

We use well-established practices for parsing and editing source code (like https://github.com/facebook/jscodeshift), covering most of the major technologies used for building web apps (React, Angular, Vue, and Ruby on Rails included).

Once our software has parsed your codebase, it generates an editing portal for your app that teams can easily use to find, manage, and edit product copy, images, and links, and then auto-generate PRs. You can edit product copy regardless of whether it is in resource files or hardcoded (fun fact: some of the largest and fastest-growing tech companies have most of their strings hardcoded!), and you can replace and upload new images and icons to your product.

The integration with GitHub (https://www.flycode.com/developers) took us a long time to get right. There’s not a lot of documentation around integrating GitHub to platforms, and things like connecting an org or connection requests turned out to be non-trivial. We're particularly proud of the result because unlike with other tools, you don’t have to do any significant integration work.

Our GitHub app finds texts and images in the source code and sends them to our platform (you have full control of what and where we scan). Once a user requests a change it updates the texts and the images in the codebase and creates a pull request.

We did a Show HN earlier this year: https://news.ycombinator.com/item?id=31166924, which helped us get some serious leads, which was awesome. Since then we’ve moved out of beta, added new content types (images), launched a new UI and visual editor (EAP), and automated the onboarding of new repos.

We have a handful of companies paying for this and spent the last year focusing on making it extremely simple to use. It only takes 3 minutes to connect our GitHub app and configure the system for your team to start editing. It doesn’t require any changes to your code, or any special maintenance. You can get started here: https://app.flycode.com

We are hoping to use this launch to get some more feedback from you all! We are far from our vision to be a platform for everything front-end but are working hard every day to improve the user experience and feature requests from our early collaborators (editing links, themes, variables, JSON configuration, defining in-code A/B tests, etc.).

We're really happy to show this to you all and thank you for reading about it. For those that sign up, time yourself to check that our “3-minute connect + config” claim isn't just a sales tactic! We look forward to further conversation in the comments.

jscodeshift