Cargo VS rustc_codegen_cranelift

https://www.rust-lang.org/ https://doc.rust-lang.org/cargo/ https://doc.rust-lang.org/cargo/getting-started/installation.html

cargo, rustfmt, clippy, rust-analyzer, and Rust’s robust unit testing capabilities together form a powerful ecosystem for managing large-scale projects like Spin.

That work is being tracked in https://github.com/rust-lang/cargo/issues/5931

Someone has taken up the work on this though there are some foundational steps first.

1. We need to delineate intermediate and final build artifacts so people have a clearer understanding in `target/` what has stability guarantees (implemented, awaiting stabilization).

2. We then need to re-organize the target directory from being organized by file type to being organized by crate instance.

3. We need to re-do the file locking for `target/` so when we share things, one cargo process won't lock out your entire system

4. We can then start exploring moving intermediate artifacts into a central location.

There are some caveats to this initial implementation

- To avoid cache poisoning, this will only items with immutable source that and an idempotent build, leaving out your local source and stuff that depends on build scripts and proc-macros. There is work to reduce the reliance on build scripts and proc-macros. We may also need a "trust me, this is idempotent" flag for some remaining cases.

- A new instance of a crate will be created in the cache if any dependency changes versions, reducing reuse. This becomes worse when foundation crates release frequently and when adding or updating a specific dependency, Cargo prefers to keep all existing versions, creating a very unpredictable dependency tree. Support for remote caches, especially if you can use your project's CI as a cache source, would help a lot with this.

> It seems wild to consider such intermediate files as part of public API. Someone relying on it does not automatically make it a breaking change if it’s not documented.

To find what is considered an intermediate vs a final artifact from cargo, you need to check out https://doc.rust-lang.org/cargo/reference/build-cache.html

We are working on making this clearer with https://github.com/rust-lang/cargo/issues/14125 where there will be `build.build-dir` (intermediate files) and `build.target-dir` (final artifacts).

When you do a `cargo build` inside of a library, like `clap`, you will get an rlip copied into `build.target-dir` (final artifacts). This is intended for integration with other build systems. There are holes with this workflow though but identifying all of the relevant cases for what might be a "safe" breakage is difficult.

See https://github.com/rust-lang/cargo/issues/13897 and https://github.com/rust-lang/cargo/issues/13897#issuecomment... .

Powerful tooling: Cargo simplifies dependency management, builds, and testing.

What I mean is that in open source, things only get done by people motivated to do them. Nobody has ever even asked for darcs support: https://github.com/rust-lang/cargo/issues?q=is%3Aissue%20sta...

So the lack of darcs isn’t because the Cargo folks think it’s bad or something. Just that things don’t get added just because.

Re quirks, sure, that’s why rustc and cargo are different. You don’t have to use Cargo. Meta does not, the Linux kernel does not.

One of the reasons why user PRs are so frequent is that Rust itself is a pretty straightforward language to contribute to. While the language itself is on the complex side (to say the least), its strict compiler and single package manager make it relatively manageable to run and test any changes and to be confident that what you've submitted will work as expected.

> That’s because, while cargo is great at building things, it is very simplistic at installing them. Cargo wants everything in a few neat binaries, and that isn’t our use case. Fish has about 1200 .fish scripts (961 completions, 217 associated functions), as well as about 130 pages of documentation (as html and man pages), and the web-config tool and the man page generator (both written in python).

Our issue for this is https://github.com/rust-lang/cargo/issues/2729

Personally, I lean away from Cargo expanding into these use cases and prefer another tool being implemented on top. I've written more about this at https://epage.github.io/blog/2023/08/are-we-gui-build-yet/

> I wonder if how much value there is in skipping LLVM in favor of having a JIT optimized linked in instead. For release builds it would get you a reasonable proxy if it optimized decently while still retaining better debugability.

Rust is in the process of building out the cranelift backend. Cranelift was originally built to be a JIT compiler. The hope is that this can become the debug build compiler.

https://github.com/rust-lang/rustc_codegen_cranelift

There's no reason? Are you sure about this?

I think you mean there could theoretically be an interpreted Rust, but I don't think anyone has ever made a prototype of a Rust interpreter.

The closest is probably rust-analyzer (the official language server), that maintains internal state and reacts to changes you make, but it doesn't create an executable artifact.

The other is probably the Cranelift Backend (https://github.com/rust-lang/rustc_codegen_cranelift), which can produce debug builds quickly.

Windows is supported. See https://github.com/rust-lang/rustc_codegen_cranelift/issues/....

> When this happens, it seems like it'll be possible to get the LLVM bits out of the bootstrap process and lead to a fully self-hosted Rust.

What do you mean by "when this happens"? GP's point is that this has already happened: the Cranelift backend is feature-complete from the perspective of the language [0], except for inline assembly and unwinding on panic. It was merged into the upstream compiler in 2020 [1], and a compiler built with only the Cranelift backend is perfectly capable of building another compiler. LLVM hasn't been a necessary component of the Rust compiler for quite some time.

[0] https://github.com/bjorn3/rustc_codegen_cranelift

[1] https://github.com/rust-lang/rust/pull/77975

Note that the Cranelift codegen will eventually become standard for debug builds to speed them up.

Maybe one day we'll have rustc b3 backend like what they did with Cranelift

Additionally, there is gcc codegen for rustc (https://github.com/rust-lang/rustc_codegen_gcc), which is not a compiler per se, but an alternative code generator, with more architectures supported and other nice things. It's also coming along, but there's still a lot of work to do there too. There's also Cranelift codegen (https://github.com/bjorn3/rustc_codegen_cranelift), which is designed to make debug builds faster, but this is not as exciting/useful as the other 2.

For the instant reloading, that's in Sycamore, so you should speak to its devs, but as for the alternative compiler backend, it's not my project, but it uses Cranelift and works pretty well! See https://github.com/bjorn3/rustc_codegen_cranelift for details.