utfcpp VS arocc

std::string is simply a string of bytes, so can already contain utf-8 encoded text. The only problem is when you want to interact with OS (Windows) API and other library APIs that don't expect utf-8 and when you need to count number of characters etc. For that you can look into existing libraries, e.g. the official Unicode ICU or whatever you can find that others have made, e.g.: https://github.com/nemtrif/utfcpp

Suffice it to say, this is a mess. However, there are libraries that make this easier.

> I've always tried as much as possible to treat strings as just opaque data and never look into them, which tends to work well, but in some domains you really need to look at and massage the characters/codepoints/grapheme clusters, and the lack of a first-citizen UTF-8-aware string type is, I think, a bit unfortunate in this day and age.

You don't need a UTF-8 type for that, you just need routines that handle UTF-8 strings, like utfcpp (https://github.com/nemtrif/utfcpp).

Possible reference as it requires to use the compiler as part of language abi: https://github.com/Vexu/arocc/issues/178 Not sure, where a better thread with explanations of the flaws is.

Zig calls clang to compile C code. This doesn't add a new dependency since Zig already depends on LLVM. In the future when Zig doesn't depend as much on LLVM, there might be a reason to use a C compiler written in Zig (e.g. https://github.com/Vexu/arocc)

Bit field rules are underspecified or plain wrongly implemented, because in their edge cases clang and GCC differ in semantics. See https://github.com/Vexu/arocc/issues/178 This should be further restricted with static asserts as compiler semantics even changed with versions and doing this manually/doing code review is error prone.

You might want to contribute or look into https://github.com/Vexu/arocc, which is planned to be eventually an alternative frontend. Is arocc able to handle your use cases?

> Does this mean that y'all are open to the self-hosted compiler supporting CPU architectures unlikely to ever have LLVM support?

Yes! We won't block 1.0 on the quality of the less mainstream targets, but that's what the tier system is for - to ship a compiler that has varying levels of quality for various targets, while communicating clearly to users what kind of experience they can expect for each one.

SuperH patches are absolutely welcome.

> how is zig cc anticipated to work with a self-hosted Zig? Will there be a dependency on clang [...]?

The main distribution of Zig will be LLVM/Clang-enabled. However it is already possible to build a version of Zig that does not have these features enabled. In such case, compiling C, C++, and Objective-C code will result in an error.

However, the arocc project[1] is emerging, which, depending on a combination of how much funding ZSF gets and how much enthusiasm the unpaid contributors working in their spare time have, is looking like a promising C frontend that would be available even without LLVM/Clang. It is C only, however, with no intention of compiling C++ or Objective-C.

> would zig cc support the planned C backend?

As it is currently implemented: no. Zig invokes clang to turn C source code into object files.

However, with the arocc frontend above, this would be converting the C source code into ZIR (or perhaps AIR), which could then be lowered with any of the backends, including the (partially complete) C backend. In such case, the C output would look drastically different than the input. It would look more like an IR than natural C code that a human would write.

[1]: https://github.com/Vexu/arocc

Implementing only the language part takes like 10k LOC.

> 9. Without a C compiler, we're stuck with, wedded to, and beholden to libclang.

> I wouldn't be surprised that the eventual cost of adapting ourselves to

> libclang will exceed the cost of doing our own C compiler.

This is a really insightful point. I had to learn this the hard way :)

We might follow your lead on this, as we have done with so many other great ideas implemented in D.

Ironically, Vexu started from the other side as you, with the preprocessor mostly done, but the backend left to-do: https://github.com/Vexu/arocc

One thing that might make libclang worth the cost, however, is its ability to compile C++ code as well. On Zig's end of things, all we have to do is provide libcxx, libcxxabi, libunwind, compiler-rt, and linking, and then libclang is really pulling a lot of weight by compiling C++ code into object files. Sadly this ability is just too useful in practice to ignore. For example, LLVM itself is C++ so if Zig wants to be able to bootstrap itself, it needs this capability.

Still, I think your maneuver here is the best long-term approach to tackle this problem, and I imagine as time goes on we'll start to migrate towards D's solution here. Maybe someday the Zig distribution that does not have LLVM extensions enabled will be the more popular one!

I'll be watching the evolution of this new feature in D with great interest!