glow VS egg

Pytorch supports other kinds of accelerators (e.g. FPGA, and https://github.com/pytorch/glow), but unless you want to become a ML systems engineer and have money and time to throw away, or a business case to fund it, it is not worth it. In general, both pytorch and tensorflow have hardware abstractions that will compile down to device code. (XLA, https://github.com/pytorch/xla, https://github.com/pytorch/glow). TPUs and GPUs have very different strengths; so getting top performance requires a lot of manual optimizations. Considering the the cost of training LLM, it is time well spent.

It's pretty clear its referring to the output of Apache TVM and Meta's Glow

I also disagree with this. For example, Meta seems desperate about AI accelerators, and in fact is already doing "hardware customers develop software stack themselves" I mentioned above: Glow is that stack. Meta is doing Glow even if there is no promising AI accelerators right now, they are that desperate.

For reference: In Tensorflow and JAX, for example, the tensor gets compiled to the intermediate XLA format (https://www.tensorflow.org/xla), then passed to the XLA complier (https://github.com/tensorflow/tensorflow/tree/master/tensorflow/compiler/xla/service) or the new TFRT runtime (https://github.com/tensorflow/runtime/blob/master/documents/tfrt_host_runtime_design.md), or some more esoteric hardware (https://github.com/pytorch/glow).

The main reasons are hiring, and depth and breadth of the product.

Compilers are hard, device support is hard, the compiler community is small and closed source compilers quickly become weird tech islands.

https://github.com/pytorch/glow

Maybe program optimization?

https://egraphs-good.github.io/

For semantic analyzers, check out egg and egglog. They're custom data structures for representing compiler rewrite rules in a non-destructive way.

E-graphs are pretty awesome, and worth keeping in your back pocket. They're like union-find structures, except they also maintain congruence relations (i.e. if `x` and `y` are in the same set, then `f(x)` and `f(y)` must likewise be in the same set).

https://egraphs-good.github.io/

(Incidentally, union-find structures are also great to know about. But they're not exactly "new".)

I would add that Equality saturation/E-graphs has become quite a hot topic recently, since their POPL21 paper, with workshops dedicated to applications of e-graphs. They have even recently been added to Cranelift as an IR for optimizations.

Egraphs solve the rewrite ordering problem quite nicely. https://egraphs-good.github.io/

Note that one solution to this problem is to use equality saturation (which, coincidentally, has a great implementation in rust!).

Have you thought about trying to parallelize e-graphs? This way you can do a bunch of rewrite rules in parallel and then extract your desired graph at the end instead of having conflicts.

I think the equality saturation papers are a good start. A good start is egg. They have a presentation, a research paper and code you can play with. I think ultimately you want to translate arithmetic operations into logical operation that can be understood by the fpga. So I think it would be good to research how adders and multipliers are implemented in logic and ultimately include equalities between adders/multipliers with their logical counterpart. Note the this translation also depends on the representations of your numbers and their bit width.

For rewriting, you may also find interesing equality saturation: https://egraphs-good.github.io/