Roslyn VS equinox

.NET is a little smarter about switch code generation these days: https://github.com/dotnet/roslyn/pull/66081

Recently, while creating some experimental C# source code generators (xafero/csharp-generators), I was just concatenating strings together. Like you do, you know, if things have to go very quickly. If you have a simple use case, use a formatted multi-line string or some template library like scriban. But I searched for a way to generate more and more complicated logic easily - like for example, adding raw SQL handler methods to my pre-generated DBSet-like classes for my ADO.NET experiment. You could now say: Use Roslyn and that's really fine if you look everything up in a website like SharpLab, which shows immediately the syntax tree of our C# code.

At runtime, the Case C# expressions are embedded into a function and dynamically compiled into an assembly using the Roslyn C# compiler. Then the function that contains the expression is called (e.g. CaseAvailableFunction.Availablle()). At runtime, the function provides various methods to access stored case values as well as the current input data.

I wrote a JAX-based neural network library (Equinox [1]) and numerical differential equation solving library (Diffrax [2]).

At the time I was just exploring some new research ideas in numerics -- and frankly, procrastinating from writing up my PhD thesis!

But then one of the teams at Google starting using them, so they offered me a job to keep developing them for their needs. Plus I'd get to work in biotech, which was a big interest of mine. This was a clear dream job offer, so I accepted.

Since then both have grown steadily in popularity (~2.6k GitHub stars) and now see pretty widespread use! I've since started writing several other JAX libraries and we now have a bit of an ecosystem going.

[1] https://github.com/patrick-kidger/equinox

The elevator pitch is Optimistix is really fast, especially to compile. It plays nicely with Optax for first-order gradient-based methods, and takes a lot of design inspiration from Equinox, representing the state of all the solvers as standard JAX PyTrees.

If you like PyTorch then you might like Equinox, by the way. (https://github.com/patrick-kidger/equinox ; 1.4k GitHub stars now!)

You're thinking of `jax.closure_convert`. :)

(Although technically that works by tracing and extracting all constants from the jaxpr, rather than introspecting the function's closure cells -- it sounds like your trick is the latter.)

When you discuss dynamic allocation, I'm guessing you're mainly referring to not being able to backprop through `jax.lax.while_loop`. If so, you might find `equinox.internal.while_loop` interesting, which is an unbounded while loop that you can backprop through! The secret sauce is to use a treeverse-style checkpointing scheme.

https://github.com/patrick-kidger/equinox/blob/f95a8ba13fb35...

I'm a big fan of using ABCs to declare interfaces -- so much so that I have an improved abc.ABCMeta that also handles abstract instance variables and abstract class variables: https://github.com/patrick-kidger/equinox/blob/main/equinox/_better_abstract.py

For the daily research, I use Equinox (https://github.com/patrick-kidger/equinox) as a DL librarry in JAX.

It sounds like you're concerned about how downstream libraries tend to wrap JAX transformations to handle their own thing? (E.g. `haiku.grad`.)

If so, then allow me to make my usual advert here for Equinox:

https://github.com/patrick-kidger/equinox

This actually works with JAX's native transformations. (There's no `equinox.vmap` for example.)

On higher-order functions more generally, Equinox offers a way to control these quite carefully, by making ubiquitous use of callables that are also pytrees. E.g. a neural network is both a callable in that it has a forward pass, and a pytree in that it records its parameters in its tree structure.