sito VS equinox

As of recent I've been weirdly fascinated with codecs, serialization protocols, file formats, and the like. It scratches that low-level hacking itch (I spent most of my professional programming in web dev space) without as much commitment to electronics (used to be super big into arduino but I find that's harder to pick up and put down).

I've started hacking on my own container format (yeah, I know, xkcd927), after finding it super frustrating to embed arbitrary time-synched data streams into mp4/matroska/ogg/etc. Also it bugs me how crusty, complicated, and arcane mp4 is, and at the same time, mastroska and ogg are weirdly opaque given how they are supposed to be open standards.

If anyone is curious, here's my container format I've been developing: https://github.com/xkortex/sito

I wrote a JAX-based neural network library (Equinox [1]) and numerical differential equation solving library (Diffrax [2]).

At the time I was just exploring some new research ideas in numerics -- and frankly, procrastinating from writing up my PhD thesis!

But then one of the teams at Google starting using them, so they offered me a job to keep developing them for their needs. Plus I'd get to work in biotech, which was a big interest of mine. This was a clear dream job offer, so I accepted.

Since then both have grown steadily in popularity (~2.6k GitHub stars) and now see pretty widespread use! I've since started writing several other JAX libraries and we now have a bit of an ecosystem going.

[1] https://github.com/patrick-kidger/equinox

The elevator pitch is Optimistix is really fast, especially to compile. It plays nicely with Optax for first-order gradient-based methods, and takes a lot of design inspiration from Equinox, representing the state of all the solvers as standard JAX PyTrees.

If you like PyTorch then you might like Equinox, by the way. (https://github.com/patrick-kidger/equinox ; 1.4k GitHub stars now!)

You're thinking of `jax.closure_convert`. :)

(Although technically that works by tracing and extracting all constants from the jaxpr, rather than introspecting the function's closure cells -- it sounds like your trick is the latter.)

When you discuss dynamic allocation, I'm guessing you're mainly referring to not being able to backprop through `jax.lax.while_loop`. If so, you might find `equinox.internal.while_loop` interesting, which is an unbounded while loop that you can backprop through! The secret sauce is to use a treeverse-style checkpointing scheme.

https://github.com/patrick-kidger/equinox/blob/f95a8ba13fb35...

I'm a big fan of using ABCs to declare interfaces -- so much so that I have an improved abc.ABCMeta that also handles abstract instance variables and abstract class variables: https://github.com/patrick-kidger/equinox/blob/main/equinox/_better_abstract.py

For the daily research, I use Equinox (https://github.com/patrick-kidger/equinox) as a DL librarry in JAX.

It sounds like you're concerned about how downstream libraries tend to wrap JAX transformations to handle their own thing? (E.g. `haiku.grad`.)

If so, then allow me to make my usual advert here for Equinox:

https://github.com/patrick-kidger/equinox

This actually works with JAX's native transformations. (There's no `equinox.vmap` for example.)

On higher-order functions more generally, Equinox offers a way to control these quite carefully, by making ubiquitous use of callables that are also pytrees. E.g. a neural network is both a callable in that it has a forward pass, and a pytree in that it records its parameters in its tree structure.