iris VS git-re-basin

Yea, after all these LLMs are predicting one sequence of tokens from another sequence of tokens and the tokens could be anything, it just "happens" that text has the most knowledge and the easiest to input, then there are image, sound, video, but tokens could also be learned from world experience in RL:

Transformers are Sample-Efficient World Models:

https://github.com/eloialonso/iris#transformers-are-sample-e...

Transformers are Sample Efficient World Models https://github.com/eloialonso/iris https://arxiv.org/abs/2209.00588v1

This is actually a pretty neat, self-contained implementation that can super easily extended beyond stereotypical natural language models, for example to create world models for video games [1] or to create robot models that can learn to imitate from large, chaotic human demonstration data [2] (disclaimer, I'm an author on the second one.) Basically, GPT (or minGPT) models are EXCELLENT sequence modelers, almost to the point where you can throw any sensible sequence data at it and hope to get interesting results, as long as you don't overfit.

Even though I have only been working on machine learning for around six years, it's crazy to see how the landscape has changed so fast so recently, including diffusion models and transformers. It's not too much to say that we might expect more major breakthroughs by the end of this decade, and end in a place we can't even imagine right now!

[1] https://github.com/eloialonso/iris

Implementation: https://github.com/samuela/git-re-basin

Wouldn't "applying the permutation" simply swap all the parameters in a model so they match on both models? For example, in https://github.com/samuela/git-re-basin/blob/main/src/cifar10_vgg_weight_matching.py, on line 184 they apply the permutation, and on line 192 they lerp from model A's params to the permuted model B's params. This lerp is basically a weighted sum merge, isn't it? At a lerp of 0.5, it would be somewhere in between model A and the permuted model B.

Definitely interesting, but you might get something useful out of https://github.com/samuela/git-re-basin ?

Git Re-Basin: Merging Models modulo Permutation Symmetries https://github.com/samuela/git-re-basin https://arxiv.org/abs/2209.04836v1