setfit VS git-re-basin

It did, sorry for the hasty edits! I removed that part b/c I realized that there isn't a compelling-enough reason for me to believe that text similarity is clearly inappropriate. In fact, you can train the Pr(condition | chat) classifier I suggested above using similarity training! Use SetFit for that. In the end you'll get a classifier and a similarity model.

[3] https://github.com/huggingface/setfit

There's also kinda related approaches like SetFit which calculate embeddings from pretrained transformer models then then fit a classifier on top of the embeddings. I've yet to try it but it supposedly works well with very few labelled examples.

You could have a look at setfit.

Found relevant code at https://github.com/huggingface/setfit + all code implementations here

Efficient Few-Shot Learning Without Prompts https://github.com/huggingface/setfit https://arxiv.org/abs/2209.11055v1

Implementation: https://github.com/samuela/git-re-basin

Wouldn't "applying the permutation" simply swap all the parameters in a model so they match on both models? For example, in https://github.com/samuela/git-re-basin/blob/main/src/cifar10_vgg_weight_matching.py, on line 184 they apply the permutation, and on line 192 they lerp from model A's params to the permuted model B's params. This lerp is basically a weighted sum merge, isn't it? At a lerp of 0.5, it would be somewhere in between model A and the permuted model B.

Definitely interesting, but you might get something useful out of https://github.com/samuela/git-re-basin ?

Git Re-Basin: Merging Models modulo Permutation Symmetries https://github.com/samuela/git-re-basin https://arxiv.org/abs/2209.04836v1