airoboros VS datablations

Just wanted to drop a note that I overhauled the airoboros tool not the models to have most of the prompts I've been using to build the datasets, plus a couple extras.

airoboros: using large language models to fine-tune large language models (https://github.com/jondurbin/airoboros)

This is a fine-tuned LlaMa model, using completely synthetic training data created by https://github.com/jondurbin/airoboros

I used airoboros, a python tool I wrote, to generate the synthetic instruction response pairs, and included a jailbreak prompt to attempt to bypass OpenAI censorship. This is the only dataset used to fine-tune the model.

This is a 7b parameter, fine-tuned on 100k synthetic instruction/response pairs generated by gpt-3.5-turbo using my version of self-instruct airoboros

GitHub Repo

1x chinchilla means it's not really undertrained but that more could be squeezed without excessive difficulty https://arxiv.org/abs/2305.16264

You might want to give a read to "Scaling Data-Constrained Language Models" [1]. They basically generalized the Chinchilla scaling law by investigating behavior on multi-epoch runs.

[1] https://arxiv.org/abs/2305.16264

I would imagine that there is a lot of overlap, yeah. That said, training on repeated data does seem to be effective at this level.

Scaling Data-Constrained Language Models (https://arxiv.org/abs/2305.16264)