LLM-Finetuning-Hub VS bedframe

Hi everyone, we've been working on benchmarking different open-source LLMs. We measure, in particular, on the performance of these models once finetued (via QLoRA) on classic NLP downstream tasks like summarization and classification. We also put particular emphasis on benchmarking inference time/cost for these models once deployed.

Hello all,

I have been working on benchmarking different LLMs -- both open-source and closed-source.

Repo: https://github.com/georgian-io/LLM-Finetuning-Hub

Precisely, I am comparing their out-of-the-box capabilities (prompting) and their fine-tuned conterparts!

So far, the following models have been benchmarked:

Open-Source:

- Finetuning with QLoRA

I evaluate how Falcon does on classification tasks when compared to Bert and Distilbert.

Moreover, I talk about different ways you can deploy the model, and the associated costs!

The code for all of my experiments are available on: https://github.com/georgian-io/LLM-Finetuning-Hub

Happy reading and learning!

Author here! the project is super new. alpha, at best.

Basically if you wanted to build browser extensions using modern stack, bedframe lets you do that. Specifically it uses Vite under the hood so you can write e.g. a chrome extension in e.g. React w/ TypeScript, Tailwind, and all that good stuff.

I think the CLI readme has a bit more useful info: https://github.com/nyaggah/bedframe/tree/main/packages/cli

The notable bit is the mvp.yml workflow you get in your project. It'll let you publish to chrome, firefox and edge automatically (ci/cd).

> It's also very unclear what state this project is in.

fair enough! Wanted to get something out there first; I'll get the docs out soon