unsloth VS ollama

Gemma 7b is 2.4x faster than HF + FA2.

Check out https://github.com/unslothai/unsloth for full benchmarks!

Here are the missing links:

* Gemma, a family of open models from Google: https://ai.google.dev/gemma

* Unsloth is a tool/method for training models faster (IIUC): https://github.com/unslothai/unsloth

Thanks! Again, partnerships over customers. If you're experienced and have the technical chops to make a MI300x sing, we want to work with you. Our model is that we are the capex/opex investor for businesses. As much as I love software, Hot Aisle is more of a hardware business. Running super high end large scale compute is an extreme challenge in itself. We are less interested in building the software side of things and want to foster those who can focus on that side.

https://github.com/unslothai/unsloth/issues/160

https://github.com/search?q=repo%3Apredibase%2Florax+rocm&ty...

https://github.com/sgl-project/sglang/issues/157

https://github.com/casper-hansen/AutoAWQ (supports rocm)

Unsloth’s colab notebooks for fine-tuning Mistral-7B are super easy to use and run fine in just about any colab instance:

https://github.com/unslothai/unsloth

It’s my default now for experimenting and basic training. If I want to get into the weeds with the training, I use axolotl, but 9/10, it’s not really necessary.

If anyone wants to finetune their own Mistral 7b model 2.2x faster and use 62% less memory - give our open source package Unsloth a try! https://github.com/unslothai/unsloth a try! :)

https://github.com/unslothai/unsloth seems good and more relevant to your aims perhaps but I haven't tried it.

The unsloth project offers some low-level optimizations for Llama et al, and as of today some prelim Mistral work (which I heard is the llama architecture?)

This seems to just be a link to the Unsloth Github repo[0], which in turn is the free version of Unsloth Pro/Max[1]. Maybe the link should be changed?

[0]: https://github.com/unslothai/unsloth

- Basic internet search (I start ollama CLI faster than I can start a browser - https://ollama.com)

- Formatting/changing text

- Troubleshooting code, esp. new frameworks/libs

- Recipes

- Data entry

- Organizing thoughts: High-level lists, comparison, classification, synonyms, jargon & nomenclature

- Learning esp. by analogy and example

RAG for:

- Website assistants (https://github.com/bennyschmidt/ragdoll-studio/tree/master/e...)

- Game NPCs (https://github.com/bennyschmidt/ragdoll-studio/tree/master/e...)

- Discord/Slack/forum bots (https://github.com/bennyschmidt/ragdoll-studio/tree/master/e...)

- Character-driven storytelling and creating art in a specific style for video game loading screens, background images, avatars, website art, etc. (https://github.com/bennyschmidt/ragdoll-studio/tree/master/r...)

Jan goes a step further by integrating with other local engines like LM Studio and ollama.

# install the Ollama curl -fsSL https://ollama.com/install.sh | sh # get the llama3 model ollama pull llama2 # install the MLFlow pip install mlflow

Ollama for running LLMs locally

curl -fsSL https://ollama.com/install.sh | sh

Streaming is not a problem (it's just a simple flag: https://github.com/wiktor-k/llama-chat/blob/main/index.ts#L2...) but I've never used voice input.

The examples show image input though: https://github.com/ollama/ollama/blob/main/docs/api.md#reque...

Maybe you can file an issue here: https://github.com/ollama/ollama/issues

I’m a huge fan of open source models, especially the newly release Llama 3. Because of the performance of both the large 70B Llama 3 model as well as the smaller and self-host-able 8B Llama 3, I’ve actually cancelled my ChatGPT subscription in favor of Open WebUI, a self-hostable ChatGPT-like UI that allows you to use Ollama and other AI providers while keeping your chat history, prompts, and other data locally on any computer you control.