chat-ui VS llamafile

Zephyr 141B is a Mixtral 8x22B fine-tune. Here are some interesting details

- Base model: Mixtral 8x22B, 8 experts, 141B total params, 35B activated params

- Fine-tuned with ORPO, a new alignment algorithm with no SFT step (hence much faster than DPO/PPO)

- Trained with 7K open data instances -> high-quality, synthetic, multi-turn

- Apache 2

Everything is open:

- Final Model: https://huggingface.co/HuggingFaceH4/zephyr-orpo-141b-A35b-v...

- Base Model: https://huggingface.co/mistral-community/Mixtral-8x22B-v0.1

- Fine-tune data: https://huggingface.co/datasets/argilla/distilabel-capybara-...

- Recipe/code to train the model: https://huggingface.co/datasets/argilla/distilabel-capybara-...

- Open-source inference engine: https://github.com/huggingface/text-generation-inference

- Open-source UI code https://github.com/huggingface/chat-ui

Have fun!

As long as you have a free Hugging Face account, you can sign up and exploit HuggingChat, a web-based chat interface where you will find 5 large language models to play with (Mixtral-7B-it v0.1 and v0.2, Command R plus, Gemma 1.1-7B-it, Dolphin). You will also have the possibility to exploit several assistants made by the Hugging Face community, or even create your own!

I submitted something about this the other day (and it got flagged)- poked around a little bit and the only interesting thing I could find is this: https://github.com/huggingface/chat-ui/issues/254 and I don't really even understand what it is, it references the stuff the dude who wrote this is discussing. I had kinda written the whole thing off as someone with too much time on their hands and is just f'ing around with stuff for whatever reason.

I think they made this as well: https://chat.openai.com/g/g-KT4gusP3Y-a-l-i-s-t-a-i-r-e-earl... - it doesn't seem very useful.

*¯\_(ツ)_/¯ to me after spending an hr or so poking around, it seemed like a bored modern tech savvy young person playing around.

- https://github.com/huggingface/chat-ui

Hugging Face Chat - open-source repo powering Hugging Chat!

If you don't want to do coding, I think HuggingFace's chat-ui can come in handy with web retrieval RAG and llama-cpp running as a server. Please check their documentation on how to setup( See "Running your own models using a custom endpoint" section on their Github).

yes! it's proprietary, invasive, and harvests your data and use it for improving the AI, Ultman went to Israel weeks after Chatgpt was introduced, Israel like any other tech-giant-country needs to make sure that it has control over that data and/or use it to achieve its goals, so it's better to find offline FOSS alternatives (if you have a decent enough PC) or use HuggingChat as an online FOSS alternative, I find it better than GPT 3.5 in many aspects

I have categorized some of the smartphone brands by their parent company using HuggingChat based on RLHF, Google's Bard, ChatGPT, and Perplexity. All of them are powered by LLMs, and both ChatGPT and Perplexity use GPT-3.5.

I'm looking for something like https://huggingface.co/chat/ or OpenAssistant, but it should target OpenAI's api.

I think the llamafile[0] system works the best. Binary works on the command line or launches a mini webserver. Llamafile offers builds of Mixtral-8x7B-Instruct, so presumably they may package this one up as well (potentially a quantized format).

You would have to confirm with someone deeper in the ecosystem, but I think you should be able to run this new model as is against a llamafile?

[0] https://github.com/Mozilla-Ocho/llamafile

Thermostats: https://www.sinopetech.com/en/products/thermostat/

I haven't tried running a local text-to-speech engine backed by an LLM to control Home Assistant. Maybe someone is working on this already?

TTS: https://github.com/SYSTRAN/faster-whisper

LLM: https://github.com/Mozilla-Ocho/llamafile/releases

LLM: https://huggingface.co/TheBloke/Nous-Hermes-2-Mixtral-8x7B-D...

It would take some tweaking to get the voice commands working correctly.

While I did not succeed in making the matmul code from https://github.com/Mozilla-Ocho/llamafile/blob/main/llamafil... work in isolation, I compared eigen, openblas, and mkl: https://gist.github.com/Dobiasd/e664c681c4a7933ef5d2df7caa87...

In this (very primitive!) benchmark, MKL was a bit better than eigen (~10%) on my machine (i5-6600).

Since the article https://justine.lol/matmul/ compared the new kernels with MLK, we can (by transitivity) compare the new kernels with Eigen this way, at least very roughly for this one use-case.

Yes, they're just ZIP files that also happen to be actually portable executables.

https://github.com/Mozilla-Ocho/llamafile?tab=readme-ov-file...

have you seen llamafile[0]?

[0] https://github.com/Mozilla-Ocho/llamafile

llama.cpp has integrated gemma support. So you can use llamafile for this. It is a standalone executable that is portable across most popular OSes.

https://github.com/Mozilla-Ocho/llamafile/releases

So, download the executable from the releases page under assets. You want either just main or just server. Don't get the huge ones with the model inlined in the file. The executable is about 30MB in size,

https://github.com/Mozilla-Ocho/llamafile/releases/download/...

The improvements in ease of use for locally hosting LLMs over the last few months have been amazing. I was ranting about how easy https://github.com/Mozilla-Ocho/llamafile is just a few hours ago [1]. Now I'm torn as to which one to use :)

1: Quite literally hours ago: https://euri.ca/blog/2024-llm-self-hosting-is-easy-now/