chatbot-ui VS llamafile

One of the first things we did when GPT-4 became available was talk to our Azure rep and get access to the OpenAI models that they'd partnered with Microsoft to host in Azure. Now, we have our own private, not-datamined (so they claim, contractually) API endpoint and we use an OpenAI integration in VS Code[1] to connect to, allowing anyone in the company to use it to help them code.

I also spun up an internal chat UI[2] to replace ChatGPT so people can feel comfortable discussing proprietary data with the LLM endpoint.

The only thing that would make it more secure would be running inference engines internally, but I wouldn't have access to as good of models, and I'd need a _lot_ of hardware to match the speeds.

[1] - https://marketplace.visualstudio.com/items?itemName=AndrewBu...

[2] - https://github.com/mckaywrigley/chatbot-ui (legacy branch)

[3] https://github.com/mckaywrigley/chatbot-ui

Thank you for the work.

Please take this in a nice way: I can't see why I would use this over ChatbotUI+Ollama https://github.com/mckaywrigley/chatbot-ui

Seem the only advantage is having it as MacOS native app and only real distinction is maybe fast import and search - I've yet to try that though.

ChatbotUI (and other similar stuff) are cross-platform, customizable, private, debuggable. I'm easily able to see what it's trying to do.

You can make a privacy request for OpenAI to not train on your data here: https://privacy.openai.com/

Alternatively, you could also use your own UI/API token (API calls aren't trained on). Chatbot UI just got a major update released and has nice things like folders, and chat search: https://github.com/mckaywrigley/chatbot-ui

Also chatbotUI is great https://github.com/mckaywrigley/chatbot-ui it has a ui similar to chatgpt

> The comment I screenshotted is passive-aggressive at best, and there's no really good way to ask "is this repo dead" without being passive-aggressive. My day-to-day job that actually pays me a salary wouldn't ever provide a bulleted list of the reasons I suck, let alone a project I develop in my spare time.

There is nothing passive-aggressive about that comment. There is nothing problematic about it at all. Nobody's calling you slurs or making demands. I see one guy who might as well be a Mormon Boy Scout from Canada. "Is this repo dead" is not passive-aggressive, just ineloquent. Fuck my eyes until the jelly leaks out my ears if a courteous and professionally-written question constitutes "applying pressure and being rude" these days.

I don't know what a "bulleted list of the reasons [you] suck" has to do with anything (I don't see where anybody sent you one) but you're coming across as someone who invites people to your garage sale and then brandishes a shotgun and starts screaming when they set foot on your property.

> I’ve never seen any discussions or articles about whether it’s appropriate to ask if an open source repository is dead. Is there an implicit contract to actively maintain any open source software you publish? Are you obligated to provide free support if you hit a certain star amount on GitHub or ask for funding through GitHub Sponsorships/Patreon? After all, most permissive open source code licenses like the MIT License contain some variant of “the software is provided ‘as is’, without warranty of any kind.”

Here's an example of why everyone should ask if an open source project is dead:

https://github.com/mckaywrigley/chatbot-ui/issues

A number of issues complain about it leaking OpenAI keys. Nobody's figured out how, but it'd be nice to know if anybody's working on it, if it's worth submitting a PR, if it should be forked, if it's worth bothering with at all. This code is a massive liability in its current state. Its creator is absent. It warrants questions being asked about its future. Yeah, it's as-is software, but it's not an affront to your mother's virtue when someone asks if your shit still works or if you have plans to fix it.

> I’ve had an existential crisis about my work in open source AI on GitHub, particularly as there has been both increasingly toxic backlash against AI and because the AI industry has been evolving so rapidly that I flat-out don’t have enough bandwidth to keep up

Herein lies the problem? You sound overwhelmed. I've been there myself. I don't know what your year's been like but you genuinely might want to get away from the screen and get some fresh air. This is a good time of year to do it, since things generally slow down at work.

I think the llamafile[0] system works the best. Binary works on the command line or launches a mini webserver. Llamafile offers builds of Mixtral-8x7B-Instruct, so presumably they may package this one up as well (potentially a quantized format).

You would have to confirm with someone deeper in the ecosystem, but I think you should be able to run this new model as is against a llamafile?

[0] https://github.com/Mozilla-Ocho/llamafile

Thermostats: https://www.sinopetech.com/en/products/thermostat/

I haven't tried running a local text-to-speech engine backed by an LLM to control Home Assistant. Maybe someone is working on this already?

TTS: https://github.com/SYSTRAN/faster-whisper

LLM: https://github.com/Mozilla-Ocho/llamafile/releases

LLM: https://huggingface.co/TheBloke/Nous-Hermes-2-Mixtral-8x7B-D...

It would take some tweaking to get the voice commands working correctly.

While I did not succeed in making the matmul code from https://github.com/Mozilla-Ocho/llamafile/blob/main/llamafil... work in isolation, I compared eigen, openblas, and mkl: https://gist.github.com/Dobiasd/e664c681c4a7933ef5d2df7caa87...

In this (very primitive!) benchmark, MKL was a bit better than eigen (~10%) on my machine (i5-6600).

Since the article https://justine.lol/matmul/ compared the new kernels with MLK, we can (by transitivity) compare the new kernels with Eigen this way, at least very roughly for this one use-case.

Yes, they're just ZIP files that also happen to be actually portable executables.

https://github.com/Mozilla-Ocho/llamafile?tab=readme-ov-file...

have you seen llamafile[0]?

[0] https://github.com/Mozilla-Ocho/llamafile

llama.cpp has integrated gemma support. So you can use llamafile for this. It is a standalone executable that is portable across most popular OSes.

https://github.com/Mozilla-Ocho/llamafile/releases

So, download the executable from the releases page under assets. You want either just main or just server. Don't get the huge ones with the model inlined in the file. The executable is about 30MB in size,

https://github.com/Mozilla-Ocho/llamafile/releases/download/...

The improvements in ease of use for locally hosting LLMs over the last few months have been amazing. I was ranting about how easy https://github.com/Mozilla-Ocho/llamafile is just a few hours ago [1]. Now I'm torn as to which one to use :)

1: Quite literally hours ago: https://euri.ca/blog/2024-llm-self-hosting-is-easy-now/