llamafile VS gpt4all

Do you want that integrated in the browser? Because Mozilla already distributes offline LLMs: https://github.com/Mozilla-Ocho/llamafile

See https://github.com/Mozilla-Ocho/llamafile, a standalone packaging of llama.cpp that runs an LLM locally. It will use the GPU, but it also falls back on the CPU. CPU performance of small, quantized models is still pretty decent, and the page has estimated memory requirements for different models.

I think the llamafile[0] system works the best. Binary works on the command line or launches a mini webserver. Llamafile offers builds of Mixtral-8x7B-Instruct, so presumably they may package this one up as well (potentially a quantized format).

You would have to confirm with someone deeper in the ecosystem, but I think you should be able to run this new model as is against a llamafile?

[0] https://github.com/Mozilla-Ocho/llamafile

Thermostats: https://www.sinopetech.com/en/products/thermostat/

I haven't tried running a local text-to-speech engine backed by an LLM to control Home Assistant. Maybe someone is working on this already?

TTS: https://github.com/SYSTRAN/faster-whisper

LLM: https://github.com/Mozilla-Ocho/llamafile/releases

LLM: https://huggingface.co/TheBloke/Nous-Hermes-2-Mixtral-8x7B-D...

It would take some tweaking to get the voice commands working correctly.

While I did not succeed in making the matmul code from https://github.com/Mozilla-Ocho/llamafile/blob/main/llamafil... work in isolation, I compared eigen, openblas, and mkl: https://gist.github.com/Dobiasd/e664c681c4a7933ef5d2df7caa87...

In this (very primitive!) benchmark, MKL was a bit better than eigen (~10%) on my machine (i5-6600).

Since the article https://justine.lol/matmul/ compared the new kernels with MLK, we can (by transitivity) compare the new kernels with Eigen this way, at least very roughly for this one use-case.

Yes, they're just ZIP files that also happen to be actually portable executables.

https://github.com/Mozilla-Ocho/llamafile?tab=readme-ov-file...

I don’t know if Ollama can do this but https://gpt4all.io/ can.

Gpt4all is a local desktop app with a Python API that can be trained on your documents: https://gpt4all.io/

The readme page is cryptic. What does 'mature' mean in this context? What is the sample text a continuation of?

Hving a gif the thing in use would be great, similar to the gpt4all readme page. (https://github.com/nomic-ai/gpt4all)

Check https://github.com/nomic-ai/gpt4all instead.

"I ran performance tests on two systems, here's the results of system 1, and heres the results of system 2. Summarize the results, and build a markdown table containing x,y,z rows."

"extract the reusable functions out of this bash script"

"write me a cfssl command to generate a intermediate CA"

"What is the regex for _____"

"Here are my accomplishments over the last 6 months, summarize them into a 1 page performance report."

etc etc etc

If you're not using GPT4 or some LLM as part of your daily flow you're working too hard.

Get GPT4All (https://gpt4all.io), log into OpenAI, drop $20 on your account, get a API key, and start using GPT4.

offline versija, nors ir ne tokia pažengus - https://github.com/nomic-ai/gpt4all ; https://gpt4all.io/index.html

Take a look at https://gpt4all.io, their docs are pretty awesome

I run one. I built an iMessage-like frontend to it using plain JS and a Python websocket backend. I mostly just use it for curiosity and playing with different prompts. I only have 16GB of RAM to dedicate to it, so I use an 8B parameter model which is enough for fun and chitchat, but I don't find it good enough to replace ChatGPT.

https://github.com/nomic-ai/gpt4all