pyscript VS llama.cpp

3) Again, already there. ESPHome is a first class citizen in HA.

I feel like you've not read the HA docs [1,] or took the time to understand the architecture [2]. And, for someone who has more than enough self-proclaimed skills, this should be a very understandable system.

[0] https://github.com/custom-components/pyscript

I see someone else mentioned AppDaemon. I would try out pyscript first ( https://github.com/custom-components/pyscript ). It is just a some HACS install, and then you are good to go.

I highly recommend using this instead of the pretty awful built-in Python thing. Pyscript is just a TON better.

Sure! I use pyscript for the automation; here's the wiki page I made.

Sounds like you might enjoy using Pyscript to write your automations. I also found Node-Red to be frustrating when I was starting off; Pyscript let's you write Python for everything and is super easy to follow IMO.

Yes, Metal seems to allow a maximum of 1/2 of the RAM for one process, and 3/4 of the RAM allocated to the GPU overall. There’s a kernel hack to fix it, but that comes with the usual system integrity caveats. https://github.com/ggerganov/llama.cpp/discussions/2182

For anyone interested in exploring this, llama.cpp has an example implementation here:

https://github.com/ggerganov/llama.cpp/tree/master/examples/...

Getting started with LLMs can be intimidating. In this tutorial we will show you how to fine-tune a large language model using LoRA, facilitated by tools like llama.cpp and KitOps.

well https://github.com/ggerganov/llama.cpp/issues/6849

Have just done this recently for local chat with pdf feature in https://recurse.chat. (It's a macOS app that has built-in llama.cpp server and local vector database)

Running an embedding server locally is pretty straightforward:

- Get llama.cpp release binary: https://github.com/ggerganov/llama.cpp/releases