Our great sponsors
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
-
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
-
-
home-assistant-assist-desktop
Use Home Assistant Assist on the desktop. Compatible with Windows, MacOS, and Linux
lamma.cpp allows you to restrict the output such that it would always generate valid JSON https://github.com/ggerganov/llama.cpp#constrained-output-wi...
I did the same thing, but I went the easy way and used OpenAI's API. Half way through, I got fed up with all the boilerplate, so I wrote a really simple (but very Pythonic) wrapper around function calling with Python functions:
https://github.com/skorokithakis/ez-openai
Then my assistant is just a bunch of Python functions and a prompt. Very very simple.
See Magentic for something similar: https://github.com/jackmpcollins/magentic
Regarding accessible local LLMs have you heard of the llamafiles project? It allows for packaging one executable LLM that works on Mac, windows and Linux.
Currently pushing for application note https://github.com/Mozilla-Ocho/llamafile/pull/178 to encourage integration. Would be good to hear your thoughts on making it easier for home assistant to integrate with llamafiles.
As a user of Home Assistant, I would want to easily be able to try out different AI models with a single click from the user interface.
Home Assistant allows users to install add-ons which are Docker containers + metadata. This is how today users install Whisper or Piper for STT and TTS. Both these engines have a wrapper that speaks Wyoming, our voice assistant standard to integrate such engines, among other things. (https://github.com/rhasspy/rhasspy3/blob/master/docs/wyoming...)
If we rely on just the ChatGPT API to allow interacting with a model, we wouldn't know what capabilities the model has and so can't know what features to use to get valid JSON actions out. Can we pass our function definitions or should we extend the prompt with instructions on how to generate JSON?
alternatively, could we not simply split by common characters such as newlines and periods, to split it within sentences? it would be fragile with special handling required for numbers with decimal points and probably various other edge cases, though.
there are also Python libraries meant for natural language parsing[0] that could do that task for us. I even see examples on stack overflow[1] that simply split text into sentences.
[0]: https://www.nltk.org/
For the ones who needs to utilize openai tts engine, here is a custom component i created for HA. Results are really good!
https://github.com/sfortis/openai_tts
Honor to meet you!
[Anonymous] founder of a similarly high-profile initiative here.
> Creating a prompt to write JSON is possible but need quite an elaborate prompt and even then the LLM can make errors. We want to make sure that all JSON coming out of the model is directly actionable without having to ask the LLM what they might have meant for a specific value
The LLM cannot make errors. The LLM spits out probabilities for the next tokens. What you do with it is up to you. You can make errors in how you handle this.
Standard usages pick the most likely token, or a random token from the top many choices. You don't need to do that. You can pick ONLY words which are valid JSON, or even ONLY words which are JSON matching your favorite JSON format. This is a library which does this:
https://github.com/outlines-dev/outlines
The one piece of advice I will give: Do NOT neuter the AI like OpenAI did. There is a near-obsession to define "AI safety" as "not hurting my feelings" (as opposed to "not hacking my computer," "not launching nuclear missiles," or "not exterminating humanity."). For technical reasons, that makes them work much worse. For practical reasons, I like AIs with humanity and personality (much as the OP has). If it says something offensive, I won't break.
AI safety, in this context, means validating that it's not:
* setting my thermostat to 300 degrees centigrade
* power-cycling my devices 100 times per second to break them
* waking me in the middle of the night
... and similar.
Also:
* Big win if it fits on a single 16GB card, and especially not just NVidia. The cheapest way to run an LLM is an Intel Arc A770 16GB. The second-cheapest is an NVidia 4060 Ti 16GB
* Azure gives a safer (not safe) way of running cloud-based models for people without that. I'm pretty sure there's a business model running these models safely too.
3) Again, already there. ESPHome is a first class citizen in HA.
I feel like you've not read the HA docs [1,] or took the time to understand the architecture [2]. And, for someone who has more than enough self-proclaimed skills, this should be a very understandable system.
[0] https://github.com/custom-components/pyscript
Everything we make is accessible via APIs and integrating our Assist via APIs is already possible. Here is an example of an app someone made that runs on Windows, Mac and Linux: https://github.com/timmo001/home-assistant-assist-desktop