esp-web-tools VS esp-sr

Some feedback to make your project easier to install and integrate better with Home Assistant (I'm the founder):

Home Assistant is building a voice assistant as part of our Year of the Voice theme. https://www.home-assistant.io/blog/2023/04/27/year-of-the-vo...

As part of our recent chapter 2 milestone, we introduced new Assist Pipelines. This allows users to configure multiple voice assistants. Your project is using the old "conversation" API. Instead it should use our new assist pipelines API. Docs: https://developers.home-assistant.io/docs/voice/pipelines/

You can even off-load the STT and TTS fully to Home Assistant and only focus on wake words.

You will see a lot higher adoption rate if users can just buy the ESP BOX and install the software on it without installing/compiling stuff. That's exactly why we created ESP Web Tools. It offers projects to offer browser-based installation directly from their website. https://esphome.github.io/esp-web-tools/

If you're going the ESP Web Tools route (and you should!), we've also created Improv Wi-Fi, a small protocol to configure Wi-Fi on the ESP device. This will allow ESP Web Tools to offer an onboarding wizard in the browser once the software has been installed. More info at https://www.improv-wifi.com/

Good luck!

Note: if anyone else wants to make an installer website like this, it’s called ESP Web Tools and open source: https://esphome.github.io/esp-web-tools/

ESP Web Tools uses WebSerial to allow users to install, update and manage firmware running on ESP microcontrollers: https://esphome.github.io/esp-web-tools/

If you want to make the project even more accessible, consider setting up a GitHub pages with ESP Web Tools: https://esphome.github.io/esp-web-tools/

Instead of esptool suggestions, use ESP web tools and your whole flow can be web based. https://esphome.github.io/esp-web-tools/

If it is the right thing, yes: https://github.com/espressif/esp-sr

For wake word and voice activity detection, audio processing, etc we use the ESP SR (speech recognition) framework from Espressif[0].

For speech to text there are two options and more to come:

1) Completely on device command recognition using the ESP SR Multinet 6 model. Willow will (currently) pull your light and switch entities from Home Assistant and generate the grammar and command definition required by Multinet. We want to develop a Willow Home Assistant component that will provide tighter Willow integration with HA and allow users to do this point and click with dynamic updates for new/changed entities, different kinds of entities, etc all in the HA dashboard/config.

The only "issue" with Multinet is that it only supports 400 defined commands. You're not going to get something like "What's the weather like in $CITY?" out of it.

For that we have:

2-?) Our own highly optimized inference server using Whisper, LLamA/Vicuna, and Speecht5 from transformers (more to come soon). We're open sourcing it next week. Willow streams audio after wake in realtime, gets the STT output, and sends it wherever you want. With the Willow Home Assistant component (doesn't exist yet) it will sit in between our inference server implementation doing STT/TTS or any other STT/TTS implementation supported by Home Assistant and handle all of this for you.

[0] - https://github.com/espressif/esp-sr

I don't believe that's possible. Seems like a pretty delicate process in the readme...