faster-whisper VS whisperX

Hi, I don't know what's SOTA, but I got good results with these (open source, on-device) :

https://github.com/SYSTRAN/faster-whisper (speech-to-text)

I've been using this:

https://github.com/bugbakery/transcribee

It's noticeably work-in-progress but it does the job and has a nice UI to edit transcriptions and speakers etc.

It's running on the CPU for me, would be nice to have something that can make use of a 4GB Nvidia GPU, which faster-whisper is actually able to [1]

https://github.com/SYSTRAN/faster-whisper?tab=readme-ov-file...

Faster-whisper (https://github.com/SYSTRAN/faster-whisper)

For our real-time STT needs, we'll employ a fantastic library called faster-whisper.

Thermostats: https://www.sinopetech.com/en/products/thermostat/

I haven't tried running a local text-to-speech engine backed by an LLM to control Home Assistant. Maybe someone is working on this already?

TTS: https://github.com/SYSTRAN/faster-whisper

LLM: https://github.com/Mozilla-Ocho/llamafile/releases

LLM: https://huggingface.co/TheBloke/Nous-Hermes-2-Mixtral-8x7B-D...

It would take some tweaking to get the voice commands working correctly.

Could someone elaborate how is this accomplished and is there any quality disparity compared to original whisper?

Repos like https://github.com/SYSTRAN/faster-whisper makes immediate sense about why it's faster than the original, but this one, not so much, especially considering it's even much faster.

Cool! I had the same project idea recently. You may be interested in this for the step of speech2text: https://github.com/SYSTRAN/faster-whisper

That's the implication. If the distil models are same format as original openai models then the Distil models can be converted for faster-whisper use as per the conversion instructions on https://github.com/guillaumekln/faster-whisper/

So then we'll see whether we get the 6x model speedup on top of the stated 4x faster-whisper code speedup.

guillaumekln/faster-whisper

I use whisperfile[1] directly. The whisper-large-v3 model seems good with non-English transcription, which is my main use-case.

I am also eyeing whisperX[2], because I want to play some more with speaker diarization.

Your use-case seems to be batch transcription, so I'd suggest you go ahead and just use whisperfile, it should work well on an M4 mini, and it also has an HTTP API if you just start it without arguments.

If you want more interactivity, I have been using Vibe[3] as an open-source replacement of SuperWhisper[4], but VoiceInk from a sibling comment seems better.

Aside: It seems that so many of the mentioned projects use whisper at the core, that it would be interesting to explicitly mark the projects that don't use whisper, so we can have a real fundamental comparison.

[1] https://huggingface.co/Mozilla/whisperfile

[2] https://github.com/m-bain/whisperX

[3] https://github.com/thewh1teagle/vibe/

[4] https://superwhisper.com/

Yes it's still relevant but I prefer WhisperX for some tasks: https://github.com/m-bain/whisperX

https://github.com/m-bain/whisperX looks promising - I'm hacking away on an always-on transcriber for my notes for later search&recall. It has support for diarization (the speaker detection you're looking for).

I'm currently hacking away on a mix of https://github.com/speaches-ai/speaches + https://github.com/ufal/whisper_streaming though - mostly because my laptop doesn't have a decent GPU, I stream the audio to a home server instead.

But overall it's pretty simple to do after you wrangle the Python dependencies - all you need is a sink for the text files (for example, create a new file for every Teams meeting, but that's another story...)

You don't need to wait, you can use: https://github.com/m-bain/whisperX right now for STT with timestamps and diarization.

I use whisper and pyannote (https://github.com/m-bain/whisperX), but it is a pain to run locally - I run it on a 4080. This seems to be actually trying to identify the speakers. Not sure what they are doing for that.

3. WhisperX

> And do you know a good speech to text model?

OpenAI's whisper, code+model are available, and multiple projects have built on it. You could try this wrapper: https://github.com/m-bain/whisperX -- or for short utterances on a smart-phone https://github.com/futo-org/whisper-acft