whisper.cpp VS faster-whisper

Whisper large v3 from openai, but we host it ourselves on Modal.com. It's easy, fast, no rate limits, and cheap as well.

If you want to run it locally, I'd still go with whisper, then I'd look at something like whisper.cpp https://github.com/ggml-org/whisper.cpp. Runs quite well.

not the gp but found this https://github.com/ggml-org/whisper.cpp/blob/master/models/c...

Actually that's a really good question, I hadn't considered that the comparison here is just CPU vs using Metal (CPU+GPU).

To answer the question though - I think this would be used for cases where you are building an app that wants to utilize a small AI model while at the same time having the GPU free to do graphics related things, which I'm guessing is why Apple stuck these into their hardware in the first place.

Here is an interesting comparison between the two from a whisper.cpp thread - ignoring startup times - the CPU+ANE seems about on par with CPU+GPU: https://github.com/ggml-org/whisper.cpp/pull/566#issuecommen...

A great thread with the type of info your looking for lives here: https://github.com/ggerganov/whisper.cpp/issues/89

But you can likely find similar threads for the llama.cpp benchmark here: https://github.com/ggerganov/llama.cpp/tree/master/examples/...

These are good examples because the llama.cpp and whisper.cpp benchmarks take full advantage of the Apple hardware but also take full advantage of non-Apple hardware with GPU support, AVX support etc.

It’s been true for a while now that the memory bandwidth of modern Apple systems in tandem with the neural cores and gpu has made them very competitive Nvidia for local inference and even training.

Not as convenient, but you could also have the user manually install the model, like whisper does.

Just forward the error message output by whisper, or even make a more user-friendly error message with instructions on how/where to download the models.

Whisper does provide a simple bash script to download models: https://github.com/ggerganov/whisper.cpp/blob/master/models/...

(As a Windows user, I can run bash scripts via Git Bash for Windows[1])

[1]: https://git-scm.com/download/win

whisper.cpp [1] has a karaoke example that uses ffmpeg's drawtext filter to display rudimentary karaoke-like captions. It also supports diarisation. Perhaps it could be a starting point to create a better script that does what you need.

--

1: https://github.com/ggerganov/whisper.cpp/blob/master/README....

Hi, I don't know what's SOTA, but I got good results with these (open source, on-device) :

https://github.com/SYSTRAN/faster-whisper (speech-to-text)

I've been using this:

https://github.com/bugbakery/transcribee

It's noticeably work-in-progress but it does the job and has a nice UI to edit transcriptions and speakers etc.

It's running on the CPU for me, would be nice to have something that can make use of a 4GB Nvidia GPU, which faster-whisper is actually able to [1]

https://github.com/SYSTRAN/faster-whisper?tab=readme-ov-file...

Faster-whisper (https://github.com/SYSTRAN/faster-whisper)

For our real-time STT needs, we'll employ a fantastic library called faster-whisper.

Thermostats: https://www.sinopetech.com/en/products/thermostat/

I haven't tried running a local text-to-speech engine backed by an LLM to control Home Assistant. Maybe someone is working on this already?

TTS: https://github.com/SYSTRAN/faster-whisper

LLM: https://github.com/Mozilla-Ocho/llamafile/releases

LLM: https://huggingface.co/TheBloke/Nous-Hermes-2-Mixtral-8x7B-D...

It would take some tweaking to get the voice commands working correctly.

Could someone elaborate how is this accomplished and is there any quality disparity compared to original whisper?

Repos like https://github.com/SYSTRAN/faster-whisper makes immediate sense about why it's faster than the original, but this one, not so much, especially considering it's even much faster.

Cool! I had the same project idea recently. You may be interested in this for the step of speech2text: https://github.com/SYSTRAN/faster-whisper

That's the implication. If the distil models are same format as original openai models then the Distil models can be converted for faster-whisper use as per the conversion instructions on https://github.com/guillaumekln/faster-whisper/

So then we'll see whether we get the 6x model speedup on top of the stated 4x faster-whisper code speedup.

guillaumekln/faster-whisper