openai-whisper-cpu
whisper-asr-webservice
openai-whisper-cpu | whisper-asr-webservice | |
---|---|---|
5 | 11 | |
221 | 1,644 | |
- | - | |
10.0 | 7.8 | |
over 1 year ago | 4 days ago | |
Jupyter Notebook | Python | |
MIT License | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
openai-whisper-cpu
-
How to run Llama 13B with a 6GB graphics card
I feel the same.
For example some stats from Whisper [0] (audio transcoding) show the following for the medium model (see other models in the link):
---
GPU medium fp32 Linear 1.7s
CPU medium fp32 nn.Linear 60.7
CPU medium qint8 (quant) nn.Linear 23.1
---
So the same model runs 35.7 times faster on GPU, and compared to an CPU-optimized model still 13.6.
I was expecting around an order or magnitude of improvement. Then again, I do not know if in the case of this article the entire model was in the GPU, or just a fraction of it (22 layers), which might explain the result.
[0] https://github.com/MiscellaneousStuff/openai-whisper-cpu
-
Whispers AI Modular Future
According to https://github.com/MiscellaneousStuff/openai-whisper-cpu the medium model needs 1.7 seconds to transcribe 30 seconds of audio when run on a GPU.
-
[P] Transcribe any podcast episode in just 1 minute with optimized OpenAI/whisper
There is a very simple method built-in to PyTorch which can give you over 3x speed improvement for the large model, which you could also combine with the method proposed in this post. https://github.com/MiscellaneousStuff/openai-whisper-cpu
-
[D] How to get the fastest PyTorch inference and what is the "best" model serving framework?
For CPU inference, model quantization is a very easy to apply method with great average speedups which is already built-in to PyTorch. For example, I applied dynamic quantization to the OpenAI Whisper model (speech recognition) across a range of model sizes (ranging from tiny which had 39M params to large which had 1.5B params). Refer to the below table for performance increases:
-
[P] OpenAI Whisper - 3x CPU Inference Speedup
GitHub
whisper-asr-webservice
- How I converted a podcast into a knowledge base using Orama search and OpenAI whisper and Astro
-
Bazarr AI subs
Check https://github.com/openai/whisper & https://github.com/ahmetoner/whisper-asr-webservice
-
Bulk download subtitles
I see that bazarr had already been mentioned. If there are no subtitles available, you can also generate the subtitles by connecting bazarr to the AI model whisper which you can self host locally. I run everything in containers, tried it a few times and it works quite well for me! It does however use some computational resources to generate the subtitles, how long processing takes depends on the chosen model accuracy.
-
Writeout.ai – Transcribe and translate any audio files. Free and open source
You (essentially) need GPU but here you go:
https://github.com/ahmetoner/whisper-asr-webservice
For your requirements the medium.en model (max) should be satisfactory.
-
Whispers AI Modular Future
What utilities related to Whisper do you wish existed? What have you had to build yourself?
On the end user application side, I wish there was something that let me pick a podcast of my choosing, get it fully transcribed, and get an embeddings search plus answer q&a on top of that podcast or set of chosen podcasts. I've seen ones for specific podcasts, but I'd like one where I can choose the podcast. (Probably won't build it)
Also on the end user side, I wish there was an Otter alternative (still paid $30/mo, but unlimited minutes per month) that had longer transcription limits. (Started building this, not much interest from users though)
Things I've seen on the dev tool side:
Gladia (API call version of Whisper)
Whisper.cpp
Whisper webservice (https://github.com/ahmetoner/whisper-asr-webservice) - via this thread
Live microphone demo (not real time, it still does it in chunks) https://github.com/mallorbc/whisper_mic
Streamlit UI https://github.com/hayabhay/whisper-ui
Whisper playground https://github.com/saharmor/whisper-playground
Real time whisper https://github.com/shirayu/whispering
Whisper as a service https://github.com/schibsted/WAAS
Improved timestamps and speaker identification https://github.com/m-bain/whisperX
MacWhisper https://goodsnooze.gumroad.com/l/macwhisper
Crossplatform desktop Whisper that supports semi-realtime https://github.com/chidiwilliams/buzz
-
I made a free transcription service powered by Whisper AI
I think there's been talk to do speaker diarization with whisper-asr-webservice[0] which is also written in python and should be able to make use of goodies such as pyannote-audio, py-webrtcvad, etc.
Whisper is great but at the point we get to kludging various things together it starts to make more sense to use something like Nvidia NeMo[1] which was built with all of this in mind and more
[0] - https://github.com/ahmetoner/whisper-asr-webservice
[1] - https://github.com/NVIDIA/NeMo
- whisper-asr-webservice-client - A self-hosted OpenAI Whisper API client
-
Show HN: A self-hosted OpenAI Whisper API client
(read the docs in the repo)
In terms of me not storing your data for this (I don't) I guess you'll just have to trust me?
[0] - https://github.com/ahmetoner/whisper-asr-webservice
-
[P] OpenAI Whisper ASR Webservice API released
For more details: https://github.com/ahmetoner/whisper-asr-webservice
What are some alternatives?
llama-cpp-python - Python bindings for llama.cpp
whisper.cpp - Port of OpenAI's Whisper model in C/C++
intel-extension-for-pytorch - A Python package for extending the official PyTorch that can easily obtain performance on Intel platform
whisper - Robust Speech Recognition via Large-Scale Weak Supervision
whisperX - WhisperX: Automatic Speech Recognition with Word-level Timestamps (& Diarization)
generate-subtitles - Generate transcripts for audio and video content with a user friendly UI, powered by Open AI's Whisper with automatic translations and download videos automatically with yt-dlp integration
FlexGen - Running large language models on a single GPU for throughput-oriented scenarios.
buzz - Buzz transcribes and translates audio offline on your personal computer. Powered by OpenAI's Whisper.
whisper-asr-webservice-client
kernl - Kernl lets you run PyTorch transformer models several times faster on GPU with a single line of code, and is designed to be easily hackable.
gitbar-2023 - New release of gitbar website