whisperX VS insanely-fast-whisper

It uses this, which does support diarization: https://github.com/m-bain/whisperX

Author compared various whisper implementation

"We found that WhisperX is the best framework for transcribing long audio files efficiently and accurately. It’s much better than using the standard openai-whisper library."

https://github.com/m-bain/whisperX

import os # Directory and file paths dir_path = './models-v1' inference_file_path = os.path.join(dir_path, 'code/inference.py') requirements_file_path = os.path.join(dir_path, 'code/requirements.txt') # Create the directory structure os.makedirs(os.path.dirname(inference_file_path), exist_ok=True) # Inference.py content inference_content = '''# inference.py # inference.py import io import json import logging import os import tempfile import time import boto3 import torch import whisperx DEVICE = 'cuda' if torch.cuda.is_available() else 'cpu' s3 = boto3.client('s3') def model_fn(model_dir, context=None): """ Load and return the WhisperX model necessary for audio transcription. """ print("Entering model_fn") logging.info("Loading WhisperX model") model = whisperx.load_model(whisper_arch=f"{model_dir}/guillaumekln/faster-whisper-large-v2", device=DEVICE, language="en", compute_type="float16", vad_options={'model_fp': f"{model_dir}/whisperx/vad/pytorch_model.bin"}) print("Loaded WhisperX model") print("Exiting model_fn with model loaded") return { 'model': model } def input_fn(request_body, request_content_type): """ Process and load audio from S3, given the request body containing S3 bucket and key. """ print("Entering input_fn") if request_content_type != 'application/json': raise ValueError("Invalid content type. Must be application/json") request = json.loads(request_body) s3_bucket = request['s3bucket'] s3_key = request['s3key'] # Download the file from S3 temp_file = tempfile.NamedTemporaryFile(delete=False) s3.download_file(Bucket=s3_bucket, Key=s3_key, Filename=temp_file.name) print(f"Downloaded audio from S3: {s3_bucket}/{s3_key}") print("Exiting input_fn") return temp_file.name def predict_fn(input_data, model, context=None): """ Perform transcription on the provided audio file and delete the file afterwards. """ print("Entering predict_fn") start_time = time.time() whisperx_model = model['model'] logging.info("Loading audio") audio = whisperx.load_audio(input_data) logging.info("Transcribing audio") transcription_result = whisperx_model.transcribe(audio, batch_size=16) try: os.remove(input_data) # input_data contains the path to the temp file print(f"Temporary file {input_data} deleted.") except OSError as e: print(f"Error: {input_data} : {e.strerror}") end_time = time.time() elapsed_time = end_time - start_time logging.info(f"Transcription took {int(elapsed_time)} seconds") print(f"Exiting predict_fn, processing took {int(elapsed_time)} seconds") return transcription_result def output_fn(prediction, accept, context=None): """ Prepare the prediction result for the response. """ print("Entering output_fn") if accept != "application/json": raise ValueError("Accept header must be application/json") response_body = json.dumps(prediction) print("Exiting output_fn with response prepared") return response_body, accept ''' # Write the inference.py file with open(inference_file_path, 'w') as file: file.write(inference_content) # Requirements.txt content requirements_content = '''speechbrain==0.5.16 faster-whisper==0.7.1 git+https://github.com/m-bain/whisperx.git@1b092de19a1878a8f138f665b1467ca21b076e7e ffmpeg-python ''' # Write the requirements.txt file with open(requirements_file_path, 'w') as file: file.write(requirements_content)

Whisper doesn't, but WhisperX <https://github.com/m-bain/whisperX/> does. I am using it right now and it's perfectly serviceable.

For reference, I'm transcribing research-related podcasts, meaning speech doesn't overlap a lot, which would be a problem for WhisperX from what I understand. There's also a lot of accents, which are straining on Whisper (though it's also doing well), but surely help WhisperX. It did have issues with figuring number of speakers on it's own, but that wasn't a problem for my use case.

I've had good results with whisperx when I needed to generate captions. https://github.com/m-bain/whisperX

There is currently a problem with diarization, but otherwise, it is SOTA.

https://github.com/m-bain/whisperX/issues/569

WhisperX with the new model. It's not fast.

How much faster in real wall-clock time is this in batched data than https://github.com/m-bain/whisperX ?

Checkout whisperx

Neat to see a new implementation, although I'll note that for those looking for a drop-in replacement for the whisper library, I believe that both faster-whisper https://github.com/guillaumekln/faster-whisper and https://github.com/m-bain/whisperX are easier (PyTorch-based, doesn't require a web browser), and a lot faster (WhisperX is up to 70X realtime).

Hi HN! Since the launch of JigsawStack.com we've been trying to dive deeper into fully managed AI APIs built and fine tuned for specific use cases. Audio/video transcription was one of the more basic things and we wanted the best open source model and at this point it is OpenAI's whisper large v3 model based on the number languages it supports and accuracy.

The thing is, the model is huge and requires tons of GPU power for it to run efficiently at scale. Even OpenAI doesn't provide an API for their best transcription model while only providing whisper v2 at a pretty high price. I tried running the whisper large v3 model on multiple cloud providers from Modal.com, Replicate, Hugging faces dedicated interface and it takes a long time to transcribe any content about ~30mins long for 150mins of audio and this doesn't include the machine startup time for on demand GPUs. Keeping in mind at JigsawStack we aim to return any heavy computation under 25s or 2mins for async cases and any basic computation under 2s.

While exploring Replicate, I came across this project https://github.com/Vaibhavs10/insanely-fast-whisper by Vaibhav Srivastav which optimises the hell out of this whisper large v3 model with a variety of techniques like batching and using FlashAttention 2. This reduces computation time by almost 30x, check out the amazing repo for more stats! Open source wins again!!

First we tried using Replicates dedicated on-demand GPU service to run this model but that did not help, the cold startup/booting time alone of a GPU made the benefits of the optimised model pretty useless for our use case. Then tried Hugging face and modal.com and we got the same results, with a A100 80GB GPU, we were seeing around an average of ~2mins start up time to load the machine and model image. It didn't make sense for us to have a always on GPU running due to the crazy high cost. At this point I was inches away from giving up.

Next day I got an email from Fly.io: "Congrats, Yoeven D Khemlani has GPU access!" I totally forgot the Fly started providing GPUs and I'm a big fan of their infra reliability and ease to deploy. We also run a bunch of our GraphQL servers for JigsawStack on Fly's infra!

I quickly picked up some Python and Docker by referring to a bunch of other Github repos and Fly's GPU tutorials then wrote the API layer with the optimised version of whisper 3 and deployed on Fly's GPU machines.

And wow the results were pretty amazing, the start up time of the machine on average was ~20 seconds compared to the other providers at ~2mins with all the performance benefits from the optimised whisper. I've added some more stats in the Github repo. The more interesting thing to me is cost↓

Based on 10mins of audio:

There's a better parallel/batching that works on the 30s chunks resulting in 40X. From HF at https://github.com/Vaibhavs10/insanely-fast-whisper

This is again not native PyTorch so there's still room to have better RTFX numbers.

Founder of Replicate here. We open pull requests on models[0] to get them running on Replicate so people can try out a demo of the model and run them with an API. They're also packaged with Cog[1] so you can run them as a Docker image.

Somebody happened to stumble across our fork of the model and submitted it. We didn't submit it nor intend for it to be an ad. I hope the submission gets replaced with the upstream repo so the author gets full credit. :)

[0] https://github.com/Vaibhavs10/insanely-fast-whisper/pull/42