pyannote-audio VS demucs

pyannote/pyannote-audio

Open Source. I've found this to be pretty nice, which is just a wrapper on some hugging face models https://github.com/pyannote/pyannote-audio

(not the creator, but I've built something similar for personal use)

This is a great library for determining which speaker is speaking during each time in an audio file (this is called speaker diarization); I imagine they used it or something like it. Works really well out of the box!

https://github.com/pyannote/pyannote-audio

Do you think pyannote could be implemented in the Pro version of the app to support diarization?

Free startup idea: Use Whisper with pyannote-audio[0]’s speaker diarization. Upload a recording, get back a multi-speaker annotated transcription.

Make a JSON API and I’ll be your first customer.

[0] https://github.com/pyannote/pyannote-audio

Whisper can’t, but pyannote-audio can. I’ve seen a couple of prototypes out there which link the two together.

You can use pyannote python library. It will identify different speakers from audio and will create small audio files with those speakers.

This Colab notebook is basically a standard template (with small changes) provided by pyannote-audio, the library implementing the speaker diarization functionality we need. (template)

I've had the best results from Facebook's DEMUCs. It's not too difficult to install, and I like the sound quality of their mdx_extra model. This is the command line I use (this will use the 2 stem version -- vocals, and everything else)

facebookresearch/demucs: Stem seperation

For those interested, Facebook's Demucs page (https://github.com/facebookresearch/demucs) gives performance comparison for several models including open-unmix.

See also: https://www.stemroller.com This runs as a local app on Windows and Mac.

Demucs [1], one of the leading/SOTA systems, has an experimental 6-source model, `htdemucs_6s`, which adds piano and guitar:

>We are also releasing an experimental 6 sources model, that adds a guitar and piano source. Quick testing seems to show okay quality for guitar, but a lot of bleeding and artifacts for the piano source.

I also believe Audioshake [2] (a company in the space) is doing guitar separation as well.

1: https://github.com/facebookresearch/demucs

I don't know which one /u/DarkMemoria used exactly but I use demucs. If you go to the Colab section you can run it by putting the audio files you want to separate into Google Drive.

Demucs v4 is the best open source currently, that's also what the snippet from the video sounds like uses

already exist, for example I use demucs to separate songs into 6 tracks, and then I mute what i need to be silenced in any daw.

I regularly use demucs (https://github.com/facebookresearch/demucs). It might be overwhelming when you are not used to work with the terminal, but it's as good as all the wrapper sites that ask for payment. Also, there are probably GUI projects that makes it even easier.

There's also the open source software https://github.com/facebookresearch/demucs which I assume is what many of the free websites are using behind the scenes. There's a demo site here: https://huggingface.co/spaces/akhaliq/demucs but I haven't tested it for time limits/upload limits etc.

I used demucs to try and separate the vocals from the instrumental with fairly decent results. When playing the instrumental you can still vaguely hear some remnants of the vocals and I worry when it's played over a real sound system at the wedding it will be very obvious.