audio_alignment VS gentle

This is a great idea; I had done something like this a couple of years ago for some specific set of videos (e.g. https://shreevatsa.net/tex/program/videos/s10/ — compare with https://youtubetranscript.com/?v=_0Cv1G_s4gQ for the same video), but never got around to making it general; glad someone has done it. It takes just a few lines of Javascript, using the Youtube API, to do this i.e. keeping the video and text in sync (just view source on either page to see the JS at the bottom).

Something like this can also help with audio recordings (generating the alignment automatically is called "forced alignment" and there are tools like "aeneas" for this). In case anyone's interested or wants to help (for Sanskrit texts): see https://github.com/shreevatsa/web-align-audio-text deployed at https://shreevatsa.net/ramayana/sarga/ and better version at https://github.com/avinashvarna/audio_alignment deployed at https://avinashvarna.github.io/audio_alignment/

Gentle actually works on Linux, there's just no pre-built binary: https://github.com/lowerquality/gentle

I just found Gentle yesterday and it looks like it might be a free tool to do what you’re looking for: https://lowerquality.com/gentle/

Thank you!

Yes, exactly. We do forced alignment when you edit your transcript. The new words don't have any timestamps, so we need to align them. For short sections we use interpolation. If we need align whole sections we use Gentle[^1].

[^1]: https://github.com/lowerquality/gentle

Gentle also has web interface, only for English. Uses DNN acoustic models.

I found this old wiki page about the LIP file format used in Fallout 2: https://falloutmods.fandom.com/wiki/LIP_File_Format. FO2 != SSE, but it seems like a LIP file is primarily: what phonemes to use, and when. That could be pretty easily generated by a modern tool like Gentle (built on Kaldi). The trick would then be to tranform Gentle's output to whatever Skyrim expects in a LIP.

This is neat, and immediately made me think of the annotations that show up when you hit the play button on https://lowerquality.com/gentle/ , but it turns out those are made with absolutely-positioned divs and a lot of offline-precalculated px math.

More info: I used gentle to convert the audio (and a text document with what was said) into a list of phonemes and the times at which they were said. I then used a Python script inside Blender to convert this list into an animation by smoothly gliding to whichever shape key has the current phoneme in its name.

Gentle by lowerquality didn't help? (https://github.com/lowerquality/gentle) It returns time aligned phoneme sequences for each word, like 'ice' -> 'ai': t0, 's': t1. I suppose it doesn't tell you which exact letters are paired, but it matches individual words with phonemes using a set vocabulary, the CMU one. (http://www.speech.cs.cmu.edu/tools/lextool.html)

We’ve had good results with gentle force alignment. https://github.com/lowerquality/gentle