syncabook VS aeneas

Amazing! I've made a similar ebooks-audiobooks aligner years ago: https://github.com/r4victor/syncabook. At that time, I chose to synthesize the text and align two audio sequences because I found texts-alignment approaches (including ML-based ones) too compute-intensive and inadequate for long texts. I see Storyteller works by aligning the texts. Could you give some view on how long it takes to sync a book?

Also, my experience was that audio and text versions are often very different (e.g. the audio having an intro missing from the text). It'd very interesting to know how well Storyteller handles such cases. Does it require manual audio/text editing or handle the differences automatically?

Unfortunaly, this audiobook isn't synchorized to the text on Kindle. So, in order to synchronize the epub and audio myself, initially, I tried using syncabook, but the results weren't great for some reason. I then tried using OpenAI's Whisper to generate the text from the audio. This might've given decent results, but ultimately I decided I wanted to keep the book's formatting (paragraphs, emphasis, etc.)

I've tried syncabook, but that didn't help. I've tried whisperX, to get word-level timings, but the results are pretty bad/unusable.

From searching a bit I found https://github.com/r4victor/syncabook this might help anyone intrigued about the idea. I suspect this is the tool you're talking about? Your post is really a long form question, and could have been a lot shorter.

This software can generate epubs with audio completely automatically for a lot of simpler languages. The problem is it doesn't support Japanese.

You could try aeneas. Syncabook apparently uses the afaligner library, which says that it was inspired by aeneas but uses FastDTW to find an approximation to the optimal warping path. This might make it slightly less accurate than aeneas.

Check out https://github.com/readbeyond/aeneas

In case you have let's say: a 20min entry from an audio book, and the sentences seperatly in a txt file and you want to cut the sentences out of the audio manually you can look at a tool like aeneas. If you still have to annotated all your data yourself i do not really know a tool for this :/

here's another solution: https://github.com/readbeyond/aeneas