Sushi VS alass

I generally use mkvmerge. And for copying over subtitles to a new file, I would recommend using the tool sushi first. It analyzes the audio tracks and adjusts the subtitle timings as necessary to make sure they properly sync (File A might start a few seconds earlier than File B, etc. )

there's automatic subtitle synchronization tools like: https://github.com/tp7/Sushi

Sushi

Try alass or ffsubsync.

I use https://github.com/kaegi/alass to retime subtitles automatically. I've never worked with a file that has the issue you're describing but it does claim to fix frame rate issues.

You need an english softsubbed source, JP subs (kitsuneko), sync them with https://github.com/kaegi/alass, then set up POTPlayer for multiple subtitle files.

Check this project here: https://github.com/kaegi/alass

If you want subtitles for anime, kitsunekko usually has subtitles (and Netflix also usually has native subtitles and you can download them using subadub). The downside with using subtitles from a different source is you need to re-time them but tools like alass make this basically painless in most cases. For movies, Netflix or the DVD version of movies usually have subtitles.

I've used https://github.com/kaegi/alass and sometimes it works, sometimes it doesn't. It's not an easy problem to address automatically...

Video players wouldn't want that in their codebase, maybe if it was available as a C library.

With tools like alass[1] (using it to synchronise against the original language subtitles) it is about as close to solved you can get.

All of the attempts I've seen of using audio information to synchronise subtitles have been awful (some languages subtitle everything, even screams and incoherent shouts -- such as Japanese -- while others only subtitle dialogue and often rework dialogue for the purposes of making the subtitles short enough to be readable easily).

[1]: https://github.com/kaegi/alass