espnet VS kaldi-gstreamer-server

You might check out this list from espnet. They list the different corpuses they use to train their models sorted by language and task (ASR, TTS etc):

https://github.com/espnet/espnet/blob/master/egs2/README.md

- https://github.com/espnet/espnet

This link has instructions for doing this for a Japanese model. It would have to be altered to work with ljspeech and the fine tune dataset.

This work is made possible by the excellent advancements in text to speech modeling. ESPnet is a great project and should be checked out for more advanced and a wider range of use cases. This pipeline was also made possible by the great work from espnet_onnx in building a framework to export models to ONNX.

CMU WavLab has ESPNet https://espnet.github.io/espnet/ which includes a number of high quality TTS models including VITS (which in my subjective experience is just as good as what is demonstrated here). Also the inference on various ESPNet pretrained TTS models is reasonable and sentences take on average 5 seconds per word to generate the waveform on my totally mid PC setup.

The reason I'm saying this is to point out that having and in-depth knowledge on speech processing/generation requires a lot of information about signal processing and human speech in general (eg. acoustics and phonetics). However, if you're not into learning everything there is to know about a subject, just take one state-of-the-art example and study that as best as you can. Pick one environment/toolkit, for example espnet and simply go with that.

https://github.com/espnet/espnet (kind of like a newer Kaldi, but also not beginner friendly)

both provide e2e ASR support but espnet does have more utilities where as speechbarain is clean

This is relatively small amount of speech to train the model from scratch, but you can train using another pre-trained model for initialization. There are numbers of end-to-end ASR toolkits which can be used for this: https://github.com/NVIDIA/NeMo and https://github.com/espnet/espnet

Through my research, the most promising real-time transcription options appear to be Vosk or Kaldi Gstreamer. I’ve set them both up & they appear to work well for general transcription, but I’m not sure how to handle the user-specific custom vocabularies.

It is kind of difficult to find something like this free of charge (and open source) since the ASR service needs to be hosted somewhere. If you are really interested in the topic then you could take a lit into kaldi and its pretrained models (but kaldi is kind of difficult to learn so I don't really recommend it if you want something quick) and then you could also combine that with kaldi-gstreamer in order to set up a server which you can turn on and off whenever you like.