SpeechT5
Unified-Modal Speech-Text Pre-Training for Spoken Language Processing (by microsoft)
AudioMAE
This repo hosts the code and models of "Masked Autoencoders that Listen". (by facebookresearch)
| SpeechT5 | AudioMAE | |
|---|---|---|
| 4 | 1 | |
| 1,044 | 480 | |
| 6.7% | 5.6% | |
| 7.1 | 0.6 | |
| 23 days ago | about 1 month ago | |
| Python | Python | |
| MIT License | GNU General Public License v3.0 or later |
The number of mentions indicates the total number of mentions that we've tracked plus the number of user suggested alternatives.
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
SpeechT5
Posts with mentions or reviews of SpeechT5.
We have used some of these posts to build our list of alternatives
and similar projects. The last one was on 2023-04-19.
-
[HELP] Speech2Speech translator with speaker voice preservation
Hey! I’m doing a somewhat similar project but for TTS / voice cloning. This might not be too relevant for you but it might be one way to solve your problem. We based our project onSpeecht5 which is a multimodal setup that can take in audio or text and output audio or text. It uses speaker embeddings to handle multiple speakers, so you could use Metas S2ST to translate audio and this model to preserve the voice by doing audio to audio speech conversion. Here’s a hugging tutorial which mentions speech conversion with speecht5 https://huggingface.co/blog/speecht5
- Nvidia Text2Video
- Foundation models for speech analysis/synthesis/modification
-
[2210.03730] SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training
The idea to separate text from speech is important. Models released today: https://github.com/microsoft/SpeechT5/tree/main/SpeechUT SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training
AudioMAE
Posts with mentions or reviews of AudioMAE.
We have used some of these posts to build our list of alternatives
and similar projects. The last one was on 2023-04-11.
-
Foundation models for speech analysis/synthesis/modification
Models that seem promising are facebook's Audio-MAE, and laion's CLAP. But I'm not finding any use of them in the wild. What gives?
What are some alternatives?
When comparing SpeechT5 and AudioMAE you can also consider the following projects:
PaddleSpeech - Easy-to-use Speech Toolkit including Self-Supervised Learning model, SOTA/Streaming ASR with punctuation, Streaming TTS with text frontend, Speaker Verification System, End-to-End Speech Translation and Keyword Spotting. Won NAACL2022 Best Demo Award.
CLAP - Contrastive Language-Audio Pretraining
espnet - End-to-End Speech Processing Toolkit
Awesome-Video-Diffusion - A curated list of recent diffusion models for video generation, editing, restoration, understanding, etc.
NeMo - A scalable generative AI framework built for researchers and developers working on Large Language Models, Multimodal, and Speech AI (Automatic Speech Recognition and Text-to-Speech)