bark VS piper

!pip install git+https://github.com/suno-ai/bark.git

To my knowledge, the model being used for this is "chirp" which is 'based on' bark[1], an AI text to speech model.

The github page for bark links to a page about chirp, which returns a 404 page for me [2]. that the model for suno.ai's song generator isn't too much different than the text to speech model.

My hunch is that it was something like a coincidence that the bark model was capable of producing music, and that was spun off into this product. Unfortunately, there seems to still be issues with bark when generating long (like book length) spoken audio. Which is too bad, as someone who's worked jobs that require lots of driving, it would be awesome to be able to have any text read to me in a natural sounding voice.

[1]https://github.com/suno-ai/bark

Stable-audio and MusicGen sounds better than Jukebox.

But the best so far is Suno.ai ( https://app.suno.ai ) especially with their V3 model they have very impressive results, the fidelity is not studio quality but they're getting very close.

It's very likely based on their TTS model they have released before Bark, but trained on more data and with higher resolution.

https://github.com/suno-ai/bark

https://github.com/suno-ai/bark

> Bark was developed for research purposes. It is not a conventional text-to-speech model but instead a fully generative text-to-audio model, which can deviate in unexpected ways from provided prompts. Suno does not take responsibility for any output generated. Use at your own risk, and please act responsibly.

I've generated probably >200 songs now with Suno, of which perhaps 10 have been any good, and I can't detect any pattern in terms of the outputs.

Here's another one which is pretty good. I accidentally copied and pasted the prompt and lyrics, and it's amazing to me how 'musically' it renders the prompt:

hahah wow! cool :-)

PS: OT, I am reading this Bark thing(https://github.com/suno-ai/bark). Can I run it locally on a Macbook 2015 with 8GB RAM?

I have it locally. The model is on huggingface. It runs with about 8GB VRAM.

suno-ai/bark

This is so great--both in terms of the project & the write-up. Thanks for sharing your work! :)

A "quick" dump of some thoughts it provoked:

(1) Really like the "meaning-nonsense continuum" concept--and that neither extreme is explicitly labelled in the image immediately following where the term is introduced. :)

(And, yes, "Gravity learns about regret." is indeed kind of beautiful.)

(Aside: Adding alt text to the generated images would be beneficial both for copy/pasting & accessibility reasons. :) )

(2) This statement made me smile: "If you don’t enjoy reading and contemplating these sentences like this, we are simply very different people." Because while I very much fall into the category of "enjoy reading and contemplating these sentences" I can also imagine people who definitely don't. :D

(3) Part of the reason for (2) was because I'd already encountered a couple of the "don't work" examples where my immediate thought was "wait, how about in this situation though?". e.g. "Aligning with compass."

Which then got me thinking about stylistic, metaphor or time-period related aspects of grammar, e.g. pirate treasure map instructions written in cryptic sentence fragments.

(4) Which leads into the whole "procedurally generated diagrams/documents" aspect of the project that I also think is particularly cool. It immediately got me thinking of being in a game and walking into an inventor character's lab or a magician's library and finding all manner of technical or arcane drawings generated for either environment or game mechanic purposes (e.g. in a detective/mystery game like "Shadows of Doubt").

Then again, since I was a kid I've always had an interest in browsing technical component and, err, office stationery catalogues, so maybe it's just me. :D

It also made me think back to "Interminal" an entry to the 2020 ProcJam procedural generation game jam, which generated an airport terminal including duty-free stores with procedurally generated perfumes (including "name, visual identity and smell"): https://nothke.itch.io/interminal

(5) The "physicalization" aspect of the project to render it in a tangible form is also cool--particularly because (at least to me but maybe also to all those vinyl-owning kids with no turntable :D ) a physical form seems to the amplify the "meaningfulness" of a message...

The underlying "digitalized cursive handwriting" project was also of interest--though I only skimmed its write-up in an attempt to avoid yet another rabbit hole. :)

The handwriting system seemed like it might also be amenable to animated use--which then made me think, "Wait! I've seen something about that recently...", had no idea what, but then some background retrieval process just produced the answer as I was writing, it was Noclip's recent "The Making of Pentiment" documentary: https://www.youtube.com/embed/ffIdgOBYwbc

(6) With regard to the technical execution/implementation side of things: I've observed that a project's need for text generation often raises a question of what implementation approach to use, in terms of a DIY vs pre-existing solution.

One issue which affects text generation specifically is "general solutions" often seem to tend toward specialization over time (e.g. "text generation for interactive fiction", "text generation for branching narrative-driven games", tools such as Yarn Spinner[0] & Ink[1]), which ends up making the solutions less suitable for simpler/different use cases & increases difficulty of the learning curve.

This was something I ran into during a small sub-project[2] last year where the text generation was somewhat "incidental" to overall project goal. I started out with a "quick DIY" solution but still ended up spending enough time on that aspect that I started to wonder if I'd be better not entirely re-inventing the wheel.

Around this time I ran into the "Blur Markup Language"[3][4][5] project which has the tag line "write text that changes" and--while I haven't yet used it--seems like it might be a promising "mid-level abstraction" solution for text generation, so thought I'd mention it as a potential option for others with text generation needs.

(7) In terms of other helpful text generation related resources, I've found various "word lists" to be of use, so thought I'd mention this "weasel words" list as a starting point: https://github.com/words/weasels/blob/main/data.txt#L14

(The repo README also links to other word lists under the same org including word categories such as "buzzwords", "filler", "hedges" & words listed in order of associated positive/negative sentiment.)

Thanks again for sharing your work & look forward to seeing where your projects go in future, should you share more in future. :)

---- footnotes ----

[0] https://github.com/YarnSpinnerTool/YarnSpinner

[1] https://github.com/inkle/ink

[2] I wanted to generate "scripted dialogue" samples[2a] to demonstrate the 900+ individual Text-To-Speech speaker voices in the Piper TTS[2b] LibriTTS voice model[2c], in a form that is: useful for evaluating the voices; not incredibly tedious to listen to; and, makes it possible to identify which speaker you are currently hearing.

[2a] Subset of resulting generated[2d] speech output can be heard in the second example here: https://rancidbacon.gitlab.io/piper-tts-demos/

[2b] https://github.com/rhasspy/piper

[2c] https://huggingface.co/rhasspy/piper-voices/blob/main/en/en_...

[2d] Text generation script: https://gitlab.com/RancidBacon/larynx-dialogue/-/blob/featur...

[3] https://bml-lang.org

[4] BML intro/overview: https://bml-lang.org/docs/guide/language-basics/

[5] Online BML editor with syntax cheat sheet: https://bml-lang.org/sandbox/

My interest in TTS is around "indie" game creation, animation and "radio plays".

A couple of years ago I started development of a tool to help with the generation of game audio such as NPC dialogue, "barks" or narration for those without access to/budget for human voice actors: https://rancidbacon.itch.io/dialogue-tool-for-larynx-text-to...

One thing I found interesting is that writing a small "scene" and then hearing dialogue being spoken by a variety of voices often prompted the writing of further lines of dialogue in response to perceived emotion contained in voices in the generated output. Plus it was just fun. :)

The version of the tool on that page is based on Larynx TTS which has continued development more recently as Piper TTS: https://github.com/rhasspy/piper

I'm yet to publish my port which uses Piper TTS though: https://gitlab.com/RancidBacon/larynx-dialogue/-/tree/featur...

Though I did upload some sample output (including some "radio announcer" samples in response to a HN comment :) ): https://rancidbacon.gitlab.io/piper-tts-demos/

Obviously there's variations in voice quality, and ability to control expression is currently limited but beats hearing my own voice. :D

Mozilla's browser tts is kind of not bad, just parse and buffer one sentence at a time and it does all right.

For the backend, I've experimented with piper, which has a lot of voices and accents, though it's tricky to buffer and sync long texts.

https://github.com/rhasspy/piper

After some brief research it seems the issue you're seeing may be a known bug in at least some versions/release of espeak-ng.

Here's some potentially related links if you'd like to dig deeper:

* "questions about mandarin data packet #1044": https://github.com/espeak-ng/espeak-ng/issues/1044

* "ESpeak NJ-1.51’s Mandarin pronunciation is corrupted #12952": https://github.com/nvaccess/nvda/issues/12952

* "The pronunciation of Mandarin Chinese using ESpeak NJ in NVDA is not normal #1028": https://github.com/espeak-ng/espeak-ng/issues/1028

* "When espeak-ng translates Chinese (cmn), IPA tone symbols are not output correctly #305": https://github.com/rhasspy/piper/issues/305

* "Please default ESpeak NG's voice role to 'Chinese (Mandarin, latin as Pinyin)' for Chinese to fix #12952 #13572": https://github.com/nvaccess/nvda/issues/13572

* "Cmn voice not correctly translated #1370": https://github.com/espeak-ng/espeak-ng/issues/1370

If you're not already aware, the primary developer of Mimic 3 (and its non-Mimic predecessor Larynx) continued TTS-related development with Larynx and the renamed project Piper: https://github.com/rhasspy/piper

Last year Piper development was supported by Nabu Casa for their "Year of Voice" project for Home Assistant and it sounds like Mike Hansen is going to continue on it with their support this year.

Coqui-ai was a commercial continuation of Mozilla TTS and STT (https://github.com/mozilla/TTS).

At the time (2018-ish), it was really impressive for on-device voice synthesis (with a quality approaching the Google and Azure cloud-based voice synthesis options) and open source, so a lot of people in the FOSS community were hoping it could be used for a privacy-respecting home assistant, Linux speech synthesis that doesn't suck, etc.

After Mozilla abandoned the project, Coqui continued development and had some really impressive one-shot voice cloning, but pivoted to marketing speech synthesis for game developers. They were probably having trouble monetizing it, and it doesn't surprise me that they shut down.

An equivalent project that's still in active development and doing really well is Piper TTS (https://github.com/rhasspy/piper).

There isn't an ElevenLabs app like that, but I think that's the most expedient method, by far.

(details and warning: in-depth, opinionated take, written almost for my own benefit, I've done a lot of work near here recently but haven't had to organize my thoughts until now)

Why? Local inference is hard. You need two things: the clips to voice model (which we have here, but bleeding edge), and text + voice -> speech model.

Text to voice to speech, locally, has excellent prior art for me, in the form of a Raspberry Pi-based ONNX inference library called [Piper](https://github.com/rhasspy/piper). I should just be able to copy that, about an afternoon of work!

Except...when these models are trained, they encode plaintext to model input using a library called eSpeak. eSpeak is basically f(plaintext) => ints representing phonemes. eSpeak is a C library and written in a style I haven't seen in a while and depends on other C libraries. So I end up needing to port like 20K lines of C to Dart...or I could use WASM, but over the last year, I lost the ability to be able to reason through how to get WASM running in Dart, both native and web.

It's a really annoying technical problem: the speech models all use this eSpeak C library to turn plaintext => model input (tokenized phonemes).

Re: ElevenLabs

I had looked into the API months ago and vaguely remembered it was _very_ complete.

I spent the last hour or two playing with it, and reconfirmed that. They have enough API surface that you could build an API that took voice recordings, created a voice, and then did POSTs / socket connection to get audio data from that voice at will.

Only issue is pricing IMHO, $0.18 for 1000 characters. :/ But this is something I feel very comfortable saying wouldn't be _that_ much work to build and open source with a "bring your own API key" type thing. I had forgotten about Eleven Labs till your post, which made me realize there was an actually meaningful and quite moving use case for it.

piper (https://github.com/rhasspy/piper)

I have been using piper-tts and it is GREAT and super lightweight / easy to use. On a 2080 I'm sure you can use the HQ models no worries!