Chi Kim

5 months ago

Chi Kim
5 months ago

F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching! The quality is pretty impressive for open source, and it even supports mps for Mac! I was able to get it going on my Mac with no problem. #TTS #ML #AI
github.com/SWivid/F5-TTS
@ZBennoui

reshared this

in reply to Chi Kim

Winter blue tardis🇧🇬🇭🇺

in reply to Chi Kim 5 months ago

This sounds really interesting. :)

in reply to Chi Kim

Luis Carlos

in reply to Chi Kim 5 months ago

@mush42 Could this be integrated on Sonata? I believe not

@Musharraf

in reply to Luis Carlos

Musharraf

in reply to Luis Carlos 5 months ago

@luiscarlosgonzalez
Too heavy for local use. Unless you have a GPU with at least 8GB of VRAM. Or, you know, a Mac.

@Luis Carlos

in reply to Musharraf

Luis Carlos

in reply to Musharraf 5 months ago

@mush42 I will have a computer with that GPU later, so my exptectation is almost high

@Musharraf

in reply to Chi Kim

the esoteric programmer

in reply to Chi Kim 5 months ago

what's this, more precisely? is this some kind of tts? how does it sound like? I tryed to understand something from the github, but it's kinda flying above my head

in reply to the esoteric programmer

Musharraf

in reply to the esoteric programmer 5 months ago

@esoteric_programmer
It converts the following text:
Every time I see someone light up, um, because of something I’ve made, it’s like, wow, a little piece of my inner child gets healed, you know? And, um, when...snip

To the attached speech.

@the esoteric programmer

Tamas G reshared this.

in reply to Musharraf

Musharraf

in reply to Musharraf 5 months ago

@esoteric_programmer
You can easily plug this into an open-source LLM and get something akin to NotebookLM.
Totally free and open-source, with very high quality.

@the esoteric programmer

in reply to Musharraf

the esoteric programmer

in reply to Musharraf 5 months ago

so, this does text completion and then generates speech using something like tts? is that correct so far? or do you attach audio of something, the model transcribes it and gets its meaning in whatever way that's considered meaning anyway, then concatenates your prompt text to that? that could create so, so many deepfakes, it's not even funny, if what I'm imagining is actually what's happening

in reply to the esoteric programmer

Musharraf

in reply to the esoteric programmer 5 months ago

@esoteric_programmer
Other than the text completion part, you are almost correct.
You give it some text, and an audio sample, and it tries to replicate the given voices characteristics.
Research is active in the areas of speaker verification and audio deep fake detection to combat misuse.

@the esoteric programmer

in reply to Musharraf

the esoteric programmer

in reply to Musharraf 5 months ago

aha, interesting! could I make it generate, say, something like a podcast? would I have to generate each part of the dialog in turn, then splice replies into the result?

in reply to the esoteric programmer

Musharraf

in reply to the esoteric programmer 5 months ago

@esoteric_programmer
Yes that's how you'd do it.

@the esoteric programmer

in reply to the esoteric programmer

Musharraf

in reply to the esoteric programmer 5 months ago

@esoteric_programmer
But again, can't you say the same thing about ElevenLabs? And other voice conversion tech?

@the esoteric programmer

in reply to Musharraf

the esoteric programmer

in reply to Musharraf 5 months ago

I dk about 11 labs and such, I don't use those, don't intend to either, but this seemns like it'd be explicitly used for such a thing, by a lot and a lot of people, much easier than with 11 labs. Yeah, I could just be imagining this wrong and blowing it out of proportion in my mind, that's always a possibility

in reply to the esoteric programmer

Erion

in reply to the esoteric programmer 5 months ago

The short answer to this is, anything can be misused. Someone could create fake clips, but someone could also create an audio book narrated by their favorite narrator for personal use, or someone could create an app that uses a loved one's lost voice if they only have a few audio clips as well.

in reply to Musharraf

Zach Bennoui

in reply to Musharraf 5 months ago

@mush42 @esoteric_programmer While the model is really really good, I find it has problems when trying to convert text that's more than a few lines. It will start splicing parts of the audio prompt into the result or just go kinda insane. I've only tried the HF space so far, but want to try runningn on my Mac tomorrow to see if I can get better results. If I could get this to read articles to me that would be great.

@Musharraf @the esoteric programmer