[Doc] Official code from FrontierLabs for the paper "F5R-TTS: Improving Flow Matching based Text-to-Speech with Group Relative Policy Optimization"

[SaibaLucky]

Checks

• This template is only for research question, not usage problems, feature requests or bug reports.
• I have thoroughly reviewed the project documentation and read the related paper(s).
• I have searched for existing issues, including closed ones, no similar questions.
• I am using English to submit this issue to facilitate community communication.

Question details

Hi,

I recently came across the paper "F5R-TTS: Improving Flow Matching based Text-to-Speech with Group Relative Policy Optimization," https://arxiv.org/pdf/2504.02407 which builds upon and improves the existing codebase. The work in the paper seems closely related to this repository.

I was wondering if there are any plans to reproduce the code from this paper?

Looking forward to your response. Thank you!

[hcsolakoglu]

I hadn't heard of this paper yet. I'll consider adding it when I have free time.

[hcsolakoglu]

I used ai to take a quick look at the repo and paper. Phase 1 and Phase 2 are straightforward. Phase 3 looks doable, but it will take some time. I don't have capacity for this right now. I'll see what I can manage when I get some free time.

[fakerybakery]

Would also be interested in seeing an open reproduction of the paper

[anonymousme]

I'm from the research group of F5R-TTS, and we have already planned to open-source the code, which will be released soon. Thank you for your attention.

@SaibaLucky @hcsolakoglu @fakerybakery

[hcsolakoglu]

I'm from the research group of F5R-TTS, and we have already planned to open-source the code, which will be released soon. Thank you for your attention.

@SaibaLucky @hcsolakoglu @fakerybakery

Alright, thanks! I would appreciate it if you tag me when you open your codebase.

[fakerybakery]

Hi @anonymousme, is there any update on the release of F5R-TTS?

[liuxin99]

hi, @anonymousme , Could you please explain how the sample-level sampling probability πθ as shown in the formula is calculated?

[anonymousme]

Hi @anonymousme, is there any update on the release of F5R-TTS?

@fakerybakery We are organizing and standardizing the codes for better understanding and usage. I will tag you once the code is released~

[anonymousme]

hi, @anonymousme , Could you please explain how the sample-level sampling probability πθ as shown in the formula is calculated?

As mentioned in the paper, the output of the model prediction consists of the mean μ and variance σ² of a Gaussian distribution. By sampling a value x from this Gaussian distribution, the corresponding probability density πθ of x can be described by the following formula.

[liuxin99]

hi, @anonymousme , Could you please explain how the sample-level sampling probability πθ as shown in the formula is calculated?

As mentioned in the paper, the output of the model prediction consists of the mean μ and variance σ² of a Gaussian distribution. By sampling a value x from this Gaussian distribution, the corresponding probability density πθ of x can be described by the following formula.

@anonymousme Thank you very much for your reply!

So, judging from the loss function, are you implementing on-policy RL training?

Additionally, I have another question: You mentioned that 100 hours of data were sampled from the training set for GRPO training. However, when calculating the Speaker Similarity reward, shouldn't the input consist of a prompt audio and a prompt text? How do you organize the training data for this specific scenario? Specifically, does the training data include pairs of utterances from the same speaker but with different content?

It seems that there is no prompt in the input condition...🤔

[anonymousme]

hi, @anonymousme , Could you please explain how the sample-level sampling probability πθ as shown in the formula is calculated?

As mentioned in the paper, the output of the model prediction consists of the mean μ and variance σ² of a Gaussian distribution. By sampling a value x from this Gaussian distribution, the corresponding probability density πθ of x can be described by the following formula.

@anonymousme Thank you very much for your reply!

So, judging from the loss function, are you implementing on-policy RL training?

Additionally, I have another question: You mentioned that 100 hours of data were sampled from the training set for GRPO training. However, when calculating the Speaker Similarity reward, shouldn't the input consist of a prompt audio and a prompt text? How do you organize the training data for this specific scenario? Specifically, does the training data include pairs of utterances from the same speaker but with different content?

It seems that there is no prompt in the input condition...🤔

1. Yes, we are implementing on-policy RL training.
2. During training, a pair of audio and text is given. ​x_gt​ represents the audio, and ​text_gt​ is the corresponding text for this audio. We randomly select a small segment from the ​beginning​ of ​x_gt​ as the prompt audio and mask the remaining part. The model then predicts the masked part, and speaker similarity is calculated between the predicted audio and the ground truth audio.
3. There are some utterances from the same speaker but with different content.

[yiwei0730]

@anonymousme When is the release date of F5R-TTS expected? Thank you!

[anonymousme]

The code of F5R-TTS is already released at https://github.com/FrontierLabs/F5R-TTS. Thank you for your attention.
@SaibaLucky @hcsolakoglu @fakerybakery @liuxin99 @yiwei0730