Artifacts and Noise in Fine-Tuned Model

[tranquangchung]

Checks

• This template is only for usage issues encountered.
• I have thoroughly reviewed the project documentation but couldn't find information to solve my problem.
• I have searched for existing issues, including closed ones, and couldn't find a solution.
• I am using English to submit this issue to facilitate community communication.

Environment Details

I fine-tune the model based on your guidance.
However, my fine-tuned model has many artifacts and noise.
You can listen to some samples in the attached file.
Mec-spectrogram is so different, with pretrained, it's so clear, but finetune is not

Do you have any suggestions to fix it ?
I use the default parameter of your config, change learning rate only: 1e-4

F5TTS.zip

Steps to Reproduce

I fine-tune the model based on your guidance.
However, my fine-tuned model has many artifacts and noise.
You can listen to some samples in the attached file.
Mec-spectrogram is so different, with pretrained, it's so clear, but finetune is not

Do you have any suggestions to fix it.
I use the default parameter of your config, change learning rate only: 1e-4

✔️ Expected Behavior

No response

❌ Actual Behavior

No response

[SWivid]

How is the sample_rate of your dataset used to finetune (and also inference).

From our experience, using 16khz or lower audio sample to train or infer, the model will produce certain artifact in high frequency area. As the model is based on 24khz, the phenomenon is somehow reasonable, and thus we encourage using 24khz dataset.

Some further suggestions: not sure what batchsize type you used, but we typically use "frame"; and the auto slicing with finetune-gradio ui when preparing dataset is rough (which might cut samples too short), you could organize a finer dataset with better pipeline.

[tranquangchung]

Thanks for your suggestion.

I fine-tuned the model with 24khz, the quality is good now.
However, some serious errors also appear, such as missing and wrong words. You can check some samples in the attachment.
All samples are synthesized with the same sentence
"To be yourself in a world that is constantly trying to make you something else is the greatest accomplishment"

The dataset used to train is LibrisTTS-960(500, 360, 100). I use our phoneme instead of the character or IPA that you shared.
I think that the problem comes from misalignment between input (phoneme or text) and mel-spectrogram.

Do you know the reason and have any suggestions for this problem?
I tried some solutions from #85, such as trimming and adding silence to audio, but the error still appears.

f5tts.zip

[SWivid]

• the maximum sample length in libritts is 20s, so when inference, need to ensure ref+gen < 20s.
• would suggest using char, you can refer to the appendix in our paper where we give detailed evaluation for model trained on libritts .

[tranquangchung]

Hi

1. During the inference process, I always set ref+gen < 20s, but word skipping still occurs.
2. We are fine-tuning the model with both characters and phonemes. However, the speech synthesis output still skips many words. This is a severe problem.

Do you have any further suggestions?

[SWivid]

How is the global batchsize and training steps, are you using our training codes here and only changed the part of text tokenizer?

---

As much training details of you experiments are blind to us, there could be many variables.
I would suggest you to train from scratch with our repo's framework and recipe (e.g. small model on libritts), which will guarantee 100% the same results as reported in our paper. After correct reproduction, you could do whatever modifications you want (e.g. mixed use of char and phonemes) and easily make ablations on that.

[tranquangchung]

hydra:
run:
dir: ckpts/${model.name}${model.mel_spec.mel_spec_type}${model.tokenizer}_${datasets.name}/${now:%Y-%m-%d}/${now:%H-%M-%S}

datasets:
name: "English" # dataset name
batch_size_per_gpu: 19200 #38400 #19200 # 8 GPUs, 8 * 38400 = 307200
batch_size_type: frame # frame # frame | sample
max_samples: 64 # max sequences per batch if use frame-wise batch_size. we set 32 for small models, 64 for base models
num_workers: 32

optim:
epochs: 1000
learning_rate: 1e-4 # initial learning rate
num_warmup_updates: 20000 # warmup updates
grad_accumulation_steps: 1 # note: updates = steps / grad_accumulation_steps
max_grad_norm: 1.0 # gradient clipping
bnb_optimizer: False # use bnb 8bit AdamW optimizer or not

model:
name: F5TTS_v1_Base # model name
tokenizer: 'char' # tokenizer type, char, phoneme, custom, or null
backbone: DiT
arch:
dim: 1024
depth: 22
heads: 16
ff_mult: 2
text_dim: 512
text_mask_padding: True
qk_norm: null # null | rms_norm
conv_layers: 4
pe_attn_head: null
checkpoint_activations: False # recompute activations and save memory for extra compute
mel_spec:
target_sample_rate: 24000
n_mel_channels: 100
hop_length: 256
win_length: 1024
n_fft: 1024
mel_spec_type: vocos # vocos | bigvgan
vocoder:
is_local: False # use local offline ckpt or not
local_path: null # local vocoder path

ckpts:
logger: tensorboard # wandb | tensorboard | null
log_samples: True # infer random sample per save checkpoint. wip, normal to fail with extra long samples
save_per_updates: 10000 # save checkpoint per updates
keep_last_n_checkpoints: -1 # -1 to keep all, 0 to not save intermediate, > 0 to keep last N checkpoints
last_per_updates: 5000 # save last checkpoint per updates
save_dir: Path2save # save directory

---

I tested the save checkpoint from model_200000 to model_340000 (at this point)
Model is trained on A100 with 4 GPUs, 80G.
The zero-shot ability is quite good, but only problem is about intelligibility

[SWivid]

How is the dataset, are you using emilia or libritts?
From the config, your global batchsize is 1/4 of the one we used, could train longer to see.
If you're using libritts, you can use small model to do experiment.

[tranquangchung]

Actually, we fine-tuned the model with two datasets:

(1) LibrisTTS-960 → This model works quite well using your tokenizer (char), it sometimes skips words.
(2) LibrisTTS-960 + Vietnamese, with around 2000 speakers → It always skips words in both languages.

For (2), I trained both shared and separated char for both English and Vietnamese.

[SWivid]

Have you tried training from scratch with En+Vi? The released v1_base checkpoint learned a lot Mandarin, which takes much modeling ability but not needed in your use case.

[SWivid]

To achieve better alignment, there are some possible ways to try:

• https://arxiv.org/pdf/2506.13053 to use a larger text refinement module and do average upsample before concatenation
• https://arxiv.org/abs/2505.19595 with repa-like supervision