Unable to read the data I prepared

[Chloey-TS]

Checks

• This template is only for usage issues encountered.
• I have thoroughly reviewed the project documentation but couldn't find information to solve my problem.
• I have searched for existing issues, including closed ones, and couldn't find a solution.
• I am using English to submit this issue to facilitate community communication.

Environment Details

Manjaro Linux
Python 3.13.3
Torch 2.7.1+cu128
Gradio 5.35.0
GPU: RTX5070

Steps to Reproduce

1. create a virtual environment
2. download latest release and install as pip package
3. download checkpoints and vocos from hugging face, and modify infer_gradio.py line 54

DEFAULT_TTS_MODEL_CFG = [
    "/home/ai/Documents/AI Voice Clone/F5-TTS/ckpts/F5TTS_v1_Base/model_1250000.safetensors",
    "/home/ai/Documents/AI Voice Clone/F5-TTS/ckpts/F5TTS_v1_Base/vocab.txt",
    # "hf://SWivid/F5-TTS/F5TTS_v1_Base/model_1250000.safetensors",
    # "hf://SWivid/F5-TTS/F5TTS_v1_Base/vocab.txt",
    json.dumps(dict(dim=1024, depth=22, heads=16, ff_mult=2, text_dim=512, conv_layers=4)),
]

and utils_infer.py line 104, to run it locally, try f5-tts_infer-gradio, works fine.

def load_vocoder(vocoder_name="vocos", is_local=True, local_path="/home/ai/Documents/AI Voice Clone/F5-TTS/ckpts/vocos", device=device, hf_cache_dir=None):
    if vocoder_name == "vocos":

4. then follow Gradio UI Training, try transcribe data first, upload audio files, wait a long time but all files transcribe failed, seems using whisper model to transcribe and my network unable to access directly, failed.
5. try use custom dataset with this guide, prepared metadata.csv and wav audio files, then run python scripts/prepare_csv_wavs.py , generated json, arrorw, vocab.txt in /home/ai/Documents/AI Voice Clone/F5-TTS/data/my_speech_pinyin folder, actually I can't found a place to modify dataset_name , then run python train.py, error is no model_cfg given. failed.
6. ok, then use prepared metadata.csv and wav audio files with gradio UI, just go to prepare data, and stuck too, Error: No audio files found in the specified path : /home/ai/Documents/AI Voice Clone/F5-TTS/src/f5_tts/../../data/my_speech_pinyin/wavs, I actually have wav audio files in path /home/ai/Documents/AI Voice Clone/F5-TTS/data/my_speech_pinyin/wavs and /home/ai/Documents/AI Voice Clone/F5-TTS/src/f5_tts/data/my_speech_pinyin/wavs, failed

I’m stuck here for 2 days, I checked issue, readme, multiple youtube videos, still no clue, please help.

✔️ Expected Behavior

I hope I can complete this training step and train my model.

❌ Actual Behavior

as mentioned above. orz

[SWivid]

Hi @Chloey-TS

• best not to use folder path with blanks: AI Voice Clone -> AI_Voice_Clone
• what is your wav file extension, e.g. .wav .mp3 .flac
• have you installed ffmpeg?

5. try use custom dataset with this guide, prepared metadata.csv and wav audio files, then run python scripts/prepare_csv_wavs.py , generated json, arrorw, vocab.txt in /home/ai/Documents/AI Voice Clone/F5-TTS/data/my_speech_pinyin folder, actually I can't found a place to modify dataset_name , then run python train.py, error is no model_cfg given. failed.

see https://github.com/SWivid/F5-TTS/tree/main/src/f5_tts/train
python train.py is never supported
you need to provide the config yaml file as mentioned
https://github.com/SWivid/F5-TTS/tree/main/src/f5_tts/train#1-training-script-used-for-pretrained-model

[Chloey-TS]

Hi @SWivid ,
Thanks for your quick reply.

1. I will change the folder name later.
2. it's wav files, I use audacity to slice and export it, seems a little large, eg. 1.3MB for 7s audio.
3. yes, ffmpeg installed, Version: 2:7.1.1-4

I actually confusion now, as the #57 here mentioned, after prepared dataset, then modify some args in train.py and then just run python train.py.

But as the gradio UI traning guide #143 and some youtube video, I can simply transcribe data or prepare data, after data prepared, I can then training data.

So the problem is I should Training script used for pretrained model? which means I should run accelerate *.py *.yaml before training?

again, thanks for your time and hard work.

[SWivid]

There are two different ways to make a custom model:

• training from scratch, e.g. our practice with Accelerate as framework:
  • https://github.com/SWivid/F5-TTS/tree/main?tab=readme-ov-file#1-with-hugging-face-accelerate
  • https://github.com/SWivid/F5-TTS/tree/main/src/f5_tts/train#1-training-script-used-for-pretrained-model
• finetune the released pretrained model, with Gradio UI interface:
  • https://github.com/SWivid/F5-TTS/tree/main?tab=readme-ov-file#2-with-gradio-app

[SWivid]

So I guess what you need is the second one, and the main problem is, as you mentioned

then follow #143, try transcribe data first, upload audio files, wait a long time but all files transcribe failed, seems using whisper model to transcribe and my network unable to access directly, failed.

The first thing you need to do is to get and make the whisper model runable, rather than bypass it.
You could:

• set a proxy to ensure the connection to huggingface
• or change the part

  F5-TTS/src/f5_tts/infer/utils_infer.py

  Lines 151 to 166 in 5f35f27

  	def initialize_asr_pipeline(device: str = device, dtype=None):
  	if dtype is None:
  	dtype = (
  	torch.float16
  	if "cuda" in device
  	and torch.cuda.get_device_properties(device).major >= 7
  	and not torch.cuda.get_device_name().endswith("[ZLUDA]")
  	else torch.float32
  	)
  	global asr_pipe
  	asr_pipe = pipeline(
  	"automatic-speech-recognition",
  	model="openai/whisper-large-v3-turbo",
  	torch_dtype=dtype,
  	device=device,
  	)

  , e.g. with some vibe coding tool to change the logic from pulling huggingface file to directly loading local files

[Chloey-TS]

Hi Swivid,
Yes, I just wanted to fine-tune the pre-trained model that has been released. At first I did suspect that the problem might be a network issue, but I had already installed whisper locally via pip, and in the subsequent steps, I tried to avoid using hugging face connections, so I left it alone.

I will check the hf_download.py file later and try to configure the proxy connection or loading local files, and I will update it here for reference later.

BTW, F5-TTS is great, the cloning effect is amazing with only one reference audio.
Thank you.

[SWivid]

if mirror sites available, can also try e.g. https://hf-mirror.com/

[Chloey-TS]

Hi @SWivid
After change the logic from pulling huggingface whisper files to directly loading local files, it's able to transcribe now.

But I encounter some new issues when finetune in gradio:
I use conda now, prepared 40 samples, I keep auto setting and start training, will give me out of memory error, I have an 12GB RTX5070 GPU.

If I lower the value (set them in yaml file), for example 10 epochs, batch size 5 per GPU, warmup updates/last updates per update/save per update are all set to 50, then run accelerate launch src/f5_tts/train/train.py --config-name F5TTS_v1_Base.yaml, it throws a "division by zero error", which is weird and seems to be related to line 333 of trainer.py, but I'm confused and don't know how to set it, I remember there is an explanation of these values somewhere but I can't find it now.

below is error message about out of memory , could you tell me where to find explanations about the training parameters?
Thank you.

ipex flag is deprecated, will be removed in Accelerate v1.10. From 2.7.0, PyTorch has all needed optimizations for Intel CPU and XPU.
The following values were not passed to `accelerate launch` and had defaults used instead:
	`--num_processes` was set to a value of `1`
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
Using logger: None
Loading dataset ...
Load vocos from local path /home/ai/Apps/vocos-mel-24khz
Sorting with sampler... if slow, check whether dataset is provided with duration: 100%|█████| 41/41 [00:00<00:00, 826761.85it/s]
Creating dynamic batches with 38400 audio frames per gpu: 100%|████████████████████████████| 41/41 [00:00<00:00, 1457342.92it/s]
Epoch 1/10:   0%|                                                                                     | 0/2 [00:00<?, ?update/s]/home/ai/Apps/AI_Voice/F5-TTS/src/f5_tts/model/dataset.py:144: UserWarning: torchaudio._backend.utils.load has been deprecated. This deprecation is part of a large refactoring effort to transition TorchAudio into a maintenance phase. The decoding and encoding capabilities of PyTorch for both audio and video are being consolidated into TorchCodec. Please see https://github.com/pytorch/audio/issues/3902 for more information. It will be removed from the 2.9 release. 
  audio, source_sample_rate = torchaudio.load(audio_path)
/home/ai/Apps/AI_Voice/F5-TTS/src/f5_tts/model/dataset.py:144: UserWarning: torchaudio._backend.utils.load has been deprecated. This deprecation is part of a large refactoring effort to transition TorchAudio into a maintenance phase. The decoding and encoding capabilities of PyTorch for both audio and video are being consolidated into TorchCodec. Please see https://github.com/pytorch/audio/issues/3902 for more information. It will be removed from the 2.9 release. 
  audio, source_sample_rate = torchaudio.load(audio_path)
Error executing job with overrides: []
Traceback (most recent call last):
  File "/home/ai/Apps/AI_Voice/F5-TTS/src/f5_tts/train/train.py", line 69, in main
    trainer.train(
  File "/home/ai/Apps/AI_Voice/F5-TTS/src/f5_tts/model/trainer.py", line 371, in train
    loss, cond, pred = self.model(
  File "/home/ai/miniconda3/envs/f5-tts/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/ai/miniconda3/envs/f5-tts/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/ai/Apps/AI_Voice/F5-TTS/src/f5_tts/model/cfm.py", line 294, in forward
    pred = self.transformer(
  File "/home/ai/miniconda3/envs/f5-tts/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/ai/miniconda3/envs/f5-tts/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/ai/Apps/AI_Voice/F5-TTS/src/f5_tts/model/backbones/dit.py", line 251, in forward
    x = block(x, t, mask=mask, rope=rope)
  File "/home/ai/miniconda3/envs/f5-tts/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/ai/miniconda3/envs/f5-tts/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/ai/Apps/AI_Voice/F5-TTS/src/f5_tts/model/modules.py", line 682, in forward
    norm, gate_msa, shift_mlp, scale_mlp, gate_mlp = self.attn_norm(x, emb=t)
  File "/home/ai/miniconda3/envs/f5-tts/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/ai/miniconda3/envs/f5-tts/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/ai/Apps/AI_Voice/F5-TTS/src/f5_tts/model/modules.py", line 317, in forward
    x = self.norm(x) * (1 + scale_msa[:, None]) + shift_msa[:, None]
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 98.00 MiB. GPU 0 has a total capacity of 11.49 GiB of which 93.19 MiB is free. Process 1491 has 45.21 MiB memory in use. Process 1532 has 60.65 MiB memory in use. Process 6484 has 29.65 MiB memory in use. Including non-PyTorch memory, this process has 10.61 GiB memory in use. Of the allocated memory 10.31 GiB is allocated by PyTorch, and 92.41 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
Epoch 1/10:   0%|                                                                                     | 0/2 [01:21<?, ?update/s]
Traceback (most recent call last):
  File "/home/ai/miniconda3/envs/f5-tts/bin/accelerate", line 8, in <module>
    sys.exit(main())
  File "/home/ai/miniconda3/envs/f5-tts/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 50, in main
    args.func(args)
  File "/home/ai/miniconda3/envs/f5-tts/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1199, in launch_command
    simple_launcher(args)
  File "/home/ai/miniconda3/envs/f5-tts/lib/python3.10/site-packages/accelerate/commands/launch.py", line 785, in simple_launcher
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/home/ai/miniconda3/envs/f5-tts/bin/python3.10', 'src/f5_tts/train/train.py', '--config-name', 'F5TTS_v1_Base.yaml']' returned non-zero exit status 1.

[Chloey-TS]

update, check other issue, I run accelerate config and start training in gradio again, seems works now, I will update the trainng result later.

accelerate config                                                 INT ✘  f5-tts  
--------------------------------------------------------------------------------------------------------------------------------In which compute environment are you running?
This machine                                                                                                                    
--------------------------------------------------------------------------------------------------------------------------------Which type of machine are you using?                                                                                            
No distributed training                                                                                                         
Do you want to run your training on CPU only (even if a GPU / Apple Silicon / Ascend NPU device is available)? [yes/NO]:NO      
Do you wish to optimize your script with torch dynamo?[yes/NO]:NO                                                               
Do you want to use DeepSpeed? [yes/NO]: NO                                                                                      
What GPU(s) (by id) should be used for training on this machine as a comma-separated list? [all]:all                            
Would you like to enable numa efficiency? (Currently only supported on NVIDIA hardware). [yes/NO]: yes                          
--------------------------------------------------------------------------------------------------------------------------------Do you wish to use mixed precision?                                                                                             
no                                                                                                                              
accelerate configuration saved at /home/ai/.cache/huggingface/accelerate/default_config.yaml 

[Chloey-TS]

Wow! It worked! I finally did it!
I will record it for other people like me.

1. Follow the guide to install this repo, dependencies and virtual environment (env or conda).
2. Download pretrained model checkpoints/vocos/whisper repo from hugging face, then you can set it to load from local, about load local ckpts and vocos method I mentioned above.
   For whisper, change F5-TTS/src/f5_tts/infer/utils_infer.py L151-L166 to

def initialize_asr_pipeline(device: str = device, dtype=None):
    if dtype is None:
        dtype = (
            torch.float16
            if "cuda" in device
            and torch.cuda.get_device_properties(device).major >= 7
            and not torch.cuda.get_device_name().endswith("[ZLUDA]")
            else torch.float32
        )
    global asr_pipe
    # asr_pipe = pipeline(
    #     "automatic-speech-recognition",
    #     model="openai/whisper-large-v3-turbo",
    #     torch_dtype=dtype,
    #     device=device,
    # )

    local_whisper_model_path = "/home/ai/Apps/whisper-large-v3-turbo" # <--- 把这里改成你下载的whisper模型的路径

    # Check if the local path exists before attempting to load
    if os.path.exists(local_whisper_model_path) and os.path.isdir(local_whisper_model_path):
        print(f"Loading Whisper model from local path: {local_whisper_model_path}")
        model_id_or_path = local_whisper_model_path
    else:
        print(f"Local Whisper model not found at {local_whisper_model_path}. Attempting to download from Hugging Face Hub...")
        model_id_or_path = "openai/whisper-large-v3-turbo"
    # --- CHANGE ENDS HERE ---

    asr_pipe = pipeline(
        "automatic-speech-recognition",
        model=model_id_or_path, # Use the determined model ID or local path
        torch_dtype=dtype,
        device=device,
    )

3. Then run accelerate config and choose the value that suits your situation. I actually only have one RTX5070 GPU.
4. Follow the gradio guide, upload audio, transcribe, and then prepare data, check some samples, check automatic settings, and start training!

Wala! After training, look how great the effect is!

Then I will collect some audios to start training.
Audio samples of 7 to 12 seconds seem good.
Whisper is good for transcription, but seems still need to improve it, such as clear expression of tone and pauses.
I will continue to learn how to make the audio closer to the target.
Thanks Swivid for the great work!