Facing issue during cross-sentence evaluation [SIM] of LibriSpeech-PC test-clean dataset.

[mukherjeesougata]

Checks

• This template is only for usage issues encountered.
• I have thoroughly reviewed the project documentation but couldn't find information to solve my problem.
• I have searched for existing issues, including closed ones, and couldn't find a solution.
• I am using English to submit this issue to facilitate community communication.

Environment Details

Ubuntu 24.04.2 LTS, Python 3.10.18

Steps to Reproduce

1. Run the command: conda create -n f5-tts python=3.10
2. Run the command: conda activate f5-tts
3. Run the command: pip install f5-tts
4. Modify line no. 16 to 19 in eval_librispeech_test_clean.py code. The modification is as follows:-

sys.path.append('/data0/Senior-Engineer-audio/TTS/F5/F5-TTS/src/f5_tts/model')
#from f5_tts.eval.utils_eval import get_librispeech_test, run_asr_wer, run_sim
from utils_eval import get_librispeech_test, run_asr_wer, run_sim

#rel_path = str(files("f5_tts").joinpath("../../"))
def find_f5_tts(start_path="."):
   for root, dirs, files in os.walk(start_path):
       if "f5_tts" in dirs:
           #print("f5_tts_path:", os.path.join(root, "f5_tts"))
           return os.path.abspath(os.path.join(root, "f5_tts"))
   return None

f5_tts_path = find_f5_tts()
print("f5_tts_path:", f5_tts_path if f5_tts_path else "Not found.")
rel_path = os.path.abspath(os.path.join(f5_tts_path, "..", ".."))

6. Inside f5-tts root run the following command:-
   python src/f5_tts/eval/eval_librispeech_test_clean.py --eval_task sim --gen_wav_dir /data0/Senior-Engineer-audio/TTS/F5_trial/F5-TTS/results/F5TTS_v1_Base_1250000/ls_pc_test_clean/seed0_euler_nfe16_vocos_ss-1_cfg2.0_speed1.0 --librispeech_test_clean_path /data0/Senior-Engineer-audio/TTS/F5_trial/F5-TTS/data/LibriSpeech/test-clean

✔️ Expected Behavior

No response

❌ Actual Behavior

I am encountering the following error:-

f5_tts_path: /data0/Senior-Engineer-audio/TTS/F5_trial/F5-TTS/src/f5_tts
100%|███████████████████████████████████| 1127/1127 [00:00<00:00, 153264.40it/s]
Using cache found in /home/ubuntu/.cache/torch/hub/s3prl_s3prl_main
Using cache found in /home/ubuntu/.cache/torch/hub/s3prl_s3prl_main
Using cache found in /home/ubuntu/.cache/torch/hub/s3prl_s3prl_main
Using cache found in /home/ubuntu/.cache/torch/hub/s3prl_s3prl_main
Using cache found in /home/ubuntu/.cache/torch/hub/s3prl_s3prl_main
Using cache found in /home/ubuntu/.cache/torch/hub/s3prl_s3prl_main
Using cache found in /home/ubuntu/.cache/torch/hub/s3prl_s3prl_main
Using cache found in /home/ubuntu/.cache/torch/hub/s3prl_s3prl_main
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/f5-tts/lib/python3.10/site-packages/f5_tts/eval/ecapa_tdnn.py", line 187, in __init__
    self.feature_extract = torch.hub.load(local_s3prl_path, feat_type, source="local", config_path=config_path)
  File "/home/ubuntu/anaconda3/envs/f5-tts/lib/python3.10/site-packages/torch/hub.py", line 647, in load
    model = _load_local(repo_or_dir, model, *args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/f5-tts/lib/python3.10/site-packages/torch/hub.py", line 677, in _load_local
    hub_module = _import_module(MODULE_HUBCONF, hubconf_path)
  File "/home/ubuntu/anaconda3/envs/f5-tts/lib/python3.10/site-packages/torch/hub.py", line 115, in _import_module
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 879, in exec_module
  File "<frozen importlib._bootstrap_external>", line 1016, in get_code
  File "<frozen importlib._bootstrap_external>", line 1073, in get_data
FileNotFoundError: [Errno 2] No such file or directory: '/home/ubuntu/.cache/torch/hub/s3prl_s3prl_main/hubconf.py'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/f5-tts/lib/python3.10/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/ubuntu/anaconda3/envs/f5-tts/lib/python3.10/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/data0/Senior-Engineer-audio/TTS/F5_trial/F5-TTS/src/f5_tts/eval/utils_eval.py", line 384, in run_sim
    model = ECAPA_TDNN_SMALL(feat_dim=1024, feat_type="wavlm_large", config_path=None)
  File "/home/ubuntu/anaconda3/envs/f5-tts/lib/python3.10/site-packages/f5_tts/eval/ecapa_tdnn.py", line 322, in ECAPA_TDNN_SMALL
    return ECAPA_TDNN(
  File "/home/ubuntu/anaconda3/envs/f5-tts/lib/python3.10/site-packages/f5_tts/eval/ecapa_tdnn.py", line 189, in __init__
    self.feature_extract = torch.hub.load("s3prl/s3prl", feat_type)
  File "/home/ubuntu/anaconda3/envs/f5-tts/lib/python3.10/site-packages/torch/hub.py", line 647, in load
    model = _load_local(repo_or_dir, model, *args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/f5-tts/lib/python3.10/site-packages/torch/hub.py", line 677, in _load_local
    hub_module = _import_module(MODULE_HUBCONF, hubconf_path)
  File "/home/ubuntu/anaconda3/envs/f5-tts/lib/python3.10/site-packages/torch/hub.py", line 115, in _import_module
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 879, in exec_module
  File "<frozen importlib._bootstrap_external>", line 1016, in get_code
  File "<frozen importlib._bootstrap_external>", line 1073, in get_data
FileNotFoundError: [Errno 2] No such file or directory: '/home/ubuntu/.cache/torch/hub/s3prl_s3prl_main/hubconf.py'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/data0/Senior-Engineer-audio/TTS/F5_trial/F5-TTS/src/f5_tts/eval/eval_librispeech_test_clean.py", line 100, in <module>
    main()
  File "/data0/Senior-Engineer-audio/TTS/F5_trial/F5-TTS/src/f5_tts/eval/eval_librispeech_test_clean.py", line 80, in main
    results = pool.map(run_sim, args)
  File "/home/ubuntu/anaconda3/envs/f5-tts/lib/python3.10/multiprocessing/pool.py", line 367, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/home/ubuntu/anaconda3/envs/f5-tts/lib/python3.10/multiprocessing/pool.py", line 774, in get
    raise self._value
FileNotFoundError: [Errno 2] No such file or directory: '/home/ubuntu/.cache/torch/hub/s3prl_s3prl_main/hubconf.py'

[SWivid]

F5-TTS/src/f5_tts/eval/ecapa_tdnn.py

Lines 185 to 189 in a275798

	try:
	local_s3prl_path = os.path.expanduser("~/.cache/torch/hub/s3prl_s3prl_main")
	self.feature_extract = torch.hub.load(local_s3prl_path, feat_type, source="local", config_path=config_path)
	except: # noqa: E722
	self.feature_extract = torch.hub.load("s3prl/s3prl", feat_type)

[SWivid]

Check the folder ~/.cache/torch/hub/s3prl_s3prl_main, probably you/someone has done manual modifications to that.
Commonly torch.hub.load will just pull https://github.com/s3prl/s3prl and has hubconf.py in it.

[mukherjeesougata]

Check the folder ~/.cache/torch/hub/s3prl_s3prl_main, probably you/someone has done manual modifications to that. Commonly torch.hub.load will just pull https://github.com/s3prl/s3prl and has hubconf.py in it.

I haven't done any manual modification to the .cache folder.
As it is related to a problem in the .cache folder so I have executed the following:-

1. Activated f5-tts conda environment
2. Run the command rm -rf ~/.cache/*
3. Run the command python src/f5_tts/eval/eval_librispeech_test_clean.py --eval_task sim --gen_wav_dir /data0/Senior-Engineer-audio/TTS/F5_trial/F5-TTS/results/F5TTS_v1_Base_1250000/ls_pc_test_clean/seed0_euler_nfe16_vocos_ss-1_cfg2.0_speed1.0 --librispeech_test_clean_path /data0/Senior-Engineer-audio/TTS/F5_trial/F5-TTS/data/LibriSpeech/test-clean

I am still encountering the error which is as follows:-

Traceback (most recent call last):
  File "/data0/Senior-Engineer-audio/TTS/F5_trial/F5-TTS/src/f5_tts/eval/ecapa_tdnn.py", line 187, in __init__
    self.feature_extract = torch.hub.load(local_s3prl_path, feat_type, source="local", config_path=config_path)
  File "/home/ubuntu/anaconda3/envs/f5-tts/lib/python3.10/site-packages/torch/hub.py", line 647, in load
    model = _load_local(repo_or_dir, model, *args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/f5-tts/lib/python3.10/site-packages/torch/hub.py", line 677, in _load_local
    hub_module = _import_module(MODULE_HUBCONF, hubconf_path)
  File "/home/ubuntu/anaconda3/envs/f5-tts/lib/python3.10/site-packages/torch/hub.py", line 115, in _import_module
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 879, in exec_module
  File "<frozen importlib._bootstrap_external>", line 1016, in get_code
  File "<frozen importlib._bootstrap_external>", line 1073, in get_data
FileNotFoundError: [Errno 2] No such file or directory: '/home/ubuntu/.cache/torch/hub/s3prl_s3prl_main/hubconf.py'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/f5-tts/lib/python3.10/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/ubuntu/anaconda3/envs/f5-tts/lib/python3.10/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/data0/Senior-Engineer-audio/TTS/F5_trial/F5-TTS/src/f5_tts/eval/utils_eval.py", line 387, in run_sim
    model = ECAPA_TDNN_SMALL(feat_dim=1024, feat_type="wavlm_large", config_path=None)
  File "/data0/Senior-Engineer-audio/TTS/F5_trial/F5-TTS/src/f5_tts/eval/ecapa_tdnn.py", line 323, in ECAPA_TDNN_SMALL
    return ECAPA_TDNN(
  File "/data0/Senior-Engineer-audio/TTS/F5_trial/F5-TTS/src/f5_tts/eval/ecapa_tdnn.py", line 189, in __init__
    self.feature_extract = torch.hub.load("s3prl/s3prl", feat_type)
  File "/home/ubuntu/anaconda3/envs/f5-tts/lib/python3.10/site-packages/torch/hub.py", line 638, in load
    repo_or_dir = _get_cache_or_reload(
  File "/home/ubuntu/anaconda3/envs/f5-tts/lib/python3.10/site-packages/torch/hub.py", line 290, in _get_cache_or_reload
    _remove_if_exists(extracted_repo)
  File "/home/ubuntu/anaconda3/envs/f5-tts/lib/python3.10/site-packages/torch/hub.py", line 124, in _remove_if_exists
    shutil.rmtree(path)
  File "/home/ubuntu/anaconda3/envs/f5-tts/lib/python3.10/shutil.py", line 731, in rmtree
    onerror(os.rmdir, path, sys.exc_info())
  File "/home/ubuntu/anaconda3/envs/f5-tts/lib/python3.10/shutil.py", line 729, in rmtree
    os.rmdir(path)
OSError: [Errno 39] Directory not empty: '/home/ubuntu/.cache/torch/hub/s3prl-s3prl-ec8064b/'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/data0/Senior-Engineer-audio/TTS/F5_trial/F5-TTS/src/f5_tts/eval/eval_librispeech_test_clean.py", line 100, in <module>
    main()
  File "/data0/Senior-Engineer-audio/TTS/F5_trial/F5-TTS/src/f5_tts/eval/eval_librispeech_test_clean.py", line 80, in main
    results = pool.map(run_sim, args)
  File "/home/ubuntu/anaconda3/envs/f5-tts/lib/python3.10/multiprocessing/pool.py", line 367, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/home/ubuntu/anaconda3/envs/f5-tts/lib/python3.10/multiprocessing/pool.py", line 774, in get
    raise self._value
OSError: [Errno 39] Directory not empty: '/home/ubuntu/.cache/torch/hub/s3prl-s3prl-ec8064b/'

The only modification which I have made has been mentioned in the previous comment otherwise I was encountering the following error while running the same command:-

Traceback (most recent call last):
  File "/data0/Senior-Engineer-audio/TTS/F5_trial/F5-TTS/src/f5_tts/eval/eval_librispeech_test_clean.py", line 100, in <module>
    main()
  File "/data0/Senior-Engineer-audio/TTS/F5_trial/F5-TTS/src/f5_tts/eval/eval_librispeech_test_clean.py", line 53, in main
    test_set = get_librispeech_test(metalst, gen_wav_dir, gpus, librispeech_test_clean_path)
  File "/data0/Senior-Engineer-audio/TTS/F5_trial/F5-TTS/src/f5_tts/eval/utils_eval.py", line 251, in get_librispeech_test
    f = open(metalst)
FileNotFoundError: [Errno 2] No such file or directory: '/home/ubuntu/anaconda3/envs/f5-tts/lib/python3.10/site-packages/f5_tts/../../data/librispeech_pc_test_clean_cross_sentence.lst'

In order to evade FileNotFoundError: [Errno 2] No such file or directory: '/home/ubuntu/.cache/torch/hub/s3prl_s3prl_main/hubconf.py' I have manually pasted the hubconf.py (downloaded from GitHub) inside /home/ubuntu/.cache/torch/hub/s3prl_s3prl_main/ directory. I am encountering errors which are as follows:-

1. ImportError: cannot import name 'hub' from 's3prl' (unknown location)
2. ModuleNotFoundError: No module named 's3prl.downstream' (This error has been encountered after manually pasting hub.py inside s3prl directory)

It seems that the s3prl GitHub repo is not getting pulled correctly.

@SWivid Do you have any comments on why is this happening?

[SWivid]

I dunno why

Maybe download manually the files of s3prl repo and place under /home/ubuntu/.cache/torch/hub/s3prl_s3prl_main see if work?

[mukherjeesougata]

I dunno why

Maybe download manually the files of s3prl repo and place under /home/ubuntu/.cache/torch/hub/s3prl_s3prl_main see if work?

Yes, I have already done that. It worked for me.