Feature request: train benchmark

[Bleach665]

Checks

• This template is only for feature request.
• I have thoroughly reviewed the project documentation but couldn't find any relevant information that meets my needs.
• I have searched for existing issues, including closed ones, and found not discussion yet.
• I am using English to submit this issue to facilitate community communication.

1. Is this request related to a challenge you're experiencing? Tell us your story.

Benchmarking on completely identical data is very useful for comparing results on different platforms, hardware and environments. Of course, the user can write such a script themselves, but a ready-made script that runs with a single command "out of the box" will be useful for everyone. And in many projects like this, such functionality has been implemented.

2. What is your suggested solution?

Add script like train-benchmark.py to... make benchmark.
Create github issue to share information about benchmark results.

3. Additional context or comments

No response

4. Can you help us with this feature?

• I am interested in contributing to this feature.

[SWivid]

BTW, have you really read our paper and documentation here in this repo, which part do you think are missing?

[Bleach665]

For you it is trivial. For the end user, IMHO, you need to run it with one command and output the results. As is done, for example, in OpenCV.

[SWivid]

For training, you cannot achieve that with one single command really.
Let's say where you have got stuck with current training instruction?

[Bleach665]

I'm training well, no problems. Thank you for the great maintenance.
As for the benchmark, it's just a suggestion. And I think it will be in demand.

For training, you cannot achieve that with one single command really.

Why? Choose a small dataset that we will consider as a standard, download it from the cloud when needed. It is not necessary that the result was 100% representative.

[SWivid]

Yep, we have provided scripts for dataset preparation here:
https://github.com/SWivid/F5-TTS/tree/main/src/f5_tts/train#prepare-dataset
And training scripts here:
https://github.com/SWivid/F5-TTS/tree/main/src/f5_tts/train#1-training-script-used-for-pretrained-model
And evaluation tools here:
https://github.com/SWivid/F5-TTS/tree/main/src/f5_tts/eval
We also have gradio UI interface for finetuning/training, along with instructions and tailored default settings.

[Bleach665]

Yes. But imagine:

• user rented servers on several platforms like Oracle, Amazon etc. with diff hardware. With this bench entering 2 commands (clone repo and benchmark) he will already know what to expect from this server.
• changed OS, environment, GPU hardware config - easy control performance
  etc.
  Here the question is not so much about the possibility of benchmarking - all the tools are already available in this project. But in a single standard (the same code and dataset) and usability.

[Bleach665]

another case:
User: Everything works very slowly for me, it seems like others are a bit faster
Maintainer: pls share output python benchmark.py and you hardware

[SWivid]

So you are talking about a simple script that to verify if the code works.
I think you can just go with:
pip install f5-tts (if training needed, clone and install editable)
f5-tts_infer-gradio (if no UI, f5-tts_infer-cli)

[SWivid]

User: Everything works very slowly for me, it seems like others are a bit faster
Maintainer: pls share output python benchmark.py and you hardware

Yes, so we mentioned at first: https://github.com/SWivid/F5-TTS?tab=readme-ov-file#install-pytorch-with-matched-device
And make detailed instruction about how to start an issue:

e.g. normally help-wanted issue, template like:

[Bleach665]

So you are talking about a simple script that to verify if the code works.

Not exactly. I'm talking about a script that downloads a certain dataset (which you choose as the standard for the benchmark), runs the benchmark in a short period of time and outputs the result.
Same code, same dataset for all users.

[Bleach665]

Yes, so we mentioned at first: https://github.com/SWivid/F5-TTS?tab=readme-ov-file#install-pytorch-with-matched-device

But now you don't know what result to expect from each piece of hardware. With the introduction of the standard benchmark, the output will show whether the problem is in the user's F5 installation/equipment or in their dataset/configuration.

[SWivid]

Sure, if you found it necessary.
We don't quite have that formed in mind as related projects do not have this kind of kit. Basically people just need inference and go with pip install f5-tts and f5-tts_infer-gradio; for developers wanna train from scratch, prerequisites are expected acquired.
Yes, you can just forward this initiative.