Questions about total training time and continuing training?

[roamindev]

I'm using the colab to train. Everything installed smoothly and it was very easy to get going (great job on that! :)...

I have 123 samples adding up to approximately 5-6 minutes of audio. With the default settings specified in the config.json that was generated (batch size: 6, epochs: 10000), the estimated training time was 100+ hours, so I adjusted the the config.json file to batch size of 20 and 2000 epochs, and now it's estimating ~10 hours.

~10 hours is a lot better than ~100, but I'm still curious if that much time is actually needed? Is my dataset size too large (or too small)? I didn't see any guidance on training time/total steps/etc anywhere.

Also, is it possible to stop training after a checkpoint is saved and test the model to see if it's good or needs more training? If so, can I just re-run the training step to have it continue where it left off? Along the same lines, if the colab happens to disconnect, can I continue training from the last saved checkpoint? Is that easy to do? (i.e. will the script detect the last checkpoint automatically?)

Thanks in advance for any help or guidance. Appears to be really great work here, so I'm excited to test the results.

Edit: I'm around 2000 steps in and just noticed the TensorBoard started working and I see now where it has a tab that let's me hear a test of the latest checkpoint. Too cool. :)

[34j]

will the script detect the last checkpoint automatically?

Yes.

There are always at least 5 Issues posted here per day and no time to train and play at all. This repository has the same functionality as so-vits-svc, so you can refer to that one for more information about training. I believe Bilibili has some useful articles (in Chinese).

[ADR3-N]

Could you link that repository please? Or if it is in fact the one entirely in Chinese, let us know? Some people including myself are likely very confused.