train on multiple gpus

[aymenbenammar]

I am trying to train on multiple gpus since the ram of a single gpu is not enough.
I tried using DataParallel from torch but it appears that it is not compatible with the model.

[SkalskiP]

hi 👋🏻 is this question specifically about maestro or general VLM fine-tuning?

[aymenbenammar]

Hi ! No it is specifically about maestro.
In fact, I have 4 gpus nvidia 4090 with 24 GB each and I couldn't use all of them in parallel for fine-tuning

[SkalskiP]

We don't support multi-GPU training yet. I was planning to roll it out over the next few months. But perhaps your use-case would be a good opportunity to accelerate this development. Would you like to help us?

[aymenbenammar]

Yes I would like to contribute.
Could you kindly suggest where in the codebase I should focus to implement it ?

[SkalskiP]

I think we need to start researching here:

• https://huggingface.co/docs/transformers/perf_train_gpu_many - we use transformers models
• https://lightning.ai/docs/pytorch/stable/accelerators/gpu_intermediate.html - we use Lightning trainer

We can start by enabling it for one model. For example here: https://github.com/roboflow/maestro/blob/develop/maestro/trainer/models/qwen_2_5_vl/core.py