[QUESTION]May I allocate and register NCCL userbuffer for activations/gradients transmitted between PP stages? #1720
Description
I noted that when use-nccl-ub is enabled, param and grad buffer can be allocated and registered with NCCL user buffer mechanism. That could save SM resources and improve performance. I would like to apply NCCL userbuffer for activations/gradients transmitted between PP stages.
From the current NCCL backend implementation in pytorch code (ProcessGroupNCCL.cpp), it looks P2P NCCL register buffer is not supported.
Please let me know if we have plan to support it.
It would be highly appreciated if any suggestion? Many thanks.