[Question]: Single process with multiple threads pure RDMA data transfer on multiple nodes.

[irexyc]

Question

My program controls multiple GPUs on a single node using a single process with multiple threads (like 1 process and 8 threads to control 8 gpus on a node). How can I implement pure RDMA transfer between GPUs on single node or multiple nodes using nvshmem? (like gpu0-on-node0 -> gpu1-on-node0 or gpu2-on-node0 -> gpu3-on-node1)

I've read the documentation, and it seems that the nvshmem host API doesn't support concurrency. I can't find a way to create 2 x 8 PEs to mapping 2 x 8 GPUs (two nodes and each nodes has eight gpus) using single process with multiple threads.

[benhg]

NVSHMEM doesn't support this mode of execution. Only one thread per process can be involved with an NVSHMEM API call at a time. PEs are always bound to a single OS process. Can you execute your application with one process per GPU instead of one thread per GPU?

[irexyc]

@benhg

Thanks for your response. May I ask f there are any plans to support this feature? I'm curious why this isn't supported as other libraries like NCCL supports multiple thread usage. For NVSHMEM, can this be achieved by having each thread maintain its own states like nvshmemi_state, nvshmemi_boot_handle etc.?

[benhg]

This doesn't mesh very naturally with the nvshmem programming model where these states are implicit and stored inside the runtime. In models that do support this (like NCCL device APIs, or MPI), the programmer will explicitly pass objects into functions (in NCCL's example, you pass a devComm into the CUDA kernel where you plan to use the APIs). Without the programmer explicitly passing the resource struct into the program, each thread doesn't know which instance of the state to use.

If you have a specific use case in mind, maybe we can suggest some method of using NVSHMEM as-is, or if you have a specific feature request, we can file an issue internally to track the RFE.

I'd also like to ask what is the main motivation for this request? For example, we've had users ask about this kind of thing because the performance (memory) overhead of spawning many independent processes can get expensive.

[benhg]

Hi @irexyc , I will close this for now. Feel free to reopen if you have follow-up questions.