[mpich-discuss] MPI jobs don't have access to CUDA devices

[Jayesh Krishna]

>> started the process in debug mode ...
 I was about to suggest that :)

Regards,
Jayesh
----- Original Message -----
From: "Florin Ciubotaru" <florinciu at gmail.com>
To: "Jayesh Krishna" <jayesh at mcs.anl.gov>
Cc: mpich-discuss at mcs.anl.gov
Sent: Wednesday, February 23, 2011 9:25:28 AM
Subject: Re: [mpich-discuss] MPI jobs don't have access to CUDA devices

Hi Jayesh, 


Thanks for the quick response. 
Indeed, running mpiexec with -localroot does allow gpu access, but as I understand, it only limits the execution to one node. 
I tried to change the "slave" smpd service settings using the Windows service manager (services.msc), but apparently all connections to those nodes fail after allowing the service to be interactive. 


After a bit of digging I found a partial solution that probably has some performance impact. For this I disabled smpd as a windows service on the cuda enabled machines and started the process in debug mode using "smpd -d 0". 
The linux versions seem to have more solid approaches but for my case debug mode should do for now. 


Thanks, 
Florin Ciubotaru 


On Tue, Feb 22, 2011 at 7:02 PM, Jayesh Krishna < jayesh at mcs.anl.gov > wrote: 


Hi, 
You are right, the launched processes do not have an interactive console session. This prevents the MPI processes from using GPUs. This is a feature that can be added in future (Although I don't have a timeline for it). 
You can however use GPUs on a local node using the "-localroot" option of mpiexec. 

Regards, 
Jayesh 




----- Original Message ----- 
From: "Florin Ciubotaru" < florinciu at gmail.com > 
To: mpich-discuss at mcs.anl.gov 
Sent: Tuesday, February 22, 2011 10:53:45 AM 
Subject: [mpich-discuss] MPI jobs don't have access to CUDA devices 


Hi, 


I'm running mpi jobs on a small cluster that has several machines with CUDA enabled devices. I'm trying to add gpu acceleration to my computations, but I'm having issues executing CUDA code while running the processes as an mpi job. The gpu logic works fine when the application is executed as an independent process. 
I guess there's something in the way smpd launches the job and prevents the process from properly accessing the CUDA driver(and device). Most probable cause is that smpd is running as a Windows service and not as an interactive console session. Is there a way to tweak smpd or mpiexec in order to have CUDA access for the jobs? 


Thanks, 
Florin Ciubotaru 




_______________________________________________ 
mpich-discuss mailing list 
mpich-discuss at mcs.anl.gov 
https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss