<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<div dir="ltr">
<div style="line-height:21px">I saw your update. In PetscCUDAInitialize we have
<div style="color:rgb(0,0,0);font-family:Menlo,Monaco,"Courier New",monospace;font-size:14px;white-space:pre">
</div>
<div style="color:rgb(0,0,0);font-family:Menlo,Monaco,"Courier New",monospace;font-size:14px;white-space:pre">
<div style="line-height:21px">
<div> <span style="color:rgb(0,128,0)">/* First get the device count */</span></div>
<div> err = <span style="color:rgb(121,94,38)">cudaGetDeviceCount</span>(&devCount);</div>
<div><br>
</div>
<div> <span style="color:rgb(0,128,0)">/* next determine the rank and then set the device via a mod */</span></div>
<div> ierr = <span style="color:rgb(121,94,38)">MPI_Comm_rank</span>(comm,&rank);<span style="color:rgb(121,94,38)">CHKERRQ</span>(ierr);</div>
<div> device = rank % devCount;</div>
<div> }</div>
<div> err = <span style="color:rgb(121,94,38)">cudaSetDevice</span>(device);</div>
</div>
</div>
<div style="color:rgb(0,0,0);font-family:Menlo,Monaco,"Courier New",monospace;font-size:14px;white-space:pre">
<br>
</div>
If we rely on the first CUDA call to do initialization, how could CUDA know these MPI stuff.
<div style="color:rgb(0,0,0);font-family:Menlo,Monaco,"Courier New",monospace;font-size:14px;white-space:pre">
<span style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(34,34,34)">--Junchao Zhang</span><br>
</div>
</div>
<br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, Sep 18, 2019 at 11:42 PM Smith, Barry F. <<a href="mailto:bsmith@mcs.anl.gov">bsmith@mcs.anl.gov</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Fixed the docs. Thanks for pointing out the lack of clarity<br>
<br>
<br>
> On Sep 18, 2019, at 11:25 PM, Zhang, Junchao via petsc-dev <<a href="mailto:petsc-dev@mcs.anl.gov" target="_blank">petsc-dev@mcs.anl.gov</a>> wrote:<br>
> <br>
> Barry,<br>
> <br>
> I saw you added these in init.c<br>
> <br>
> <br>
> + -cuda_initialize - do the initialization in PetscInitialize()<br>
> <br>
> <br>
> <br>
> <br>
> <br>
> <br>
> <br>
> <br>
> Notes:<br>
> <br>
> Initializing cuBLAS takes about 1/2 second there it is done by default in PetscInitialize() before logging begins<br>
> <br>
> <br>
> <br>
> But I did not get otherwise with -cuda_initialize 0, when will cuda be initialized?<br>
> --Junchao Zhang<br>
<br>
</blockquote>
</div>
</body>
</html>