<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">On 08/20/14 12:11, Karl Rupp wrote:<br>
</div>
<blockquote cite="mid:53F47446.7070300@iue.tuwien.ac.at" type="cite">Hi
Pierre,
<br>
<br>
> I have a cluster with nodes of 2 sockets of 4 cores+1 GPU.
<br>
<blockquote type="cite">
<br>
Is there a way to run a calculation with 4*N MPI tasks where
<br>
my matrix is first built outside PETSc, then to solve the
<br>
linear system using PETSc Mat, Vec, KSP on only N MPI
<br>
tasks to adress efficiently the N GPUs ?
<br>
</blockquote>
<br>
as far as I can tell, this should be possible with a suitable
subcommunicator. The tricky piece, however, is to select the right
MPI ranks for this. Note that you generally have no guarantee on
how the MPI ranks are distributed across the nodes, so be prepared
for something fairly specific to your MPI installation.
<br>
</blockquote>
Yes, I am ready to face this point too.<br>
<blockquote cite="mid:53F47446.7070300@iue.tuwien.ac.at" type="cite">
<br>
<br>
<blockquote type="cite">I am playing with the communicators
without success, but I
<br>
am surely confusing things...
<br>
</blockquote>
<br>
To keep matters simple, try to get this scenario working with a
purely CPU-based solve. Once this works, the switch to GPUs should
be just a matter of passing the right flags. Have a look at
PetscInitialize() here:
<br>
<a class="moz-txt-link-freetext" href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscInitialize.html">http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscInitialize.html</a>
<br>
which mentions that you need to create the subcommunicator of
MPI_COMM_WORLD first.
<br>
<br>
</blockquote>
I also started the work with a purely CPU-based solve only to test,
but without success. When<br>
I read this:<br>
<br>
"If you wish PETSc code to run ONLY on a subcommunicator of
MPI_COMM_WORLD, create that
communicator first and assign it to <a
href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PETSC_COMM_WORLD.html#PETSC_COMM_WORLD">PETSC_COMM_WORLD</a>
BEFORE calling <a
href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscInitialize.html#PetscInitialize">PetscInitialize</a>().
<br>
Thus if you are running a
four process job and two processes will run PETSc and have <a
href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscInitialize.html#PetscInitialize">PetscInitialize</a>()
and <a
href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscFinalize.html#PetscFinalize">PetscFinalize</a>()
and two process will not,
then do this. If ALL processes in <br>
the job are using <a
href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscInitialize.html#PetscInitialize">PetscInitialize</a>()
and <a
href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscFinalize.html#PetscFinalize">PetscFinalize</a>()
then you don't need to do this, even
if different subcommunicators of the job are doing different things
with PETSc."
<p>
</p>
I think I am not in this special scenario, because as my matrix is
initially partitionned on 4 <br>
processes, I need to call PetscInitialize() on each 4 processes in
order to build the PETSc matrix <br>
with MatSetValues. And my goal is after to solve the linear system
on only 2 processes... So<br>
building a sub-communicator will really do the trick ? Or i miss
something ?<br>
<br>
Thanks Karli for your answer,<br>
<br>
Pierre<br>
<blockquote cite="mid:53F47446.7070300@iue.tuwien.ac.at" type="cite">Best
regards,
<br>
Karli
<br>
<br>
</blockquote>
<br>
<br>
<div class="moz-signature">-- <br>
<b>Trio_U support team</b>
<br>
Marthe ROUX (01 69 08 00 02) Saclay
<br>
Pierre LEDAC (04 38 78 91 49) Grenoble
</div>
</body>
</html>