<div dir="ltr">MPI_Comm are opaque handles in C and integers in Fortran, which is required by MPI standard. The same applies to other types, like MPI_Op, MPI_Win etc. MPICH and OpenMPI have different implements for the handles. In MPICH handles are integer/bitfield, with some bits being offset to an array of objects. This makes it easy to do things like MPI_Comm_f2c(). In OpenMPI handles are pointers. OpenMPI has to transform pointers to integer offsets in MPI_Comm_c2f().<div><br></div><div>Running your tests with OpenMPI, you can see different pointers but same offsets</div><div>test_comms.c:</div><div><font face="monospace">0 4 -1419258464 909680992 909680992<br>1 4 1152255392 -2144517440 -2144517440<br>2 4 -306719328 768197312 768197312<br>3 4 -1766709856 715374384 715374384</font><br></div><div><br></div><div>test_comms.f90:<br></div><div> 0 0 3<br></div><div> 1 0 3<br> 2 0 3<br> 3 0 3<br><br></div><div>Running with MPICH, you can see C/Fortran MPI_Comm's are the same. But why ranks do not have the same integer/bitfield, I don't know. You need to dig into mpich code.</div><div>test_comms.c:<br></div><div>0 4 1140850688 -2080374780 -2080374780<br>1 4 1140850688 -2080374780 -2080374780<br>2 4 1140850688 -2080374782 -2080374782<br>3 4 1140850688 -2080374782 -2080374782<br></div><div><br></div>test_comms.f90:<div> 0 1140850688 -2080374780<br> 1 1140850688 -2080374780<br> 2 1140850688 -2080374782<br> 3 1140850688 -2080374782</div><div><br></div><div>In summary, users should not expect MPI_Comm variables are equal across ranks, and MPI_Send an MPI_Comm variable to remote ranks.</div><div><div><div dir="ltr" data-smartmail="gmail_signature"><div dir="ltr">--Junchao Zhang</div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Apr 22, 2020 at 8:56 AM Patrick Sanan <<a href="mailto:patrick.sanan@gmail.com" target="_blank">patrick.sanan@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Perhaps the confusion here is related to the fact that an MPI_Comm is not an integer identifying the communicator. Rather,<div>it's a pointer to a data structure which contains information about the communicator (I'm not positive but probably something like <a href="https://github.com/pmodels/mpich/blob/master/src/include/mpir_comm.h#L150" target="_blank">this</a>). </div><div><br></div><div>You're converting that pointer to an int and printing it out. The value happens to be the same on all ranks except 0, but this</div><div>doesn't directly tell you anything about equality of the MPI_comm objects that those pointers point to.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Am Mi., 22. Apr. 2020 um 15:28 Uhr schrieb Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>>:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Wed, Apr 22, 2020 at 3:07 AM Marius Buerkle <<a href="mailto:mbuerkle@web.de" target="_blank">mbuerkle@web.de</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div style="font-family:Verdana;font-size:12px"><div>I see, but I am still puzzeled, why are the communicators different on different notes eventhough it is the same object.</div></div></div></blockquote><div><br></div><div>This is the output of MPI_Comm_dup() on line 126 of tagm.c. Therefore, dup comms are not guaranteed to have the same id</div><div>across multiple processes.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div style="font-family:Verdana;font-size:12px"><div> </div><div><div>
<div name="quote" style="margin:10px 5px 5px 10px;padding:10px 0px 10px 10px;border-left:2px solid rgb(195,217,229)">
<div style="margin:0px 0px 10px"> </div>
<div name="quoted-content">PETSc creates a duplicate of the communicator during object creation.<br>
<a href="https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscCommDuplicate.html" target="_blank">https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscCommDuplicate.html</a><br>
<br>
Jose<br>
<br>
<br>
> El 22 abr 2020, a las 8:40, Marius Buerkle <<a href="mailto:mbuerkle@web.de" target="_blank">mbuerkle@web.de</a>> escribió:<br>
><br>
> Hi Dave,<br>
><br>
> I want to use it in Fortran if possible. But I tried both C and Fortran just to see if it works in general. I am using MPICH 3.3.2. I attached the MWE for C and Fortran with the output I get.<br>
><br>
> Marius<br>
><br>
><br>
><br>
><br>
><br>
> Hi,<br>
><br>
> What is PetscObjectGetComm expected to return?<br>
><br>
> As Patrick said, it returns the communicator associated with the petsc object.<br>
><br>
> I thought it would give the MPI communicator the object lives on. So if I create A matrix on PETSC_COMM_WORLD a call of PetscObjectGetComm for A it would return PETSC_COMM_WORLD? But it seems to return something else, and while most of the nodes return a similar communicator some are giving a different one.<br>
><br>
> How are you actually comparing the communicators (send code snippet)? Which MPI implementation are you using? And when are comparing comms is the comparison code written in C it FORTRAN?<br>
><br>
><br>
> That said, is there a way to get the MPI communicator a matrix lives on?<br>
><br>
> You are using the correct function. There is a macro as well but it’s best to use the function.<br>
><br>
> Thanks,<br>
> Dave<br>
><br>
><br>
><br>
><br>
> Best,<br>
> Marius<br>
> <test_comm.tar.gz><br>
</div>
</div>
</div>
</div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div>
</blockquote></div>