<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Tue, Sep 16, 2014 at 8:28 AM, Pierre Jolivet <span dir="ltr"><<a href="mailto:jolivet@ann.jussieu.fr" target="_blank">jolivet@ann.jussieu.fr</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 2014-09-15 11:40, Matthew Knepley wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On Sun, Sep 14, 2014 at 8:36 PM, Barry Smith <<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>><br>
wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Pierre,<br>
<br>
Thanks for reporting this, it is, indeed our bug. In<br>
petsclog.h we have macros for the various MPI calls in order to log<br>
their usage, for example,<br>
<br>
#define<br>
<br>
</blockquote>
MPI_Scatter(sendbuf,sendcount,<u></u>sendtype,recvbuf,recvcount,<u></u>recvtype,root,comm)<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
((petsc_scatter_ct++,0) ||<br>
PetscMPITypeSize(&petsc_recv_<u></u>len,recvcount,recvtype) ||<br>
<br>
</blockquote>
MPI_Scatter(sendbuf,sendcount,<u></u>sendtype,recvbuf,recvcount,<u></u>recvtype,root,comm))<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
but PetscMPITypeSize() simply called MPI_Type_size() which<br>
generated an MPI error for MPI_DATATYPE_NULL<br>
<br>
PETSC_STATIC_INLINE PetscErrorCode PetscMPITypeSize(<u></u>PetscLogDouble<br>
*buff,PetscMPIInt count,MPI_Datatype type)<br>
{<br>
PetscMPIInt mysize; return (MPI_Type_size(type,&mysize) ||<br>
((*buff += (PetscLogDouble) (count*mysize)),0));<br>
}<br>
</blockquote>
<br>
What error did you get? Why did I not get this error when I ran it? I<br>
ran with MPICH 3.0.4 since that was the one I had compiled for C++.<br>
</blockquote>
<br>
(sorry for the late answer)<br>
$ mpicxx in-place.cpp -I$PETSC_DIR/include -I$PETSC_DIR/$PETSC_ARCH/<u></u>include -L$PETSC_DIR/$PETSC_ARCH/lib -lpetsc -DPETSC_BUG<br>
$ mpirun -np 4 ./a.out<br>
Fatal error in PMPI_Type_size: Invalid datatype, error stack:<br>
PMPI_Type_size(117): MPI_Type_size(MPI_DATATYPE_<u></u>NULL) failed<br>
PMPI_Type_size(67).: Datatype for argument datatype is a null datatype<br>
$ mpicxx -show<br>
g++ -I/opt/mpich-3.1/build/include -L/opt/mpich-3.1/build/lib -lmpichcxx -Wl,-rpath -Wl,/opt/mpich-3.1/build/lib -lmpich -lopa -lmpl -lrt -lpthread<br></blockquote><div><br></div><div>Thanks for tracking this down. This seems to be a change from MPICH 3.0.4 to 3.1 Luckily, Barry found it too.</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Pierre<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Matt<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
In the branch barry/fix-usage-with-<u></u>mpidatatypenull I have added a<br>
check for this special case and avoid the MPI_Type_size() call. I<br>
will put this branch into next and if all tests pass it will be<br>
merged into maint and master and be in the next patch release.<br>
<br>
Thank you for reporting the problem.<br>
<br>
Barry<br>
<br>
Barry still thinks MPI 1.1 is the height of HPC computing :-(<br>
<br>
On Sep 14, 2014, at 4:16 PM, Pierre Jolivet<br>
<<a href="mailto:jolivet@ann.jussieu.fr" target="_blank">jolivet@ann.jussieu.fr</a>> wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hello,<br>
Could you please explain to me why the following example is not<br>
</blockquote>
working properly when <petsc.h> (from master, with OpenMPI 1.8.1) is<br>
included ?<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
$ mpicxx in-place.cpp -I$PETSC_DIR/include<br>
</blockquote>
-I$PETSC_DIR/$PETSC_ARCH/<u></u>include -L$PETSC_DIR/$PETSC_ARCH/lib<br>
-lpetsc<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
$ mpirun -np 2 ./a.out<br>
Done with the scatter !<br>
0 0 0 0 (this line should be filled with 0)<br>
1 1 1 1 (this line should be filled with 1)<br>
Done with the gather !<br>
<br>
$ mpicxx in-place.cpp -I$PETSC_DIR/include<br>
</blockquote>
-I$PETSC_DIR/$PETSC_ARCH/<u></u>include -L$PETSC_DIR/$PETSC_ARCH/lib<br>
-lpetsc -DPETSC_BUG<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
$ mpirun -np 2 ./a.out<br>
[:3367] *** An error occurred in MPI_Type_size<br>
[:3367] *** reported by process [4819779585,140733193388032]<br>
[:3367] *** on communicator MPI_COMM_WORLD<br>
[:3367] *** MPI_ERR_TYPE: invalid datatype<br>
[:3367] *** MPI_ERRORS_ARE_FATAL (processes in this communicator<br>
</blockquote>
will now abort,<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
[:3367] *** and potentially your MPI job)<br>
<br>
Thank you for looking,<br>
Pierre<br>
<br>
<in-place.cpp><br>
</blockquote></blockquote>
<br><span class="HOEnZb"><font color="#888888">
--<br>
What most experimenters take for granted before they begin their<br>
experiments is infinitely more interesting than any results to which<br>
their experiments lead.<br>
-- Norbert Wiener<br>
</font></span></blockquote>
</blockquote></div><br><br clear="all"><div><br></div>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener
</div></div>