<html><head><meta http-equiv="Content-Type" content="text/html charset=iso-8859-1"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><br><div><div>On Sep 15, 2014, at 1:01 AM, Matthew Knepley <<a href="mailto:knepley@gmail.com">knepley@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Sun, Sep 14, 2014 at 5:56 PM, Christophe Prud'homme<span class="Apple-converted-space"> </span><span dir="ltr"><<a href="mailto:christophe.prudhomme@feelpp.org" target="_blank">christophe.prudhomme@feelpp.org</a>></span><span class="Apple-converted-space"> </span>wrote:<br><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; position: static; z-index: auto;"><div dir="ltr">Hello<div><br></div><div>Pierre and I found this problem occurring using openmpi 1.8.1 petsc 3.5.1 (this was done on my side using homebrew on osx)</div><div>we haven't checked mpich. However it is disturbing that removing the petsc.h header actually solve the problem.</div><div><br></div><div>from the mpi standard, if we understand correctly, we shouldn't have to specify the data type for MPI_IN_PLACE scatter/gather operations. To avoid either a crash with, deadlock(scatter) or wrong results (gather) with boost::mpi we need to specify the data type.</div><div><br></div><div>In our code Feel++ we have added some tests to verify this behavior [1]</div><div>basically</div><div> - mpi alone OK</div><div> - boost::mpi alone OK (boost::mpi is just used as an alternative to initialize mpi)</div><div> - mpi+petsc.h with MPI_DATATYPE_NULL in scatter crash</div><div> - mpi+petsc.h+proper datatype OK</div><div><div> - boost::mpi.h+petsc with MPI_DATATYPE_NULL hangs in scatter</div><div> - boost::mpi.h+petsc+proper datatype OK</div></div><div><br></div><div> 1.<span class="Apple-converted-space"> </span><a href="https://github.com/feelpp/feelpp/blob/develop/testsuite/feelcore/test_gatherscatter.cpp" target="_blank">https://github.com/feelpp/feelpp/blob/develop/testsuite/feelcore/test_gatherscatter.cpp</a></div></div></blockquote><div><br></div><div>1) I think this is an OpenMPI bug</div><div><br></div><div>2) I think this because I cannot reproduce with MPICH, and it is not valgrind clean</div><div><br></div><div>3) OpenMPI has lots of bugs. If you guys can reproduce with MPICH, I will track it</div><div>down and fix it.</div></div></div></div></div></blockquote><div><br></div><div>True, but they fixed the similar wrong behavior (which was inherited from libNBC) when I pinged them (<a href="http://www.open-mpi.org/community/lists/users/2013/11/23034.php">http://www.open-mpi.org/community/lists/users/2013/11/23034.php</a>).</div><div>BTW, I had the same problem with master and MPICH 3.1.</div><div><br></div><div>Anyways, thanks Barry for the quick fix.</div><div><br></div><div>Pierre</div><br><blockquote type="cite"><div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div dir="ltr"><div><br></div><div>Best regards</div><div>C.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Sep 15, 2014 at 12:37 AM, Matthew Knepley<span class="Apple-converted-space"> </span><span dir="ltr"><<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>></span><span class="Apple-converted-space"> </span>wrote:<br><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>On Sun, Sep 14, 2014 at 4:16 PM, Pierre Jolivet<span class="Apple-converted-space"> </span><span dir="ltr"><<a href="mailto:jolivet@ann.jussieu.fr" target="_blank">jolivet@ann.jussieu.fr</a>></span><span class="Apple-converted-space"> </span>wrote:<br><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;">Hello,<br>Could you please explain to me why the following example is not working properly when <petsc.h> (from master, with OpenMPI 1.8.1) is included ?<br><br>$ mpicxx in-place.cpp -I$PETSC_DIR/include -I$PETSC_DIR/$PETSC_ARCH/include -L$PETSC_DIR/$PETSC_ARCH/lib -lpetsc<br>$ mpirun -np 2 ./a.out<br>Done with the scatter !<br>0 0 0 0 (this line should be filled with 0)<br>1 1 1 1 (this line should be filled with 1)<br>Done with the gather !<br><br>$ mpicxx in-place.cpp -I$PETSC_DIR/include -I$PETSC_DIR/$PETSC_ARCH/include -L$PETSC_DIR/$PETSC_ARCH/lib -lpetsc -DPETSC_BUG<br>$ mpirun -np 2 ./a.out<br>[:3367] *** An error occurred in MPI_Type_size<br>[:3367] *** reported by process [4819779585,140733193388032]<br>[:3367] *** on communicator MPI_COMM_WORLD<br>[:3367] *** MPI_ERR_TYPE: invalid datatype<br>[:3367] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,<br>[:3367] *** and potentially your MPI job)<br></blockquote><div><br></div></div><div>I just built this with MPICH and it runs fine:</div><div><br></div><div><div>master:/PETSc3/petsc/petsc-pylith$ /PETSc3/petsc/petsc-pylith/arch-pylith-cxx-debug/bin/mpiexec -host localhost -n 2 /PETSc3/petsc/petsc-pylith/arch-pylith-cxx-debug/lib/in-place-obj/in-place </div><span><div>Done with the scatter !</div><div>0 0 0 0 (this line should be filled with 0)</div><div>1 1 1 1 (this line should be filled with 1)</div><div>Done with the gather !</div></span></div><div><br></div><div>Will valgrind.</div><div><br></div><div> Matt</div><span><div> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;">Thank you for looking,<br>Pierre<br><br></blockquote></span></div><span><font color="#888888"><br><br clear="all"><span class="HOEnZb"><font color="#888888"><div><br></div>--<span class="Apple-converted-space"> </span><br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</font></span></font></span></div></div><span class="HOEnZb"><font color="#888888"></font></span></blockquote></div><span class="HOEnZb"><font color="#888888"><br><br clear="all"><div><br></div>--<span class="Apple-converted-space"> </span><br>Christophe Prud'homme<div>Feel++ Project Manager</div><div>Professor in Applied Mathematics </div><div>@ Université Joseph Fourier (Grenoble, France)</div><div>@ Université de Strasbourg (France)</div></font></span></div></blockquote></div><br><br clear="all"><div><br></div>--<span class="Apple-converted-space"> </span><br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div></div></div></blockquote></div><br></body></html>