<p>Open MPI one-sided operations with datatypes still have known bugs. They have had bug report s with reduced test cases for several years now. They need to fix those bugs. Please let them know that you are also waiting...</p>
<p>To work around that, and for other reasons, I will write a new SF implementation using point-to-point.</p>
<div class="gmail_quote">On Sep 11, 2012 12:44 PM, "fdkong" <<a href="mailto:fd.kong@foxmail.com">fd.kong@foxmail.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>Hi Matt,</div><div><br></div><div>Thanks. I guess there are two reasons:</div><div><br></div><div>(1) The MPI function MPI_Accumulate with operation MPI_RELACE is not supported in the implementation of OpenMPI 1.4.3. or other OpenMPI versions.</div>
<div><br></div><div>(2) The MPI function dose not accept the datatype MPIU_2INT, when we use 64-bit integers. But when we run on MPICH, it works well!</div><div><br></div><div><div style="color:#909090;font-family:Arial Narrow;font-size:12px">
------------------</div><div style="font-size:14px;font-family:Verdana"><div>Fande Kong</div><div>ShenZhen Institutes of Advanced Technology</div><div>Chinese Academy of Sciences</div>
</div></div><div> </div><div><u></u><div><br></div><div><br></div><div style="font-size:12px;font-family:Arial Narrow;padding:2px 0 2px 0">------------------ Original ------------------</div><div style="font-size:12px;background:#efefef;padding:8px">
<div><b>From: </b> "knepley"<<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>>;</div><div><b>Date: </b> Tue, Sep 11, 2012 01:33 PM</div><div><b>To: </b> "fdkong"<<a href="mailto:fd.kong@foxmail.com" target="_blank">fd.kong@foxmail.com</a>>; <u></u></div>
<div><b>Cc: </b> "petsc-users"<<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>; <u></u></div><div><b>Subject: </b> Re: PetscSFReduceBegin does not work correctly on openmpi-1.4.3with64 integers</div>
</div><div><br></div>On Tue, Sep 11, 2012 at 12:05 AM, fdkong <span dir="ltr"><<a href="mailto:fd.kong@foxmail.com" target="_blank">fd.kong@foxmail.com</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>Hi Matt,</div><div><br></div><div>I tested src/sys/sf/examples/tutorials/ex1 on OpenMPI and MPICH seperately respectively. I found the error come from the function PetscSFReduceBegin called by PetscSFCreateInverseSF. I used the script below:</div>
</blockquote><div><br></div><div>Thanks for testing this. I will run it myself and track down the bug.</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>mpirun -n 2 ./ex1 -test_invert</div><div><br></div><div>(1) On OpenMPI, got the result below:</div><div><br></div><div><div>Star Forest Object: 2 MPI processes</div><div> type not yet set</div><div> synchronization=FENCE sort=rank-order</div>
<div> [0] Number of roots=3, leaves=2, remote ranks=1</div><div> [0] 0 <- (1,1)</div><div> [0] 1 <- (1,0)</div><div> [1] Number of roots=2, leaves=3, remote ranks=1</div><div> [1] 0 <- (0,1)</div><div> [1] 1 <- (0,0)</div>
<div> [1] 2 <- (0,2)</div><div>## Multi-SF</div><div>Star Forest Object: 2 MPI processes</div><div> type not yet set</div><div> synchronization=FENCE sort=rank-order</div><div> [0] Number of roots=3, leaves=2, remote ranks=1</div>
<div> [0] 0 <- (1,1)</div><div> [0] 1 <- (1,0)</div><div> [1] Number of roots=2, leaves=3, remote ranks=1</div><div> [1] 0 <- (0,2)</div><div> [1] 1 <- (0,0)</div><div> [1] 2 <- (0,2)</div><div>## Inverse of Multi-SF</div>
<div>Star Forest Object: 2 MPI processes</div><div> type not yet set</div><div> synchronization=FENCE sort=rank-order</div><div> [0] Number of roots=2, leaves=0, remote ranks=0</div><div> [1] Number of roots=3, leaves=0, remote ranks=0</div>
</div><div><br></div><div>(2) On MPICH, got the result below:</div><div><br></div><div><div>Star Forest Object: 2 MPI processes</div><div> type not yet set</div><div> synchronization=FENCE sort=rank-order</div><div> [0] Number of roots=3, leaves=2, remote ranks=1</div>
<div> [0] 0 <- (1,1)</div><div> [0] 1 <- (1,0)</div><div> [1] Number of roots=2, leaves=3, remote ranks=1</div><div> [1] 0 <- (0,1)</div><div> [1] 1 <- (0,0)</div><div> [1] 2 <- (0,2)</div><div>## Multi-SF</div>
<div>Star Forest Object: 2 MPI processes</div><div> type not yet set</div><div> synchronization=FENCE sort=rank-order</div><div> [0] Number of roots=3, leaves=2, remote ranks=1</div><div> [0] 0 <- (1,1)</div><div>
[0] 1 <- (1,0)</div>
<div> [1] Number of roots=2, leaves=3, remote ranks=1</div><div> [1] 0 <- (0,1)</div><div> [1] 1 <- (0,0)</div><div> [1] 2 <- (0,2)</div><div>## Inverse of Multi-SF</div><div>Star Forest Object: 2 MPI processes</div>
<div> type not yet set</div><div> synchronization=FENCE sort=rank-order</div><div> [0] Number of roots=2, leaves=3, remote ranks=1</div><div> [0] 0 <- (1,1)</div><div> [0] 1 <- (1,0)</div><div> [0] 2 <- (1,2)</div>
<div> [1] Number of roots=3, leaves=2, remote ranks=1</div><div> [1] 0 <- (0,1)</div><div> [1] 1 <- (0,0)</div></div><div><br></div><div>From two above results, you could found that the inverse of Multi-SF is incorrect on OpenMPI. Could you please take some debugs on OpenMPI (1.4.3) with 64-bit integers?</div>
<div><br></div><div>In my code, I call DMComplexDistribute that calls PetscSFCreateInverseSF that calls PetscSFReduceBegin. I had taken a lot of debugs, and found the error come from the PetscSFReduceBegin.</div><div><br>
</div><div>On Mon, Sep 10, 2012 at 10:47 PM, fdkong <span dir="ltr"><<a href="mailto:fd.kong@foxmail.com" target="_blank">fd.kong@foxmail.com</a>></span> wrote:</div><div><u></u><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div><br></div><div>>> Hi all,</div><div>>></div><div>>> The function PetscSFReduceBegin runs well on MPICH, but does not work</div><div>>> on openmpi-1.4.3, with 64 integers. Anyone knows why?</div>
<div>>></div><div><br></div><div>>1) What error are you seeing? There are no errors in the build.</div><div><br></div><div>Yes, There are no errors in the build and configure. But when I ran my code involved the function PetscSFReduceBegin on supercomputer, I got the error below:</div>
</blockquote><div><br></div><div>Can you run src/sys/sf/examples/tutorials/ex1? There are several tests in the makefile there. I suspect</div><div>that your graph is not correctly specified.</div><div><br></div><div> Matt</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div>[0]PETSC ERROR: ------------------------------------------------------------------------</div><div>
[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range</div><div>[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger</div><div>[0]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind%5B0%5DPETSC" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC</a> ERROR: or try <a href="http://valgrind.org" target="_blank">http://valgrind.org</a> on GNU/linux and Apple Mac OS X to find memory corruption errors</div>
<div>[0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run </div><div>[0]PETSC ERROR: to get more information on the crash.</div><div>[0]PETSC ERROR: --------------------- Error Message ------------------------------------</div>
<div>[0]PETSC ERROR: Signal received!</div><div>[0]PETSC ERROR: ------------------------------------------------------------------------</div><div>[0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 3, Wed Aug 29 11:26:24 CDT 2012 </div>
<div>[0]PETSC ERROR: See docs/changes/index.html for recent updates.</div><div>[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.</div><div>[0]PETSC ERROR: See docs/index.html for manual pages.</div><div>
[0]PETSC ERROR: ------------------------------------------------------------------------</div><div>[0]PETSC ERROR: ./linearElasticity on a arch-linu named node0353 by fako9399 Mon Sep 10 16:50:42 2012</div><div>[0]PETSC ERROR: Libraries linked from /projects/fako9399/petsc-3.3-p3/arch-linux264-cxx-opt/lib</div>
<div>[0]PETSC ERROR: Configure run at Mon Sep 10 13:58:46 2012</div><div>[0]PETSC ERROR: Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=8 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --with-clanguage=cxx --with-shared-libraries=1 --with-dynamic-loading=1 --download-f-blas-lapack=1 --with-batch=1 --known-mpi-shared-libraries=0 --with-mpi-shared=1 --download-parmetis=1 --download-metis=1 --with-64-bit-indices=1 --with-netcdf-dir=/projects/fako9399/petsc-3.3-p3/externalpackage/netcdf-4.1.3install --download-exodusii=1 --with-debugging=no --download-ptscotch=1</div>
<div>[0]PETSC ERROR: ------------------------------------------------------------------------</div><div>[0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file</div><div>--------------------------------------------------------------------------</div>
<div>MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD </div><div>with errorcode 59.</div><div><br></div><div>NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.</div><div>You may or may not see output from other processes, depending on</div>
<div>exactly when Open MPI kills them.</div><div>--------------------------------------------------------------------------</div><div>--------------------------------------------------------------------------</div><div>mpirun has exited due to process rank 0 with PID 1517 on</div>
<div>node node0353 exiting without calling "finalize". This may</div><div>have caused other processes in the application to be</div><div>terminated by signals sent by mpirun (as reported here).</div><div>--------------------------------------------------------------------------</div>
</div><div><br></div><div>I had done some debugs, and then found the error came from the function PetscSFReduceBegin.</div><div><br></div><div>>2) Please do not send logs to petsc-users, send them to</div><div>>petsc-maint at <a href="http://mcs.anl.gov" target="_blank">mcs.anl.gov</a></div>
<div><br></div><div>Ok, Thanks.</div><div><br></div><div>> Matt</div><div><br></div><div><br></div><div>>> Maybe this link could help us guess why?</div><div>>> <a href="http://www.open-mpi.org/community/lists/devel/2005/11/0517.php" target="_blank">http://www.open-mpi.org/community/lists/devel/2005/11/0517.php</a></div>
<div>>></div><div>>> I attached the configure.log and make.log files.</div><div>>> ------------------</div><div>>> Fande Kong</div><div>>> ShenZhen Institutes of Advanced Technology</div><div>
>> Chinese Academy of Sciences</div><div>>></div><div>>></div></blockquote></div><span><font color="#888888"><br><br clear="all"><div><br></div>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<br><u></u></font></span></div></blockquote></div><br><br clear="all"><div><br></div>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<br><u></u></div></blockquote></div>