<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">Hui :</div><div class="gmail_quote">It is impossible to know what causes the problem from these error messages.</div><div class="gmail_quote">I notice that you use Petsc-<span style="color:rgb(0,0,0);font-family:Menlo;font-size:11px">3.5.2.</span></div><div class="gmail_quote"><font color="#000000" face="Menlo"><span style="font-size:11px">I would suggest</span></font></div><div class="gmail_quote"><font color="#000000" face="Menlo"><span style="font-size:11px">1. update to the latest petsc-3.6;</span></font></div><div class="gmail_quote"><font color="#000000" face="Menlo"><span style="font-size:11px">2. if problem remains, send us your matrices C and B in petsc binary format, we can investigate. With matrix files, one can test </span></font><span style="color:rgb(0,0,0);font-family:Menlo;font-size:11px">MatTransposeMatMult() using</span></div><div class="gmail_quote"><font color="#000000" face="Menlo"><span style="font-size:11px">petsc/src/mat/examples/tests/ex94.c</span></font><br></div><div class="gmail_quote"><font color="#000000" face="Menlo"><span style="font-size:11px"><br></span></font></div><div class="gmail_quote"><font color="#000000" face="Menlo"><span style="font-size:11px">Hong</span></font></div><div class="gmail_quote"><font color="#000000" face="Menlo"><span style="font-size:11px"><br></span></font></div><div class="gmail_quote"><font color="#000000" face="Menlo"><span style="font-size:11px"><br></span></font><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div>
<div style="direction:ltr;font-family:Tahoma;color:rgb(0,0,0);font-size:10pt">I'm running some code written by myself, using PETSC with MPI. It runs fine with less than or equal to 12 cores. However, if I ran it with 16 cores, it gives me an error. By looking
at the error message, it seems that the error happens at a line where I have something like this:
<div><br>
</div>
<div>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo">
ierr = MatTransposeMatMult(C,B,MAT_INITIAL_MATRIX,<span style="color:rgb(39,42,216)">1</span>,&A);CHKERRQ(ierr);</p>
<div><br>
</div>
<div>And I have attached the error message here. Does anyone sort of having an idea of what might be causing the problem here? Or is there any hint of what might be a direction for looking into this bug?
<div><br>
</div>
<div>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo">
[hus003@tscc-login2 test]$ cat /home/hus003/output/errFluid3d | grep '\[4\]'</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo">
[4]PETSC ERROR: ------------------------------------------------------------------------</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo">
[4]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo">
[4]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo">
[4]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[4]PETSC" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[4]PETSC</a> ERROR: or try <a href="http://valgrind.org" target="_blank">http://valgrind.org</a> on GNU/linux and Apple Mac OS X to find memory corruption errors</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo">
[4]PETSC ERROR: likely location of problem given in stack below</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo">
[4]PETSC ERROR: --------------------- Stack Frames ------------------------------------</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo">
[4]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo">
[4]PETSC ERROR: INSTEAD the line number of the start of the function</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo">
[4]PETSC ERROR: is given.</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo">
[4]PETSC ERROR: [4] PetscGatherMessageLengths2 line 180 /state/partition1/jjh/All-Rolls/math-roll/BUILD/sdsc-petsc_intel_openmpi_ib-3.5.2/petsc-3.5.2/src/sys/utils/mpimesg.c</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo">
[4]PETSC ERROR: [4] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ line 1573 /state/partition1/jjh/All-Rolls/math-roll/BUILD/sdsc-petsc_intel_openmpi_ib-3.5.2/petsc-3.5.2/src/mat/impls/aij/mpi/mpimatmatmult.c</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo">
[4]PETSC ERROR: [4] MatTransposeMatMult_MPIAIJ_MPIAIJ line 812 /state/partition1/jjh/All-Rolls/math-roll/BUILD/sdsc-petsc_intel_openmpi_ib-3.5.2/petsc-3.5.2/src/mat/impls/aij/mpi/mpimatmatmult.c</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo">
[4]PETSC ERROR: [4] MatTransposeMatMult line 8945 /state/partition1/jjh/All-Rolls/math-roll/BUILD/sdsc-petsc_intel_openmpi_ib-3.5.2/petsc-3.5.2/src/mat/interface/matrix.c</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo">
[4]PETSC ERROR: [4] Schur_Prec line 2885 /home/hus003/DISM/LSM_DISM3D/test/../Stokes3D.h</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo">
[4]PETSC ERROR: [4] StokesSetup line 140 /home/hus003/DISM/LSM_DISM3D/test/../Stokes3D.h</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo">
[4]PETSC ERROR: [4] test line 102 /home/hus003/DISM/LSM_DISM3D/test/test.cpp</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo">
[4]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo">
[4]PETSC ERROR: Signal received</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo">
[4]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo">
[4]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 </p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo">
[4]PETSC ERROR: ./test on a intel-openmpi_ib named <a href="http://tscc-gpu-6-5.sdsc.edu" target="_blank">tscc-gpu-6-5.sdsc.edu</a> by hus003 Mon Jul 13 23:57:09 2015</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo">
[4]PETSC ERROR: Configure options CFLAGS= FFLAGS= CXXFLAGS= --prefix=/opt/petsc/intel/openmpi_ib --with-mpi=1 --download-pastix=../pastix_5.2.2.12.tar.bz2 --download-ptscotch=../scotch_6.0.0_esmumps.tar.gz --with-blas-lib="-Wl,--start-group /opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/intel64/libmkl_intel_lp64.a
/opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/intel64/libmkl_sequential.a /opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/intel64/libmkl_core.a -Wl,--end-group -lpthread -lm" --with-lapack-lib="-Wl,--start-group /opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/intel64/libmkl_intel_lp64.a
/opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/intel64/libmkl_sequential.a /opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/intel64/libmkl_core.a -Wl,--end-group -lpthread -lm" --with-superlu_dist-include=/opt/superlu/intel/openmpi_ib/include
--with-superlu_dist-lib="-L/opt/superlu/intel/openmpi_ib/lib -lsuperlu" --with-parmetis-dir=/opt/parmetis/intel/openmpi_ib --with-metis-dir=/opt/parmetis/intel/openmpi_ib --with-mpi-dir=/opt/openmpi/intel/ib --with-scalapack-dir=/opt/scalapack/intel/openmpi_ib
--download-mumps=../MUMPS_4.10.0-p3.tar.gz --download-blacs=../blacs-dev.tar.gz --download-fblaslapack=../fblaslapack-3.4.2.tar.gz --with-pic=1 --with-shared-libraries=1</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo">
[4]PETSC ERROR: #1 User provided function() line 0 in unknown file</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo">
<br>
</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo">
<br>
</p>
</div>
</div>
</div>
</div>
</div>
</blockquote></div><br></div></div>