<div dir="ltr">On Mon, Nov 11, 2013 at 11:06 PM, Rongliang Chen <span dir="ltr"><<a href="mailto:rongliang.chan@gmail.com" target="_blank">rongliang.chan@gmail.com</a>></span> wrote:<br><div class="gmail_extra"><div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Jed,<br>
<br>
I tried the mpich version petsc on Janus (configured with option --download-mpich) and my code stopped at another place. The error message is followed. Do you have any suggestions?<br></blockquote><div><br></div><div>1) I believe you said that you ran under valgrind without errors, so we guess that GetPieceData() is fine.</div>
<div><br></div><div>2) I think it is quite unlikely that there is an error in PetscBinaryRead()</div><div><br></div><div>3) Wrong file size or permission should not cause an SEGV</div><div><br></div><div>4) Thus to me it clearly looks like a driver issue here with the filesystem. If this is reproducible, it should be easy</div>
<div> for the administrators of the machine to look at, and this is definitely there job. Move up the hierarchy now.</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
For the core dump, I emailed the administrators of the Janus for help about a week ago but have not get any reply yet.<br>
<br>
Best,<br>
Rongliang<br>
<br>
----------------------------<br>
[0]PETSC ERROR: ------------------------------<u></u>------------------------------<u></u>------------<br>
[0]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end<br>
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br>
[0]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC" target="_blank">http://www.mcs.anl.gov/petsc/<u></u>documentation/faq.html#<u></u>valgrind[0]PETSC</a> ERROR: or try <a href="http://valgrind.org" target="_blank">http://valgrind.org</a> on GNU/linux and Apple Mac OS X to find memory corruption errors<br>
[0]PETSC ERROR: likely location of problem given in stack below<br>
[0]PETSC ERROR: --------------------- Stack Frames ------------------------------<u></u>------<br>
[0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,<br>
[0]PETSC ERROR: INSTEAD the line number of the start of the function<br>
[0]PETSC ERROR: is given.<br>
[0]PETSC ERROR: [0] PetscBinaryRead line 234 /projects/ronglian/soft/petsc-<u></u>3.4.3/src/sys/fileio/sysio.c<br>
[0]PETSC ERROR: [0] GetPieceData line 1096 readbinary3d.c<br>
[0]PETSC ERROR: [0] DataReadAndSplitGeneric line 962 readbinary3d.c<br>
[0]PETSC ERROR: [0] DataRead line 621 readbinary3d.c<br>
[0]PETSC ERROR: [0] ReadBinary line 184 readbinary3d.c<br>
[0]PETSC ERROR: [0] LoadGrid line 720 loadgrid3d.c<br>
[0]PETSC ERROR: --------------------- Error Message ------------------------------<u></u>------<br>
[0]PETSC ERROR: Signal received!<br>
[0]PETSC ERROR: ------------------------------<u></u>------------------------------<u></u>------------<br>
[0]PETSC ERROR: Petsc Release Version 3.4.3, Oct, 15, 2013<br>
[0]PETSC ERROR: See docs/changes/index.html for recent updates.<br>
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.<br>
[0]PETSC ERROR: See docs/index.html for manual pages.<br>
[0]PETSC ERROR: ------------------------------<u></u>------------------------------<u></u>------------<br>
[0]PETSC ERROR: ./fsi3d on a Janus-debug-64bit-mpich named node0718 by ronglian Mon Nov 11 20:54:09 2013<br>
[0]PETSC ERROR: Libraries linked from /projects/ronglian/soft/petsc-<u></u>3.4.3/Janus-debug-64bit-mpich/<u></u>lib<br>
[0]PETSC ERROR: Configure run at Mon Nov 11 20:49:25 2013<br>
[0]PETSC ERROR: Configure options --known-level1-dcache-size=<u></u>32768 --known-level1-dcache-<u></u>linesize=64 --known-level1-dcache-assoc=8 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-c-double-complex=1 --download-mpich=1 --download-blacs=1 --download-f-blas-lapack=1 --download-metis=1 --download-parmetis=1 --download-scalapack=1 --download-superlu_dist=1 --known-mpi-shared-libraries=0 --with-64-bit-indices --with-batch=1 --download-exodusii=1 --download-hdf5=1 --download-netcdf=1 --known-64-bit-blas-indices --with-debugging=1<br>
[0]PETSC ERROR: ------------------------------<u></u>------------------------------<u></u>------------<br>
[0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file<br>
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0<br>
[unset]: aborting job:<br>
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0<br>
[0]PETSC ERROR: ------------------------------<u></u>------------------------------<u></u>------------<br>
[0]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end<br>
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br>
[0]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC" target="_blank">http://www.mcs.anl.gov/petsc/<u></u>documentation/faq.html#<u></u>valgrind[0]PETSC</a> ERROR: or try <a href="http://valgrind.org" target="_blank">http://valgrind.org</a> on GNU/linux and Apple Mac OS X to find memory corruption errors<br>
[0]PETSC ERROR: likely location of problem given in stack below<br>
<br>
<br>
On 11/07/2013 10:38 AM, Jed Brown wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Rongliang Chen <<a href="mailto:rongliang.chan@gmail.com" target="_blank">rongliang.chan@gmail.com</a>> writes:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Jed,<br>
<br>
I have not find a way to "dump core on selected ranks" yet and I will<br>
continue to do that.<br>
</blockquote>
Ask the administrators at your facility. There are a few common ways,<br>
but I'm not going to play a guessing game on the mailing list.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I run my code with the option "-on_error_attach_debugger" and got the<br>
following message:<br>
<br>
------------------------------<u></u>------------------------------<u></u>--------------<br>
An MPI process has executed an operation involving a call to the<br>
"fork()" system call to create a child process. Open MPI is currently<br>
operating in a condition that could result in memory corruption or<br>
other system errors; your MPI job may hang, crash, or produce silent<br>
data corruption. The use of fork() (or system() or other calls that<br>
create child processes) is strongly discouraged.<br>
<br>
The process that invoked fork was:<br>
<br>
Local host: node1529 (PID 3701)<br>
MPI_COMM_WORLD rank: 0<br>
<br>
If you are *absolutely sure* that your application will successfully<br>
and correctly survive a call to fork(), you may disable this warning<br>
by setting the mpi_warn_on_fork MCA parameter to 0.<br>
------------------------------<u></u>------------------------------<u></u>--------------<br>
[node1529:03700] 13 more processes have sent help message<br>
help-mpi-runtime.txt / mpi_init:warn-fork<br>
[node1529:03700] Set MCA parameter "orte_base_help_aggregate" to 0 to<br>
see all help / error messages<br>
------------------------------<u></u>------------------------------<u></u>--------------<br>
<br>
Is this message useful for the debugging?<br>
</blockquote>
This is just a possibly technical problem attaching a debugger in your<br>
environment, but you have to actually attach the debugger and poke<br>
around (stack trace, etc).<br>
<br>
Can you create an interactive session and run your job from there?<br>
</blockquote>
<br>
</blockquote></div><br><br clear="all"><div><br></div>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener
</div></div>