<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Hi Matt,<br>
<br>
Thanks for your reply. The job was killed in the function
DMPlexInterpolateFaces_Internal and the administrator told me that
the reason is out of memory. The error messages are followed:<br>
<br>
---------------------------<br>
[0]PETSC ERROR:
------------------------------------------------------------------------<br>
[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation
Violation, probably memory access out of range<br>
[0]PETSC ERROR: Try option -start_in_debugger or
-on_error_attach_debugger<br>
[0]PETSC ERROR: or see
<a class="moz-txt-link-freetext" href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind">http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind</a>[0]PETSC
ERROR: or try <a class="moz-txt-link-freetext" href="http://valgrind.org">http://valgrind.org</a> on GNU/linux and Apple Mac OS X
to find memory corruption errors<br>
[0]PETSC ERROR: likely location of problem given in stack below<br>
[0]PETSC ERROR: --------------------- Stack Frames
------------------------------------<br>
[0]PETSC ERROR: Note: The EXACT line numbers in the stack are not
available,<br>
[0]PETSC ERROR: INSTEAD the line number of the start of the
function<br>
[0]PETSC ERROR: is given.<br>
[0]PETSC ERROR: [0] PetscHashIJKLPut line 1261
/home/rlchen/soft/petsc-3.5.2/include/../src/sys/utils/hash.h<br>
[0]PETSC ERROR: [0] DMPlexInterpolateFaces_Internal line 154
/home/rlchen/soft/petsc-3.5.2/src/dm/impls/plex/plexinterpolate.c<br>
[0]PETSC ERROR: [0] DMPlexInterpolate line 333
/home/rlchen/soft/petsc-3.5.2/src/dm/impls/plex/plexinterpolate.c<br>
[0]PETSC ERROR: [0] DMPlexCreateExodusNwtun line 98
/home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.5/src/plexexodusii.c<br>
[0]PETSC ERROR: [0] DMPlexCreateExodusFromFileNwtun line 44
/home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.5/src/plexexodusii.c<br>
[0]PETSC ERROR: [0] CreateMesh line 19
/home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.5/SetupMeshes.c<br>
[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------<br>
[0]PETSC ERROR: Signal received<br>
[0]PETSC ERROR: See
<a class="moz-txt-link-freetext" href="http://www.mcs.anl.gov/petsc/documentation/faq.html">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble
shooting.<br>
[0]PETSC ERROR: Petsc Release Version 3.5.2, unknown <br>
[0]PETSC ERROR: ./Nwtun on a 64bit-debug named
leszek-ThinkStation-C20 by rlchen Thu Feb 18 23:07:32 2016<br>
[0]PETSC ERROR: Configure options --download-fblaslapack
--download-blacs --download-scalapack --download-metis
--download-parmetis --download-exodusii --download-netcdf
--download-hdf5 --with-64-bit-indices --with-c2html=0 --with-mpi=1
--with-debugging=1 --with-shared-libraries=0<br>
[0]PETSC ERROR: #1 User provided function() line 0 in unknown
file<br>
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0<br>
<br>
=====================================================================================<br>
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES<br>
= EXIT CODE: 15104<br>
= CLEANING UP REMAINING PROCESSES<br>
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES<br>
=====================================================================================<br>
----------------------------------<br>
<br>
Best,<br>
Rongliang<br>
<br>
On 02/19/2016 09:58 AM, Matthew Knepley wrote:<br>
</div>
<blockquote
cite="mid:CAMYG4GkW0DwE0RgVduEq=R6Z95z8WC7MU6=kp+ph70_XbtzsuQ@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Thu, Feb 18, 2016 at 7:37 PM,
Rongliang Chen <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:rongliang.chan@gmail.com" target="_blank">rongliang.chan@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">Hi
Barry,<br>
<br>
When I increase the size of the mesh, another memory issue
comes out. The DMPlexInterpolateFaces_Internal takes too
much memory. The massif output is attached. Any
suggestions for this? Thanks.<br>
</blockquote>
<div><br>
</div>
<div>It is creating parts of the mesh. I do not believe it
allocates much temp storage. Is that what you see?</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
Best,<br>
Rongliang<br>
<br>
On 02/18/2016 04:11 AM, Barry Smith wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
Hmm, Matt, this is nuts. This<br>
<br>
if (size > 1) {<br>
/* create a sequential iscoloring on all processors
*/<br>
ierr =
MatGetSeqNonzeroStructure(mat,&mat_seq);CHKERRQ(ierr);<br>
}<br>
<br>
It sequentializes the graph.<br>
<br>
It looks like the only parallel coloring for
Jacobians is MATCOLORINGGREEDY? Try that.<br>
<br>
Maybe that should be the default?<br>
<br>
Barry<br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
On Feb 17, 2016, at 1:49 AM, Rongliang Chen <<a
moz-do-not-send="true"
href="mailto:rongliang.chan@gmail.com"
target="_blank">rongliang.chan@gmail.com</a>>
wrote:<br>
<br>
Dear Barry,<br>
<br>
The massif output for the large case is attached. It
shows that the job was kill in the function
MatGetSubMatrix_MPIAIJ_All(). Any suggestions?<br>
<br>
--------------------<br>
[8]PETSC ERROR: #1 MatGetSubMatrix_MPIAIJ_All() line
622 in
/home/rlchen/soft/petsc-3.5.2/src/mat/impls/aij/mpi/mpiov.c<br>
[8]PETSC ERROR: #2 MatGetSeqNonzeroStructure_MPIAIJ()
line 3010 in
/home/rlchen/soft/petsc-3.5.2/src/mat/impls/aij/mpi/mpiaij.c<br>
[8]PETSC ERROR: #3 MatGetSeqNonzeroStructure() line
6487 in
/home/rlchen/soft/petsc-3.5.2/src/mat/interface/matrix.c<br>
[8]PETSC ERROR: #4 MatColoringApply_SL() line 78 in
/home/rlchen/soft/petsc-3.5.2/src/mat/color/impls/minpack/color.c<br>
[8]PETSC ERROR: #5 MatColoringApply() line 379 in
/home/rlchen/soft/petsc-3.5.2/src/mat/color/interface/matcoloring.c<br>
[8]PETSC ERROR: #6 SNESComputeJacobianDefaultColor()
line 71 in
/home/rlchen/soft/petsc-3.5.2/src/snes/interface/snesj2.c<br>
[8]PETSC ERROR: #7 FormJacobian() line 58 in
/home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.5/SetupJacobian.c<br>
[8]PETSC ERROR: #8 SNESComputeJacobian() line 2193 in
/home/rlchen/soft/petsc-3.5.2/src/snes/interface/snes.c<br>
[8]PETSC ERROR: #9 SNESSolve_NEWTONLS() line 230 in
/home/rlchen/soft/petsc-3.5.2/src/snes/impls/ls/ls.c<br>
[8]PETSC ERROR: #10 SNESSolve() line 3743 in
/home/rlchen/soft/petsc-3.5.2/src/snes/interface/snes.c<br>
[8]PETSC ERROR: #11 SolveTimeDependent() line 758 in
/home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.5/Nwtun.c<br>
[8]PETSC ERROR: #12 main() line 417 in
/home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.5/Nwtun.c<br>
-------------------<br>
<br>
Best,<br>
Rongliang<br>
<br>
On 02/17/2016 02:09 PM, Barry Smith wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
Yes, this is the type of output I was expecting.
Now you need to produce it for a large case.<br>
<br>
Barry<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
On Feb 16, 2016, at 11:49 PM, Rongliang Chen <<a
moz-do-not-send="true"
href="mailto:rongliang.chan@gmail.com"
target="_blank">rongliang.chan@gmail.com</a>>
wrote:<br>
<br>
Hi Barry,<br>
<br>
I run the code with valgrind on a workstation for
a smaller case and it produces some ASCII
information (see attached). Is this helpful?<br>
<br>
Best,<br>
Rongliang<br>
<br>
On 02/17/2016 01:30 PM, Barry Smith wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0
0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
Hmm, something didn't work right with massif.
I should give a bunch of ASCII information about
how much memory is used at different times in
the code. You may need to play around with the
massif options and google documentation on how
to get it to provide useful information. Once it
produces the useful information it will be very
helpful.<br>
<br>
time=0<br>
mem_heap_B=0<br>
mem_heap_extra_B=0<br>
mem_stacks_B=0<br>
heap_tree=empty<br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
On Feb 16, 2016, at 9:44 PM, Rongliang Chen
<<a moz-do-not-send="true"
href="mailto:rongliang.chan@gmail.com"
target="_blank">rongliang.chan@gmail.com</a>>
wrote:<br>
<br>
Dear Barry,<br>
<br>
Many thanks for your reply.<br>
<br>
I checked with the valgrind and did not obtain
any outputs (massif.out.<pid>) because
the job was killed before it reached the end.<br>
<br>
Then I switch to a smaller case, it works well
and one of the output is attached (I did not
find any useful information in it). The output
with the option -mat_coloring_view is
followed, which shows that the number of
colors is 65. Any ideas for this?<br>
<br>
MatColoring Object: 480 MPI processes<br>
type: sl<br>
Weight type: RANDOM<br>
Distance 2, Max. Colors 65535<br>
Number of colors 65<br>
Number of total columns 1637350<br>
<br>
Best regards,<br>
Rongliang<br>
<br>
On 02/17/2016 01:13 AM, Barry Smith wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">
How many colors are needed?<br>
<br>
You need to produce a breakdown of where
all the memory is being used. For example
valgrind with the<br>
<a moz-do-not-send="true"
href="http://valgrind.org/docs/manual/ms-manual.html"
rel="noreferrer" target="_blank">http://valgrind.org/docs/manual/ms-manual.html</a><br>
<br>
<br>
Barry<br>
<br>
<br>
<blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">
On Feb 16, 2016, at 6:45 AM, Rongliang
Chen <<a moz-do-not-send="true"
href="mailto:rongliang.chan@gmail.com"
target="_blank">rongliang.chan@gmail.com</a>><br>
wrote:<br>
<br>
Dear all,<br>
<br>
I am using the DMPlex to solve a PDE on a
unstructured mesh and I use the
SNESComputeJacobianDefaultColor to compute
the Jacobian matrix.<br>
<br>
My code works well for small problems
(such as problem with 3.3x10^5 cells using
120 cores) but when I increase the number
of cells (2.6x10^6 cells using 1920
cores), I go out of memory in the function
MatColoringApply. It shows that one of the
cores uses over 11G memory. I think this
is unreasonable. Do you have any
suggestions for debugging this problem?<br>
<br>
Best regards,<br>
Rongliang<br>
<br>
</blockquote>
</blockquote>
<massif.out.12562><br>
</blockquote>
</blockquote>
<massif.out.26358><br>
</blockquote>
</blockquote>
<massif.out.26539><br>
</blockquote>
</blockquote>
<br>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div class="gmail_signature">What most experimenters take for
granted before they begin their experiments is infinitely
more interesting than any results to which their experiments
lead.<br>
-- Norbert Wiener</div>
</div>
</div>
</blockquote>
<br>
</body>
</html>