<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Thu, Feb 18, 2016 at 7:37 PM, Rongliang Chen <span dir="ltr"><<a href="mailto:rongliang.chan@gmail.com" target="_blank">rongliang.chan@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Barry,<br>
<br>
When I increase the size of the mesh, another memory issue comes out. The DMPlexInterpolateFaces_Internal takes too much memory. The massif output is attached. Any suggestions for this? Thanks.<br></blockquote><div><br></div><div>It is creating parts of the mesh. I do not believe it allocates much temp storage. Is that what you see?</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Best,<br>
Rongliang<br>
<br>
On 02/18/2016 04:11 AM, Barry Smith wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hmm, Matt, this is nuts. This<br>
<br>
if (size > 1) {<br>
/* create a sequential iscoloring on all processors */<br>
ierr = MatGetSeqNonzeroStructure(mat,&mat_seq);CHKERRQ(ierr);<br>
}<br>
<br>
It sequentializes the graph.<br>
<br>
It looks like the only parallel coloring for Jacobians is MATCOLORINGGREEDY? Try that.<br>
<br>
Maybe that should be the default?<br>
<br>
Barry<br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On Feb 17, 2016, at 1:49 AM, Rongliang Chen <<a href="mailto:rongliang.chan@gmail.com" target="_blank">rongliang.chan@gmail.com</a>> wrote:<br>
<br>
Dear Barry,<br>
<br>
The massif output for the large case is attached. It shows that the job was kill in the function MatGetSubMatrix_MPIAIJ_All(). Any suggestions?<br>
<br>
--------------------<br>
[8]PETSC ERROR: #1 MatGetSubMatrix_MPIAIJ_All() line 622 in /home/rlchen/soft/petsc-3.5.2/src/mat/impls/aij/mpi/mpiov.c<br>
[8]PETSC ERROR: #2 MatGetSeqNonzeroStructure_MPIAIJ() line 3010 in /home/rlchen/soft/petsc-3.5.2/src/mat/impls/aij/mpi/mpiaij.c<br>
[8]PETSC ERROR: #3 MatGetSeqNonzeroStructure() line 6487 in /home/rlchen/soft/petsc-3.5.2/src/mat/interface/matrix.c<br>
[8]PETSC ERROR: #4 MatColoringApply_SL() line 78 in /home/rlchen/soft/petsc-3.5.2/src/mat/color/impls/minpack/color.c<br>
[8]PETSC ERROR: #5 MatColoringApply() line 379 in /home/rlchen/soft/petsc-3.5.2/src/mat/color/interface/matcoloring.c<br>
[8]PETSC ERROR: #6 SNESComputeJacobianDefaultColor() line 71 in /home/rlchen/soft/petsc-3.5.2/src/snes/interface/snesj2.c<br>
[8]PETSC ERROR: #7 FormJacobian() line 58 in /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.5/SetupJacobian.c<br>
[8]PETSC ERROR: #8 SNESComputeJacobian() line 2193 in /home/rlchen/soft/petsc-3.5.2/src/snes/interface/snes.c<br>
[8]PETSC ERROR: #9 SNESSolve_NEWTONLS() line 230 in /home/rlchen/soft/petsc-3.5.2/src/snes/impls/ls/ls.c<br>
[8]PETSC ERROR: #10 SNESSolve() line 3743 in /home/rlchen/soft/petsc-3.5.2/src/snes/interface/snes.c<br>
[8]PETSC ERROR: #11 SolveTimeDependent() line 758 in /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.5/Nwtun.c<br>
[8]PETSC ERROR: #12 main() line 417 in /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.5/Nwtun.c<br>
-------------------<br>
<br>
Best,<br>
Rongliang<br>
<br>
On 02/17/2016 02:09 PM, Barry Smith wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Yes, this is the type of output I was expecting. Now you need to produce it for a large case.<br>
<br>
Barry<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On Feb 16, 2016, at 11:49 PM, Rongliang Chen <<a href="mailto:rongliang.chan@gmail.com" target="_blank">rongliang.chan@gmail.com</a>> wrote:<br>
<br>
Hi Barry,<br>
<br>
I run the code with valgrind on a workstation for a smaller case and it produces some ASCII information (see attached). Is this helpful?<br>
<br>
Best,<br>
Rongliang<br>
<br>
On 02/17/2016 01:30 PM, Barry Smith wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hmm, something didn't work right with massif. I should give a bunch of ASCII information about how much memory is used at different times in the code. You may need to play around with the massif options and google documentation on how to get it to provide useful information. Once it produces the useful information it will be very helpful.<br>
<br>
time=0<br>
mem_heap_B=0<br>
mem_heap_extra_B=0<br>
mem_stacks_B=0<br>
heap_tree=empty<br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On Feb 16, 2016, at 9:44 PM, Rongliang Chen <<a href="mailto:rongliang.chan@gmail.com" target="_blank">rongliang.chan@gmail.com</a>> wrote:<br>
<br>
Dear Barry,<br>
<br>
Many thanks for your reply.<br>
<br>
I checked with the valgrind and did not obtain any outputs (massif.out.<pid>) because the job was killed before it reached the end.<br>
<br>
Then I switch to a smaller case, it works well and one of the output is attached (I did not find any useful information in it). The output with the option -mat_coloring_view is followed, which shows that the number of colors is 65. Any ideas for this?<br>
<br>
MatColoring Object: 480 MPI processes<br>
type: sl<br>
Weight type: RANDOM<br>
Distance 2, Max. Colors 65535<br>
Number of colors 65<br>
Number of total columns 1637350<br>
<br>
Best regards,<br>
Rongliang<br>
<br>
On 02/17/2016 01:13 AM, Barry Smith wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
How many colors are needed?<br>
<br>
You need to produce a breakdown of where all the memory is being used. For example valgrind with the<br>
<a href="http://valgrind.org/docs/manual/ms-manual.html" rel="noreferrer" target="_blank">http://valgrind.org/docs/manual/ms-manual.html</a><br>
<br>
<br>
Barry<br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On Feb 16, 2016, at 6:45 AM, Rongliang Chen <<a href="mailto:rongliang.chan@gmail.com" target="_blank">rongliang.chan@gmail.com</a>><br>
wrote:<br>
<br>
Dear all,<br>
<br>
I am using the DMPlex to solve a PDE on a unstructured mesh and I use the SNESComputeJacobianDefaultColor to compute the Jacobian matrix.<br>
<br>
My code works well for small problems (such as problem with 3.3x10^5 cells using 120 cores) but when I increase the number of cells (2.6x10^6 cells using 1920 cores), I go out of memory in the function MatColoringApply. It shows that one of the cores uses over 11G memory. I think this is unreasonable. Do you have any suggestions for debugging this problem?<br>
<br>
Best regards,<br>
Rongliang<br>
<br>
</blockquote></blockquote>
<massif.out.12562><br>
</blockquote></blockquote>
<massif.out.26358><br>
</blockquote></blockquote>
<massif.out.26539><br>
</blockquote></blockquote>
<br>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div>
</div></div>