[petsc-users] SNESComputeJacobianDefaultColor use too much memory

Thu Feb 18 19:58:14 CST 2016

On Thu, Feb 18, 2016 at 7:37 PM, Rongliang Chen <rongliang.chan at gmail.com>
wrote:

> Hi Barry,
>
> When I increase the size of the mesh, another memory issue comes out. The
> DMPlexInterpolateFaces_Internal takes too much memory. The massif output is
> attached. Any suggestions for this? Thanks.
>

It is creating parts of the mesh. I do not believe it allocates much temp
storage. Is that what you see?

  Matt

> Best,
> Rongliang
>
> On 02/18/2016 04:11 AM, Barry Smith wrote:
>
>>    Hmm, Matt, this is nuts. This
>>
>>   if (size > 1) {
>>      /* create a sequential iscoloring on all processors */
>>      ierr = MatGetSeqNonzeroStructure(mat,&mat_seq);CHKERRQ(ierr);
>>    }
>>
>>    It sequentializes the graph.
>>
>>     It looks like the only parallel coloring for Jacobians is
>> MATCOLORINGGREEDY? Try that.
>>
>>     Maybe that should be the default?
>>
>>     Barry
>>
>>
>> On Feb 17, 2016, at 1:49 AM, Rongliang Chen <rongliang.chan at gmail.com>
>>> wrote:
>>>
>>> Dear Barry,
>>>
>>> The massif output for the large case is attached. It shows that the job
>>> was kill in the function  MatGetSubMatrix_MPIAIJ_All(). Any suggestions?
>>>
>>> --------------------
>>> [8]PETSC ERROR: #1 MatGetSubMatrix_MPIAIJ_All() line 622 in
>>> /home/rlchen/soft/petsc-3.5.2/src/mat/impls/aij/mpi/mpiov.c
>>> [8]PETSC ERROR: #2 MatGetSeqNonzeroStructure_MPIAIJ() line 3010 in
>>> /home/rlchen/soft/petsc-3.5.2/src/mat/impls/aij/mpi/mpiaij.c
>>> [8]PETSC ERROR: #3 MatGetSeqNonzeroStructure() line 6487 in
>>> /home/rlchen/soft/petsc-3.5.2/src/mat/interface/matrix.c
>>> [8]PETSC ERROR: #4 MatColoringApply_SL() line 78 in
>>> /home/rlchen/soft/petsc-3.5.2/src/mat/color/impls/minpack/color.c
>>> [8]PETSC ERROR: #5 MatColoringApply() line 379 in
>>> /home/rlchen/soft/petsc-3.5.2/src/mat/color/interface/matcoloring.c
>>> [8]PETSC ERROR: #6 SNESComputeJacobianDefaultColor() line 71 in
>>> /home/rlchen/soft/petsc-3.5.2/src/snes/interface/snesj2.c
>>> [8]PETSC ERROR: #7 FormJacobian() line 58 in
>>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.5/SetupJacobian.c
>>> [8]PETSC ERROR: #8 SNESComputeJacobian() line 2193 in
>>> /home/rlchen/soft/petsc-3.5.2/src/snes/interface/snes.c
>>> [8]PETSC ERROR: #9 SNESSolve_NEWTONLS() line 230 in
>>> /home/rlchen/soft/petsc-3.5.2/src/snes/impls/ls/ls.c
>>> [8]PETSC ERROR: #10 SNESSolve() line 3743 in
>>> /home/rlchen/soft/petsc-3.5.2/src/snes/interface/snes.c
>>> [8]PETSC ERROR: #11 SolveTimeDependent() line 758 in
>>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.5/Nwtun.c
>>> [8]PETSC ERROR: #12 main() line 417 in
>>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.5/Nwtun.c
>>> -------------------
>>>
>>> Best,
>>> Rongliang
>>>
>>> On 02/17/2016 02:09 PM, Barry Smith wrote:
>>>
>>>>    Yes, this is the type of output I was expecting. Now you need to
>>>> produce it for a large case.
>>>>
>>>>    Barry
>>>>
>>>> On Feb 16, 2016, at 11:49 PM, Rongliang Chen <rongliang.chan at gmail.com>
>>>>> wrote:
>>>>>
>>>>> Hi Barry,
>>>>>
>>>>> I run the code with valgrind on a workstation for a smaller case and
>>>>> it produces some ASCII information (see attached).  Is this helpful?
>>>>>
>>>>> Best,
>>>>> Rongliang
>>>>>
>>>>> On 02/17/2016 01:30 PM, Barry Smith wrote:
>>>>>
>>>>>>    Hmm, something didn't work right with massif. I should give a
>>>>>> bunch of ASCII information about how much memory is used at different times
>>>>>> in the code. You may need to play around with the massif options and google
>>>>>> documentation on how to get it to provide useful information. Once it
>>>>>> produces the useful information it will be very helpful.
>>>>>>
>>>>>> time=0
>>>>>> mem_heap_B=0
>>>>>> mem_heap_extra_B=0
>>>>>> mem_stacks_B=0
>>>>>> heap_tree=empty
>>>>>>
>>>>>>
>>>>>> On Feb 16, 2016, at 9:44 PM, Rongliang Chen <rongliang.chan at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Dear Barry,
>>>>>>>
>>>>>>> Many thanks for your reply.
>>>>>>>
>>>>>>> I checked with the valgrind and did not obtain any outputs
>>>>>>> (massif.out.<pid>) because the job was killed before it reached the end.
>>>>>>>
>>>>>>> Then I switch to a smaller case, it works well and one of the output
>>>>>>> is attached (I did not find any useful information in it). The output with
>>>>>>> the option -mat_coloring_view is followed, which shows that the number of
>>>>>>> colors is 65. Any ideas for this?
>>>>>>>
>>>>>>> MatColoring Object: 480 MPI processes
>>>>>>>    type: sl
>>>>>>>    Weight type: RANDOM
>>>>>>>    Distance 2, Max. Colors 65535
>>>>>>>    Number of colors 65
>>>>>>>    Number of total columns 1637350
>>>>>>>
>>>>>>> Best regards,
>>>>>>> Rongliang
>>>>>>>
>>>>>>> On 02/17/2016 01:13 AM, Barry Smith wrote:
>>>>>>>
>>>>>>>>    How many colors are needed?
>>>>>>>>
>>>>>>>>    You need to produce a breakdown of where all the memory is being
>>>>>>>> used. For example valgrind with the
>>>>>>>> http://valgrind.org/docs/manual/ms-manual.html
>>>>>>>>
>>>>>>>>
>>>>>>>>    Barry
>>>>>>>>
>>>>>>>>
>>>>>>>> On Feb 16, 2016, at 6:45 AM, Rongliang Chen <
>>>>>>>>> rongliang.chan at gmail.com>
>>>>>>>>>   wrote:
>>>>>>>>>
>>>>>>>>> Dear all,
>>>>>>>>>
>>>>>>>>> I am using the DMPlex to solve a PDE on a unstructured mesh and I
>>>>>>>>> use the SNESComputeJacobianDefaultColor to compute the Jacobian matrix.
>>>>>>>>>
>>>>>>>>> My code works well for small problems (such as problem with
>>>>>>>>> 3.3x10^5 cells using 120 cores) but when I increase the number of cells
>>>>>>>>> (2.6x10^6 cells using 1920 cores), I go out of memory in the function
>>>>>>>>> MatColoringApply. It shows that one of the cores uses over 11G memory. I
>>>>>>>>> think this is unreasonable. Do you have any suggestions for debugging this
>>>>>>>>> problem?
>>>>>>>>>
>>>>>>>>> Best regards,
>>>>>>>>> Rongliang
>>>>>>>>>
>>>>>>>>> <massif.out.12562>
>>>>>>>
>>>>>> <massif.out.26358>
>>>>>
>>>> <massif.out.26539>
>>>
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20160218/3e13162d/attachment.html>