[petsc-users] Mat preallocation for adaptive grid

[Matthew Knepley]

On Sat, Jun 11, 2022 at 8:43 PM Samuel Estes <samuelestes91 at gmail.com>
wrote:

> I'm sorry, would you mind clarifying? I think my email was so long and
> rambling that it's tough for me to understand which part was being
> answered.
>
> On Sat, Jun 11, 2022 at 7:38 PM Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Sat, Jun 11, 2022 at 8:32 PM Samuel Estes <samuelestes91 at gmail.com>
>> wrote:
>>
>>> Hello,
>>>
>>> My question concerns preallocation for Mats in adaptive FEM problems.
>>> When the grid refines, I destroy the old matrix and create a new one of the
>>> appropriate (larger size). When the grid “un-refines” I just use the same
>>> (extra large) matrix and pad the extra unused diagonal entries with 1’s.
>>> The problem comes in with the preallocation. I use the MatPreallocator,
>>> MatPreallocatorPreallocate() paradigm which requires a specific sparsity
>>> pattern. When the grid un-refines, although the total number of nonzeros
>>> allocated is (most likely) more than sufficient, the particular sparsity
>>> pattern changes which leads to mallocs in the MatSetValues routines and
>>> obviously I would like to avoid this.
>>>
>>> One obvious solution is just to destroy and recreate the matrix any time
>>> the grid changes, even if it gets smaller. By just using a new matrix every
>>> time, I would avoid this problem although at the cost of having to rebuild
>>> the matrix more often than necessary. This is the simplest solution from a
>>> programming perspective and probably the one I will go with.
>>>
>>> I'm just curious if there's an alternative that you would recommend?
>>> Basically what I would like to do is to just change the sparsity pattern
>>> that is created in the MatPreallocatorPreallocate() routine. I'm not sure
>>> how it works under the hood, but in principle, it should be possible to
>>> keep the memory allocated for the Mat values and just assign them new
>>> column numbers and potentially add new nonzeros as well. Is there a
>>> convenient way of doing this? One thought I had was to just fill in the
>>> MatPreallocator object with the new sparsity pattern of the coarser mesh
>>> and then call the MatPreallocatorPreallocate() routine again with the new
>>> MatPreallocator matrix. I'm just not sure how exactly that would work since
>>> it would have already been called for the FEM matrix for the previous,
>>> finer grid.
>>>
>>> Finally, does this really matter? I imagine the bottleneck (assuming
>>> good preallocation) is in the solver so maybe it doesn't make much
>>> difference whether or not I reuse the old matrix. In that case, going with
>>> option 1 and simply destroying and recreating the matrix would be the way
>>> to go just to save myself some time.
>>>
>>> I hope that my question is clear. If not, please let me know and I will
>>> clarify. I am very curious if there's a convenient solution for the second
>>> option I mentioned to recycle the allocated memory and redo the sparsity
>>> pattern.
>>>
>>
>> I have not run any tests of this kind of thing, so I cannot say
>> definitively.
>>
>> I can say that I consider the reuse of memory a problem to be solved at
>> allocation time. You would hope that a good malloc system would give
>> you back the same memory you just freed when getting rid of the prior
>> matrix, so you would get the speedup you want using your approach.
>>
>
> What do you mean by "your approach"? Do you mean the first option where I
> just always destroy the matrix? Are you basically saying that when I
> destroy the old matrix and create a new one, it should just give me the
> same block of memory that was just freed by the destruction of the previous
> one?
>

Yes.


>
>> Second, I think the allocation cost is likely to pale in comparison to
>> the cost of writing the matrix itself (passing all those indices and values
>> through
>> the memory bus), and so reuse of the memory is not that important (I
>> think).
>>
>
> This seems to suggest that the best option is just to destroy and recreate
> and not worry about "re-preallocating". Do I understand that correctly?
>

Yes.

  Thanks,

     Matt


>
>>   Thanks,
>>
>>       Matt
>>
>>
>>> Thanks!
>>>
>>> Sam
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220611/b80697c4/attachment.html>