[petsc-users] A number of questions about DMDA with SNES and Quasi-Newton methods

[zakaryah .]

First I set up the composite DM, which contains one redundant field coupled
to everything else, and a DMDA, which of course only has local couplings
given by the stencil.  I then create the matrix with DMCreateMatrix, with
the composite, packer, as the first argument.  I immediately call
MatSetOption with MAT_NEW_NONZERO_ALLOCATION_ERR set to FALSE.  The matrix
values are set by first calling MatGetLocalSubMatrix, and then
MatSetValuesLocal.  Apparently this is working, but when I move to larger
problems, setting the off-diagonal submatrices is very slow, even when I
set the entire submatrix in a single call to MatSetValuesLocal.  I call the
submatrices Jbb, Jbh, Jhb, and Jhh, where b is the redundant field and h
are the displacements in the DMDA.  Then Jbh is a 1xN matrix where N is the
size of the DMDA, and all of its values are non-zero.  When N is around
3e6, setting the values in Jbh takes several hours, even when I only call
MatSetValuesLocal on it once.

I think the preallocation for the DMDA is working well, because if I forego
setting the "off-diagonal" submatrices Jbh and Jhb, then setting Jhh is
very fast (seconds).  I assume it's the allocation on the fly that is
killing performance, because Jhh has 57 times as many non-zero elements as
Jbh and Jhb, but setting it is several orders of magnitudes FASTER.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20171031/97b59e5e/attachment-0001.html>