[petsc-users] A number of questions about DMDA with SNES and Quasi-Newton methods

[Smith, Barry F.]

> On Oct 31, 2017, at 11:59 AM, zakaryah . <zakaryah at gmail.com> wrote:
> 
> First I set up the composite DM, which contains one redundant field coupled to everything else, and a DMDA, which of course only has local couplings given by the stencil.  I then create the matrix with DMCreateMatrix, with the composite, packer, as the first argument.  I immediately call MatSetOption with MAT_NEW_NONZERO_ALLOCATION_ERR set to FALSE.  The matrix values are set by first calling MatGetLocalSubMatrix, and then MatSetValuesLocal.  Apparently this is working, but when I move to larger problems, setting the off-diagonal submatrices is very slow, even when I set the entire submatrix in a single call to MatSetValuesLocal.  I call the submatrices Jbb, Jbh, Jhb, and Jhh, where b is the redundant field and h are the displacements in the DMDA.  Then Jbh is a 1xN matrix where N is the size of the DMDA, and all of its values are non-zero.  When N is around 3e6, setting the values in Jbh takes several hours, even when I only call MatSetValuesLocal on it once.
> 
> I think the preallocation for the DMDA is working well, because if I forego setting the "off-diagonal" submatrices Jbh and Jhb, then setting Jhh is very fast (seconds).  I assume it's the allocation on the fly that is killing performance, because Jhh has 57 times as many non-zero elements as Jbh and Jhb, but setting it is several orders of magnitudes FASTER.
>