<div dir="ltr"><div dir="ltr"><div dir="ltr">On Thu, Feb 21, 2019 at 11:16 AM Thibaut Appel via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<font size="-1" face="Arial, sans-serif"><span style="font-size:10pt">Dear PETSc developers/users,</span></font><br>
<font size="-1" face="Arial, sans-serif">
</font><br>
<font size="-1" face="Arial, sans-serif"><span style="font-size:10pt">I’m solving linear PDEs on a regular grid with high-order
finite differences, assembling an MPIAIJ matrix to solve linear
systems or eigenvalue problems. I’ve been using vertex major,
natural ordering for the parallelism with PetscSplitOwnership
(yielding rectangular slices of the physical domain) and wanted
to move to DMDA to have a more square-ish domain decomposition
and minimize communication between processes.</span></font><br>
<font size="-1" face="Arial, sans-serif"><span style="font-size:10pt"></span></font><br>
<font size="-1" face="Arial, sans-serif"><span style="font-size:10pt">However, my application is memory critical, and I have
finely-tuned matrix preallocation routines for allocating memory
“optimally”. It seems the memory of a DMDA matrix is allocated
along the value of the stencil width of DMDACreate and the
manual says about it</span></font><br>
<font size="-1" face="Arial, sans-serif">
</font><br>
<font size="-1" face="Arial, sans-serif"><span style="font-size:10pt">“</span>These DMDA stencils have nothing directly to do
with any finite difference stencils one might chose to use for a
discretization”</font><br>
<font size="-1" face="Arial, sans-serif">
</font><br>
<font size="-1" face="Arial, sans-serif">And despite reading the
manual pages there must be something I do not understand in the DM
topology, what is that "stencil width" for then? I will not use
ghost values for my FD-method, right?</font></div></blockquote><div><br></div><div>What this is saying is, "You might be using some stencil that is not STAR or BOX, but we are preallocating according to one of those".</div><div>If you really care about how much memory is preallocated, which it seems you do, then you might be able to use</div><div><br></div><div> <a href="https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMDA/DMDASetBlockFills.html">https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMDA/DMDASetBlockFills.html</a></div><div><br></div><div>to tell use exactly how to preallocate.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF">
<font size="-1" face="Arial, sans-serif">I was then wondering if I
could just create a MPIAIJ matrix, and with a PETSc routine get
the global indices of the domain for each process: in other words,
an equivalent of PetscSplitOwnership that gives me the DMDA
unknown ordering. So I can feed and loop on that in my
preallocation and assembly routines.</font></div></blockquote><div><br></div><div>You can make an MPIAIJ matrix yourself of course. It should have the same division of rows as the DMDA division of dofs. Also, MatSetValuesStencil() will not work for a custom matrix.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF">
<font size="-1" face="Arial, sans-serif">T</font><font size="-1" face="Arial, sans-serif">hanks very much,<span style="font-size:10pt"></span></font><br>
<font size="-1" face="Arial, sans-serif"><span style="font-size:10pt"> </span></font><br>
<font size="-1" face="Arial, sans-serif">
<span style="font-size:10pt">Thibaut</span></font>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div></div>