[petsc-users] Nested SNES in FormFunction

[Matthew Knepley]

On Fri, May 21, 2021 at 9:34 AM Pierre Bernigaud <pierrebernigaudl at gmail.com>
wrote:

> Greetings,
>
> I am currently working on a CFD solver using PETSc. I have a non linear
> system which is solved using 2D_DMDA/SNES, and submitted to boundary
> conditions that are treated implicitly and updated in the FormFunction. The
> calculation of one of these boundary conditions requires the resolution of
> an other non linear system.
>
> I am hence using a nested 1D_DMDA/SNES system within the FormFunction of
> my main SNES solver to solve for this boundary condition. This is working
> fine, but doing a scalability study we found out that this causes the code
> to show sub-par acceleration properties.
>
> Have you ever encountered this kind of nested SNES application, and are
> there some critical points to be aware of in order to avoid a loss of
> performances?
> For instance, the sub 1D_DMDA/SNES objects are created and destroyed at
> each update of the boundary, hence at each call to FormFunction, which
> results in an important number of object manipulation. Could this be a
> problem?
>

Yes. You should keep this subsolver around for as long as the outer solver
lives.


> Furthermore, the use of a sub 1D_DMDA/SNES allows to use multiple
> processors to solve for the boundary condition, composed of say N cells.
> When running the code with M > N processors, everything is working great,
> but I am curious about the state of the (M-N) processors which aren’t
> working on boundary condition problem. Do they just stay idle?
>

Yes, they are idle. You could try to fold the boundary condition into the
same system as the bulk, but this is not a clear win since you will still
have load imbalance.
What percentage of time does the boundary solve take?

  Thanks,

     Matt


> Thank you for your help.
> Respectfully,
> Pierre Bernigaud



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20210521/56ee6421/attachment.html>