[petsc-users] Nested SNES in FormFunction

[Pierre Bernigaud]

Greetings, 

I am currently working on a CFD solver using PETSc. I have a non linear system which is solved using 2D_DMDA/SNES, and submitted to boundary conditions that are treated implicitly and updated in the FormFunction. The calculation of one of these boundary conditions requires the resolution of an other non linear system. 

I am hence using a nested 1D_DMDA/SNES system within the FormFunction of my main SNES solver to solve for this boundary condition. This is working fine, but doing a scalability study we found out that this causes the code to show sub-par acceleration properties. 

Have you ever encountered this kind of nested SNES application, and are there some critical points to be aware of in order to avoid a loss of performances? 
For instance, the sub 1D_DMDA/SNES objects are created and destroyed at each update of the boundary, hence at each call to FormFunction, which results in an important number of object manipulation. Could this be a problem? 

Furthermore, the use of a sub 1D_DMDA/SNES allows to use multiple processors to solve for the boundary condition, composed of say N cells. When running the code with M > N processors, everything is working great, but I am curious about the state of the (M-N) processors which aren’t working on boundary condition problem. Do they just stay idle? 

Thank you for your help. 
Respectfully, 
Pierre Bernigaud