[petsc-users] Parallel vector layout for TAO optimization with separable state/design structure

[Jed Brown]

For a bit of assistance, you can use DMComposite and DMRedundantCreate; see src/snes/tutorials/ex21.c and ex22.c.

Note that when computing redundantly, it's critical that the computation be deterministic (i.e., not using atomics or randomness without matching seeds) so the logic stays collective.

This merge request may also be relevant and comments related to your needs would be welcome in the discussion. 

https://gitlab.com/petsc/petsc/-/merge_requests/6531

Barry Smith <bsmith at petsc.dev> writes:

>    This is a problem with MPI programming and optimization; I am unaware of a perfect solution.
>
>    Put the design variables into the solution vector on MPI rank 0, and when doing your objective/gradient, send the values to all the MPI processes where you use them. You can use a VecScatter to handle the communication you need or MPI_Scatter() etc whatever makes the most sense in your code. 
>
>    Barry
>
>
>> On Jan 30, 2024, at 10:53 AM, Guenther, Stefanie via petsc-users <petsc-users at mcs.anl.gov> wrote:
>> 
>> Hi Petsc team, 
>>  
>> I have a question regarding parallel layout of a Petsc vector to be used in TAO optimizers for cases where the optimization variables split into ‘design’ and ‘state’ variables (e.g. such as in PDE-constrained optimization as in tao_lcl). In our case, the state variable naturally parallelizes evenly amongst multiple processors and this distribution is fixed. The ‘design’ vector however does not, it is very short compared to the state vector and it is required on all state-processors when evaluating the objective function and gradient. My question would be how the TAO optimization vector x = [design,state] should be created in such a way that the ‘state’ part is distributed as needed in our solver, while the design part is not.
>>  
>> My only idea so far was to copy the design variables to all processors and augment / interleave the optimization vector as x = [state_proc1,design, state_proc2, design, … ] . When creating this vector in parallel on PETSC_COMM_WORLD, each processor would then own the same number of variables ( [state_proc<i>, design] ), as long as the numbers match up, and I would only need to be careful when gathering the gradient wrt the design parts from all processors.
>>  
>> This seems cumbersome however, and I would be worried whether the optimization problem is harder to solve this way. Is there any other way to achieve this splitting, that I am missing here? Note that the distribution of the state itself is given and can not be changed, and that the state vs design vectors have very different (and independent) dimensions.
>>  
>> Thanks for your help and thoughts!
>> Best,
>> Stefanie