[petsc-users] handling multi physics applications on multiple MPI_Comm

[Manav Bhatia]

Thanks for your comments, Matt. 

I have a fluid-structural application with a really large fluid discretization and a really small structural discretization. Due to the relative difference in size, I have defined the structural system on only a single node, and the fluid system on (say) N nodes. 

So far, I have hand-coded a Schur-Complement for a frequency-domain analysis that is able to handle the difference in comms. 

I am attempting to migrate to the nested matrix constructs for some future work, and was looking at the possibility of reusing the same distribution of comms. Additionally, I am looking to add additional disciplines and was considering the possibility of defining the systems on different comms. 

I wasn’t sure if I was creating more problems with this approach than what I was trying to solve.

Would you recommend that all objects exist on a global_comm so that there is no confusion about these operations? 

Thanks,
Manav



> On Jul 25, 2016, at 3:21 PM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> On Mon, Jul 25, 2016 at 1:13 PM, Manav Bhatia <bhatiamanav at gmail.com <mailto:bhatiamanav at gmail.com>> wrote:
> Hi,
> 
>     I have a multi physics application with discipline1 defined on comm1 and discipline2 on comm2.
> 
>     My intent is to use the nested matrix for the KSP solver where each diagonal block is provided by the disciplines, and the off-diagonal blocks are defined as shell-matrices with matrix vector products.
> 
>     I am a bit unclear about how to deal with the case of different set of processors on comm1 and comm2. I have the following questions and would appreciate some guidance:
> 
> — Would it make sense to define a comm_global as a union of comm1 and comm2 for the MatCreateNest?
> 
> — The diagonal blocks are available on comm1 and comm2 only. Should MatAssemblyBegin/End for these diagonal blocks be called on comm1 and comm2 separately?
> 
> — What comm should be used for the off-diagonal shell matrices?
> 
> — Likewise, when calling VecGetSubVector and VecRestoreSubVector to get sub-vectors corresponding to discipline1 (or 2), what comm should these function calls be made?
> 
> I would first ask if you have a convincing reason for doing this, because it sounds like the genesis of a million programming errors.
> 
> All the linear algebra objects would have to be in a global comm that contained any subcomms you want to use. I don't
> think it would make sense to define submatrices on subcomms. You can have your assembly code run on a subcomm certainly,
> but again this is a tricky business and I find it hard to understand the gain.
> 
>    Matt
>  
> Thanks,
> Manav
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20160725/1e5fb2dd/attachment.html>