<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">Thanks for your comments, Matt. </div><div class=""><br class=""></div><div class="">I have a fluid-structural application with a really large fluid discretization and a really small structural discretization. Due to the relative difference in size, I have defined the structural system on only a single node, and the fluid system on (say) N nodes. </div><div class=""><br class=""></div><div class="">So far, I have hand-coded a Schur-Complement for a frequency-domain analysis that is able to handle the difference in comms. </div><div class=""><br class=""></div><div class="">I am attempting to migrate to the nested matrix constructs for some future work, and was looking at the possibility of reusing the same distribution of comms. Additionally, I am looking to add additional disciplines and was considering the possibility of defining the systems on different comms. </div><div class=""><br class=""></div><div class="">I wasn’t sure if I was creating more problems with this approach than what I was trying to solve.</div><div class=""><br class=""></div><div class="">Would you recommend that all objects exist on a global_comm so that there is no confusion about these operations? </div><div class=""><br class=""></div><div class="">Thanks,</div><div class="">Manav</div><div class=""><br class=""></div><div class=""><br class=""></div><br class=""><div><blockquote type="cite" class=""><div class="">On Jul 25, 2016, at 3:21 PM, Matthew Knepley <<a href="mailto:knepley@gmail.com" class="">knepley@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_extra"><div class="gmail_quote">On Mon, Jul 25, 2016 at 1:13 PM, Manav Bhatia<span class="Apple-converted-space"> </span><span dir="ltr" class=""><<a href="mailto:bhatiamanav@gmail.com" target="_blank" class="">bhatiamanav@gmail.com</a>></span><span class="Apple-converted-space"> </span>wrote:<br class=""><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;">Hi,<br class=""><br class=""> <span class="Apple-converted-space"> </span>I have a multi physics application with discipline1 defined on comm1 and discipline2 on comm2.<br class=""><br class=""> <span class="Apple-converted-space"> </span>My intent is to use the nested matrix for the KSP solver where each diagonal block is provided by the disciplines, and the off-diagonal blocks are defined as shell-matrices with matrix vector products.<br class=""><br class=""> <span class="Apple-converted-space"> </span>I am a bit unclear about how to deal with the case of different set of processors on comm1 and comm2. I have the following questions and would appreciate some guidance:<br class=""><br class="">— Would it make sense to define a comm_global as a union of comm1 and comm2 for the MatCreateNest?<br class=""><br class="">— The diagonal blocks are available on comm1 and comm2 only. Should MatAssemblyBegin/End for these diagonal blocks be called on comm1 and comm2 separately?<br class=""><br class="">— What comm should be used for the off-diagonal shell matrices?<br class=""><br class="">— Likewise, when calling VecGetSubVector and VecRestoreSubVector to get sub-vectors corresponding to discipline1 (or 2), what comm should these function calls be made?<br class=""></blockquote><div class=""><br class=""></div><div class="">I would first ask if you have a convincing reason for doing this, because it sounds like the genesis of a million programming errors.</div><div class=""><br class=""></div><div class="">All the linear algebra objects would have to be in a global comm that contained any subcomms you want to use. I don't</div><div class="">think it would make sense to define submatrices on subcomms. You can have your assembly code run on a subcomm certainly,</div><div class="">but again this is a tricky business and I find it hard to understand the gain.</div><div class=""><br class=""></div><div class=""> Matt</div><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;">Thanks,<br class="">Manav<br class=""></blockquote></div>--<span class="Apple-converted-space"> </span><br class=""><div class="gmail_signature" data-smartmail="gmail_signature">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">-- Norbert Wiener</div></div></div></div></blockquote></div><br class=""></body></html>