<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Mon, Jul 25, 2016 at 1:34 PM, Manav Bhatia <span dir="ltr"><<a href="mailto:bhatiamanav@gmail.com" target="_blank">bhatiamanav@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div>Thanks for your comments, Matt. </div><div><br></div><div>I have a fluid-structural application with a really large fluid discretization and a really small structural discretization. Due to the relative difference in size, I have defined the structural system on only a single node, and the fluid system on (say) N nodes. </div><div><br></div><div>So far, I have hand-coded a Schur-Complement for a frequency-domain analysis that is able to handle the difference in comms. </div><div><br></div><div>I am attempting to migrate to the nested matrix constructs for some future work, and was looking at the possibility of reusing the same distribution of comms. Additionally, I am looking to add additional disciplines and was considering the possibility of defining the systems on different comms. </div><div><br></div><div>I wasn’t sure if I was creating more problems with this approach than what I was trying to solve.</div><div><br></div><div>Would you recommend that all objects exist on a global_comm so that there is no confusion about these operations? </div></div></blockquote><div><br></div><div>Yes. I think the confusion here is between the problem you are trying to solve, and the tool for doing it.</div><div><br></div><div>Disparate size of subsystems seems to me to be a _load balancing_ problem. Here you can use data layout to alleviate this.</div><div>On the global comm, you can put all the fluid unknowns on ranks 0..N-2, and the structural unknowns on N-1. You can have</div><div>more general splits than that.</div><div><br></div><div>IF for some reason in the structural assembly you used a large number of collective operations (like say did artificial timestepping</div><div>to get to some steady state property), then it might make sense to pull out a subcomm of only the occupied ranks, but only above</div><div>1000 procs, and only on a non-BlueGene machine. This is also easily measure before you do this work.</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div>Thanks,</div><div>Manav</div><div><br></div><div><br></div><br><div><blockquote type="cite"><div>On Jul 25, 2016, at 3:21 PM, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:</div><br><div><div dir="ltr" style="font-family:Helvetica;font-size:12px;font-style:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><div class="gmail_extra"><div class="gmail_quote">On Mon, Jul 25, 2016 at 1:13 PM, Manav Bhatia<span> </span><span dir="ltr"><<a href="mailto:bhatiamanav@gmail.com" target="_blank">bhatiamanav@gmail.com</a>></span><span> </span>wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Hi,<br><br> <span> </span>I have a multi physics application with discipline1 defined on comm1 and discipline2 on comm2.<br><br> <span> </span>My intent is to use the nested matrix for the KSP solver where each diagonal block is provided by the disciplines, and the off-diagonal blocks are defined as shell-matrices with matrix vector products.<br><br> <span> </span>I am a bit unclear about how to deal with the case of different set of processors on comm1 and comm2. I have the following questions and would appreciate some guidance:<br><br>— Would it make sense to define a comm_global as a union of comm1 and comm2 for the MatCreateNest?<br><br>— The diagonal blocks are available on comm1 and comm2 only. Should MatAssemblyBegin/End for these diagonal blocks be called on comm1 and comm2 separately?<br><br>— What comm should be used for the off-diagonal shell matrices?<br><br>— Likewise, when calling VecGetSubVector and VecRestoreSubVector to get sub-vectors corresponding to discipline1 (or 2), what comm should these function calls be made?<br></blockquote><div><br></div><div>I would first ask if you have a convincing reason for doing this, because it sounds like the genesis of a million programming errors.</div><div><br></div><div>All the linear algebra objects would have to be in a global comm that contained any subcomms you want to use. I don't</div><div>think it would make sense to define submatrices on subcomms. You can have your assembly code run on a subcomm certainly,</div><div>but again this is a tricky business and I find it hard to understand the gain.</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Thanks,<br>Manav<span class="HOEnZb"><font color="#888888"><br></font></span></blockquote></div><span class="HOEnZb"><font color="#888888">--<span> </span><br><div data-smartmail="gmail_signature">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div></font></span></div></div></div></blockquote></div><br></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div>
</div></div>