<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jun 27, 2017 at 11:41 AM, Hong <span dir="ltr"><<a href="mailto:hzhang@mcs.anl.gov" target="_blank">hzhang@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">For '-pc_type mg', we use '-mg_levels_0_pc_type redundant -mg_coarse_pc_<wbr>redundant_number <n>' to control num of processors for each coarse-grid solve, n<=np and n=np is the default which solves coarse-grid sequentially.<div><br></div><div>See petsc/src/snes/examples/<wbr>tutorials/runex48_4, for which</div><div>you can add option '-mg_coarse_pc_redundant_<wbr>number 2' to have two subcommunicators at coarse-grid level.</div><div><br></div><div>Can you use pcredundant for gamg?</div></div></blockquote><div><br></div><div>Humm, maybe. I don't know the number of processor before the run, But I could manually set the flag or have a functional interface to set this, does that exist?</div><div><br></div><div>Since this is for a redundant solve I assume the coarse grid has to be an factor in the number of global processors. Is that right? I don't do that now but I suppose I could change my logic. Not a big deal. (I could also look at Telescope parameters in the process and try to align the two)</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><span class="HOEnZb"><font color="#888888"><div><br></div><div>Hong</div></font></span></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jun 27, 2017 at 9:46 AM, Mark Adams <span dir="ltr"><<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span>On Tue, Jun 27, 2017 at 8:35 AM, Matthew Knepley <span dir="ltr"><<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span>On Tue, Jun 27, 2017 at 6:36 AM, Mark Adams <span dir="ltr"><<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">In talking with Garth, this will not work.<div><br></div><div>I/we am now thinking that we should replace the MG object with Telescope. Telescope seems to be designed to be a superset of MG. Telescope does the processor reduction, and GAMG does as well, so we would have to reconcile this. Does this sound like a good idea? Am I missing anything important?</div></div></blockquote><div><br></div></span><div>I don't think "replace" is the right word. Telescope only does process reduction. It does not do control flow for solvers,</div><div>or restriction/prolongation. You can see telescope interacting with MG here</div></div></div></div></blockquote><div><br></div></span><div>Oh it is not the answer at all!</div><span><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><br></div><div> <a href="https://arxiv.org/abs/1604.07163" target="_blank">https://arxiv.org/abs/1604.0<wbr>7163</a></div><div><br></div><div>I think more of this should be "default", in that the options are turned on if you are running GMG on a large number of procs.</div><div><br></div><div>I also think GAMG should reuse the telescope code for doing reduction, but I am not sure how hard this is. Mark?</div></div></div></div></blockquote><div><br></div></span><div>There is a little logic in there for selecting the number of processors on the coarse grid. I think the way we integrate this is to integrate parameters, if we want. I take a hint on the number of equations to try to keep on a process (nnz would be better). I don't take hints on cluster size, I don't reduce the number of processors by an integer. I could change these to be more in line with Telescope, but that does not solve our problem.</div><div><br></div><div>Does PETSc now support matrices with an LHS and RHS communicator? I think it does. I could just make sub communicators for each level in the GAMG setup. I run PtAP and the see what I get and reduce the number of processors (using something like MatGetSubMtrix as I recall, to aggregate the matrix), and repartition if desired (one should).</div><div><br></div><div>Would it make sense for me to do this in GAMG, and see if it breaks anything in MG?</div><span><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><span><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Mark</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jun 27, 2017 at 4:48 AM, Mark Adams <span dir="ltr"><<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Parallel coarse grid solvers are a bit broken at large scale where you don't want to use all processors on the coarse grid. The ideal thing might be to create a sub communicator, but it's not clear how to integrate this in (eg, check if the sub communicator exists before calling the coarse grid solver and convert if necessary). A bit messy. It would be nice if a parallel direct solver would not redistribute the matrix, but then it would be asking too much for it to reorder also, so we could have a crappy ordering. So maybe the first option would be best long term.<div><br></div><div>I see we have MUMPS and<span style="white-space:pre-wrap"> </span>PaStiX. Do either of these not redistribute if asked?</div></div>
</blockquote></div><br></div>
</blockquote></span></div><span class="m_-798788731186575411m_-8660978938084322145HOEnZb"><font color="#888888"><br><br clear="all"><div><br></div>-- <br><div class="m_-798788731186575411m_-8660978938084322145m_433039802326153291gmail_signature"><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.caam.rice.edu/~mk51/" target="_blank">http://www.caam.rice.edu/~mk51<wbr>/</a><br></div></div></div>
</font></span></div></div>
</blockquote></span></div><br></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div></div>