<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Thanks for the discussion. I think we got further.<div class=""><br class=""><div><blockquote type="cite" class=""><div class="">8. 11. 2017 v 20:42, Matthew Knepley <<a href="mailto:knepley@gmail.com" class="">knepley@gmail.com</a>>:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" style="font-family: Menlo-Regular; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_extra"><div class="gmail_quote">On Wed, Nov 8, 2017 at 2:27 PM, Jed Brown<span class="Apple-converted-space"> </span><span dir="ltr" class=""><<a href="mailto:jed@jedbrown.org" target="_blank" class="">jed@jedbrown.org</a>></span><span class="Apple-converted-space"> </span>wrote:<br class=""><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div class="HOEnZb"><div class="h5">Matthew Knepley <<a href="mailto:knepley@gmail.com" class="">knepley@gmail.com</a>> writes:<br class=""><br class="">> On Wed, Nov 8, 2017 at 1:49 PM, Jed Brown <<a href="mailto:jed@jedbrown.org" class="">jed@jedbrown.org</a>> wrote:<br class="">><br class="">>> Matthew Knepley <<a href="mailto:knepley@gmail.com" class="">knepley@gmail.com</a>> writes:<br class="">>><br class="">>> >> > No, this is the right structure.<br class="">>> >><br class="">>> >> Oh come on. You're defending a quadratic algorithm.<br class="">>> >><br class="">>> >> ierr = ParMETIS_V3_PartKway(vtxdist, xadj, adjncy, vwgt, adjwgt,<br class="">>> >> &wgtflag, &numflag, &ncon, &nparts, tpwgts, ubvec, options, &edgeCut,<br class="">>> >> assignment, &comm);<br class="">>> >> // ...<br class="">>> >> for (p = 0, i = 0; p < nparts; ++p) {<br class="">>> >> for (v = 0; v < nvtxs; ++v) {<br class="">>> >> if (assignment[v] == p) points[i++] = v;<br class="">>> >> }<br class="">>> >> }<br class="">>> >><br class="">>> >> MatPartitioningApply creates an IS with "assignment" and can be<br class="">>> >> converted to a global numbering with ISPartitioningToNumbering. You<br class="">>> >> could as well have an ISPartitioningToSectionAndIS() that produces your<br class="">>> >> representation, preferably without this silly quadratic algorithm.<br class="">>> >><br class="">>> ><br class="">>> > Time it. Tell me if it matters. Telling me it matters in the long run is<br class="">>> > metaphysics.<br class=""></div></div></blockquote></div></div></div></div></blockquote><div><br class=""></div><div><span style="font-family: Menlo-Regular;" class="">Jed replied exactly what I needed to know - whether there's some easy conversion from the natural IS output of MatPartitioningApply into IS+PetscSection DMPlexDistribute needs. He showed there actually is a way, and it won't have worse performance (theoretical performance benefit). Moreover, what you do now for every PetscPartitioner implementation (all those "Convert to PetscSection+IS" blocks) would be done just in one place (design benefit). In this context, the request for timing was slightly unfair, I think.</span></div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-family: Menlo-Regular; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div class="HOEnZb"><div class="h5">>><br class="">>> I realize ParMETIS isn't scalable and that if you have a modest number<br class="">>> of parts and only a million or so elements per rank, the cost of what<br class="">>> you do here will be acceptable for most uses.<br class="">>><br class="">>> But you didn't refute my point that ISPartitioningToSectionAndIS can<br class="">>> produce the representation you want.<br class="">><br class="">><br class="">> I do not think its an either or thing. Many equivalent interfaces are<br class="">> possible,<br class="">> so I should have told Vaclav "I think this is the right one", but I thought<br class="">> that<br class="">> was implicit in me being the responder, and none of us thinking that there<br class="">> is<br class="">> a true "right" in interface design.<br class="">><br class="">><br class="">>> The IS you're creating is similar<br class="">>> to the inverse of the Numbering (which is a permutation). You are the<br class="">>> one that replaced a scalable algorithm that has existed for a long time<br class="">>> and uses types correctly with PetscPartitioner which has some ugly<br class="">>> warts, duplicates a lot of code, and isn't a viable replacement for<br class="">>> MatPartitioning.<br class="">><br class="">><br class="">> 1) MatPartitioning is not a replacement for what I do. According to you,<br class="">> that<br class="">> should end the argument right there.<br class="">><br class="">> 2) Deep inside MatPartitioning, it must split things up as I do in order to<br class="">> pack<br class="">> stuff to be sent. I think it should be exposed as I have done.<br class=""><br class=""></div></div>Pack what stuff to be sent? PetscPartitionerPartition() takes the<br class="">arguments to ParMETIS directly as arrays. To use MatPartitioning, you<br class="">pass those same arrays to MatCreateMPIAdj which literally just holds the<br class="">pointers. Then MatPartitioningApply passes those on to ParMETIS or<br class="">whichever partitioning package. PetscPartitionerPartition_*<br class="">implementations depends on DM strictly as a way to define the weights.<br class="">That isn't more general or better; it's more restrictive.<br class=""></blockquote></div></div></div></div></blockquote><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-family: Menlo-Regular; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_extra"><div class="gmail_quote"><div class=""><br class=""></div><div class="">This sounds like deliberate misunderstanding. We are not talking about the input to</div><div class="">ParMetis, but the output. The biggest shortcoming of ParMetis (and indeed all mesh</div><div class="">partitioners) is that they tell you what must move, but do not move it. This is the value</div><div class="">add of Plex, it moves the mesh (including all connectivity) and the field data according</div><div class="">to the partition. In order to do this communication, you always want the partition segmented</div><div class="">by rank, as it is in the Section/IS output.</div></div></div></div></div></blockquote><div><br class=""></div><div>I think we are talking here about both input and output!</div><div><div><br class=""></div><div><font face="Menlo-Regular" class="">What PetscPartitionerPartition actually does is getting the adjacency information from the DM using</font></div><div><font face="Menlo-Regular" class="">DMPlexCreatePartitionerGraph(DM dm, PetscInt height, PetscInt *numVertices, PetscInt **offsets, PetscInt **adjacency, IS *globalNumbering)</font></div><div><font face="Menlo-Regular" class="">and these raw arrays are then passed to</font></div><div><font face="Menlo-Regular" class="">(*part->ops->partition)(part, dm, size, numVertices, start, adjacency, partSection, partition);</font></div><div><span style="font-family: Menlo-Regular;" class=""><br class=""></span></div><div><span style="font-family: Menlo-Regular;" class="">This I don't like from couple of reasons:</span></div><div><span style="font-family: Menlo-Regular;" class="">1) the type-specific function has completely different arguments than the interface function - this is slightly non-standard, and if you talk about exposing, there should be an interface function with these arguments</span></div><div><font face="Menlo-Regular" class="">2) on the other hand you derive IS+PetscSection output from the "raw" partitioner output inside each PetscPartitionerPartition_* separately </font><span style="font-family: Menlo-Regular;" class="">("Convert to PetscSection+IS" blocks)</span></div><div><span style="font-family: Menlo-Regular;" class="">3) so </span><span style="font-family: Menlo-Regular;" class="">part->ops->partition</span><span style="font-family: Menlo-Regular;" class=""> mixes "raw" inputs and "derived" outputs which sounds messy </span></div><div><br class=""></div><div><font face="Menlo-Regular" class="">I fully agree with Jed (and it seems it's compatible with Barry and Lisandro) this should be doable in a clearer way using </font><span style="font-family: Menlo-Regular;" class="">MatCreateMPIAdj</span><font face="Menlo-Regular" class="">, MatPartitioningSetAdjacency, </font><span style="font-family: Menlo-Regular;" class="">MatPartitioningApply, </span><span style="font-family: Menlo-Regular;" class="">ISPartitioningToSectionAndIS (the latter to be done)</span><span style="font-family: Menlo-Regular;" class="">.</span></div></div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-family: Menlo-Regular; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_extra"><div class="gmail_quote"><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;">I see no reason why PetscPartitioner cannot be implemented as a very<br class="">small amount of code around MatPartitioning (little more than the<br class="">interface function). The opposite would necessarily bring in concepts<br class="">like DM that are unnecessary when a user is partitioning matrices and<br class="">graphs.</blockquote><div class=""><br class=""></div><div class="">I am not arguing that it should not be done. I am saying</div><div class=""> - I do not want to do it, because the payoff is small</div><div class=""> - I do not want it to break anything that currently works</div><div class=""> - I do not want it to break the interface we currently have to PetscPartitioner because I think it is genuinely good</div></div></div></div></div></blockquote><div><br class=""></div><div>I think the payoff isn't necessarily small. Just imagine how much code could be avoided for further maintenance, or if there would be some new partitioner we wanted to interface. And I think the code much clearer to anyone who wants to learn how it works or even improve something.</div><div><br class=""></div><div>I'm not that afraid about breaking. Grep tells me there's exactly one call to <span style="font-family: Menlo-Regular;" class="">PetscPartitionerPartition in whole PETSc - in </span><font face="Menlo-Regular" class="">DMPlexDistribute.</font></div><div><br class=""></div><div>One thing I always liked on PETSc is a culture of not sticking to status quo!</div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-family: Menlo-Regular; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_extra"><div class="gmail_quote"><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><span class="">> 3) I think you exaggerate the faults of the code for effect. Do what you<br class="">> must.<br class="">><br class="">><br class="">>> And now you seem to be arguing against Vaclav unifying<br class="">>> it, claiming technical rationale that don't appear to hold up.<br class="">><br class="">><br class="">> Of course, I disagree with you, and think you make no attempt to understand<br class="">> why<br class="">> someone would write this code, because you never solve the problems its<br class="">> necessary<br class="">> for.<br class=""></span></blockquote></div></div></div></div></blockquote><div><br class=""></div><div>I understand you spent the most time with DMPlex, and it's a masterpiece! But it can always be beneficial to take into account a fresh view of someone who was not involved before.</div><div><br class=""></div><div>I think I need to create a proof-of-concept. I would start by employing MatPartitioning in PetscPartitionerPartition with anything outside of this function untouched for now (as already suggested in #192), if you agree.</div><div><br class=""></div><div>Or what about implementing a special temporary PetscPartitioner implementation wrapping MatPartitioning?</div><div>PETSCPARTITIONERMATPARTITIONING sounds crazy, though :) But could be a good starting point.</div><div><br class=""></div><div>Thanks</div><div><br class=""></div><div>Vaclav</div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-family: Menlo-Regular; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><span class=""><br class=""></span>I have written a parallel unstructured FEM that did this sort of<br class="">partitioning and redistribution on the fly. I wrote it in terms of<br class="">MatPartitioning and it worked fine. I abandoned it because it had<br class="">screwy dependencies and you had abandoned Sieve to work on<br class="">DMComplex/DMPlex. I'm happy to have made that change, but the concepts<br class="">and requirements are hardly foreign to me.</blockquote><div class=""><br class=""></div><div class="">Worked fine for you :) It always different with other people.</div><div class=""><br class=""></div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><span class="">>> I don't<br class="">>> understand why, but it certainly looks like PetscPartitioner can be<br class="">>> implemented as a thin layer around MatPartitioning, then that layer can<br class="">>> be refactored into helper functions at the IS/PetscSection level.<br class="">><br class="">><br class="">> It might be true. To me, it looked like Mat stuff was draped on so it would<br class="">> be hard<br class="">> to pull the raw result out of the partitioner, and that things would get<br class="">> created (like<br class="">> the MatMult Scatter) which I did not want.<br class=""><br class=""></span>Barry created the MPIADJ representations in 1997 precisely for this purpose.<br class=""></blockquote></div><br class="">Fair.</div><div class="gmail_extra"><br class=""></div><div class="gmail_extra"> Matt<br clear="all" class=""><div class=""><br class=""></div>--<span class="Apple-converted-space"> </span><br class=""><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">-- Norbert Wiener</div><div class=""><br class=""></div><div class=""><a href="http://www.caam.rice.edu/~mk51/" target="_blank" class="">https://www.cse.buffalo.edu/~knepley/</a></div></div></div></div></div></div></div></div></blockquote></div><br class=""></div></body></html>