<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Wed, Nov 8, 2017 at 2:27 PM, Jed Brown <span dir="ltr"><<a href="mailto:jed@jedbrown.org" target="_blank">jed@jedbrown.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">Matthew Knepley <<a href="mailto:knepley@gmail.com">knepley@gmail.com</a>> writes:<br>
<br>
> On Wed, Nov 8, 2017 at 1:49 PM, Jed Brown <<a href="mailto:jed@jedbrown.org">jed@jedbrown.org</a>> wrote:<br>
><br>
>> Matthew Knepley <<a href="mailto:knepley@gmail.com">knepley@gmail.com</a>> writes:<br>
>><br>
>> >> > No, this is the right structure.<br>
>> >><br>
>> >> Oh come on. You're defending a quadratic algorithm.<br>
>> >><br>
>> >> ierr = ParMETIS_V3_PartKway(vtxdist, xadj, adjncy, vwgt, adjwgt,<br>
>> >> &wgtflag, &numflag, &ncon, &nparts, tpwgts, ubvec, options, &edgeCut,<br>
>> >> assignment, &comm);<br>
>> >> // ...<br>
>> >> for (p = 0, i = 0; p < nparts; ++p) {<br>
>> >> for (v = 0; v < nvtxs; ++v) {<br>
>> >> if (assignment[v] == p) points[i++] = v;<br>
>> >> }<br>
>> >> }<br>
>> >><br>
>> >> MatPartitioningApply creates an IS with "assignment" and can be<br>
>> >> converted to a global numbering with ISPartitioningToNumbering. You<br>
>> >> could as well have an ISPartitioningToSectionAndIS() that produces your<br>
>> >> representation, preferably without this silly quadratic algorithm.<br>
>> >><br>
>> ><br>
>> > Time it. Tell me if it matters. Telling me it matters in the long run is<br>
>> > metaphysics.<br>
>><br>
>> I realize ParMETIS isn't scalable and that if you have a modest number<br>
>> of parts and only a million or so elements per rank, the cost of what<br>
>> you do here will be acceptable for most uses.<br>
>><br>
>> But you didn't refute my point that ISPartitioningToSectionAndIS can<br>
>> produce the representation you want.<br>
><br>
><br>
> I do not think its an either or thing. Many equivalent interfaces are<br>
> possible,<br>
> so I should have told Vaclav "I think this is the right one", but I thought<br>
> that<br>
> was implicit in me being the responder, and none of us thinking that there<br>
> is<br>
> a true "right" in interface design.<br>
><br>
><br>
>> The IS you're creating is similar<br>
>> to the inverse of the Numbering (which is a permutation). You are the<br>
>> one that replaced a scalable algorithm that has existed for a long time<br>
>> and uses types correctly with PetscPartitioner which has some ugly<br>
>> warts, duplicates a lot of code, and isn't a viable replacement for<br>
>> MatPartitioning.<br>
><br>
><br>
> 1) MatPartitioning is not a replacement for what I do. According to you,<br>
> that<br>
> should end the argument right there.<br>
><br>
> 2) Deep inside MatPartitioning, it must split things up as I do in order to<br>
> pack<br>
> stuff to be sent. I think it should be exposed as I have done.<br>
<br>
</div></div>Pack what stuff to be sent? PetscPartitionerPartition() takes the<br>
arguments to ParMETIS directly as arrays. To use MatPartitioning, you<br>
pass those same arrays to MatCreateMPIAdj which literally just holds the<br>
pointers. Then MatPartitioningApply passes those on to ParMETIS or<br>
whichever partitioning package. PetscPartitionerPartition_*<br>
implementations depends on DM strictly as a way to define the weights.<br>
That isn't more general or better; it's more restrictive.<br></blockquote><div><br></div><div>This sounds like deliberate misunderstanding. We are not talking about the input to</div><div>ParMetis, but the output. The biggest shortcoming of ParMetis (and indeed all mesh</div><div>partitioners) is that they tell you what must move, but do not move it. This is the value</div><div>add of Plex, it moves the mesh (including all connectivity) and the field data according</div><div>to the partition. In order to do this communication, you always want the partition segmented</div><div>by rank, as it is in the Section/IS output.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I see no reason why PetscPartitioner cannot be implemented as a very<br>
small amount of code around MatPartitioning (little more than the<br>
interface function). The opposite would necessarily bring in concepts<br>
like DM that are unnecessary when a user is partitioning matrices and<br>
graphs.</blockquote><div><br></div><div>I am not arguing that it should not be done. I am saying</div><div> - I do not want to do it, because the payoff is small</div><div> - I do not want it to break anything that currently works</div><div> - I do not want it to break the interface we currently have to PetscPartitioner because I think it is genuinely good</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">
> 3) I think you exaggerate the faults of the code for effect. Do what you<br>
> must.<br>
><br>
><br>
>> And now you seem to be arguing against Vaclav unifying<br>
>> it, claiming technical rationale that don't appear to hold up.<br>
><br>
><br>
> Of course, I disagree with you, and think you make no attempt to understand<br>
> why<br>
> someone would write this code, because you never solve the problems its<br>
> necessary<br>
> for.<br>
<br>
</span>I have written a parallel unstructured FEM that did this sort of<br>
partitioning and redistribution on the fly. I wrote it in terms of<br>
MatPartitioning and it worked fine. I abandoned it because it had<br>
screwy dependencies and you had abandoned Sieve to work on<br>
DMComplex/DMPlex. I'm happy to have made that change, but the concepts<br>
and requirements are hardly foreign to me.</blockquote><div><br></div><div>Worked fine for you :) It always different with other people.</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">
>> I don't<br>
>> understand why, but it certainly looks like PetscPartitioner can be<br>
>> implemented as a thin layer around MatPartitioning, then that layer can<br>
>> be refactored into helper functions at the IS/PetscSection level.<br>
><br>
><br>
> It might be true. To me, it looked like Mat stuff was draped on so it would<br>
> be hard<br>
> to pull the raw result out of the partitioner, and that things would get<br>
> created (like<br>
> the MatMult Scatter) which I did not want.<br>
<br>
</span>Barry created the MPIADJ representations in 1997 precisely for this purpose.<br>
</blockquote></div><br>Fair.</div><div class="gmail_extra"><br></div><div class="gmail_extra"> Matt<br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.caam.rice.edu/~mk51/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div>
</div></div>