<div dir="ltr"><div dir="ltr">On Tue, Jun 2, 2020 at 4:49 AM Lawrence Mitchell <<a href="mailto:wence@gmx.li">wence@gmx.li</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">> On 2 Jun 2020, at 09:35, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br>
> <br>
> On Tue, Jun 2, 2020 at 4:25 AM Lawrence Mitchell <<a href="mailto:wencel@gmail.com" target="_blank">wencel@gmail.com</a>> wrote:<br>
> Hi Dave,<br>
> <br>
> > On 2 Jun 2020, at 05:43, Dave May <<a href="mailto:dave.mayhem23@gmail.com" target="_blank">dave.mayhem23@gmail.com</a>> wrote:<br>
> > <br>
> > <br>
> > <br>
> > On Tue 2. Jun 2020 at 03:30, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br>
> > On Mon, Jun 1, 2020 at 7:03 PM Danyang Su <<a href="mailto:danyang.su@gmail.com" target="_blank">danyang.su@gmail.com</a>> wrote:<br>
> > Thanks Jed for the quick response. Yes I am asking about the repartitioning of coarse grids in geometric multigrid for unstructured mesh. I am happy with AMG. Thanks for letting me know.<br>
> > <br>
> > All the pieces are there, we just have not had users asking for this, and it will take some work to put together.<br>
> > <br>
> > Matt - I created a branch for you and Lawrence last year which added full support for PLEX within Telescope. This implementation was not a fully automated algmoeration strategy - it utilized the partition associated with the DM returned from DMGetCoarseDM. Hence the job of building the distributed coarse hierarchy was let to the user.<br>
> > <br>
> > I’m pretty sure that code got merged into master as the branch also contained several bug mixes for Telescope. Or am I mistaken?<br>
> <br>
> I think you're right. I didn't manage to get the redistribution of the DMPlex object done last summer (it's bubbling up again).<br>
> <br>
> As I see it, for redistributed geometric multigrid on plexes, the missing piece is a function:<br>
> <br>
> DMPlexRedistributeOntoComm(DM old, MPI_Comm comm, DM *new)<br>
> <br>
> I went down a rabbit hole of trying to do this, since I actually think this replaced the current interface to DMPlexDistribute, which is<br>
> <br>
> DMPlexDistribute(DM old, PetscInt overlap, PetscSF *pointDistSF, DM *new)<br>
> <br>
> Where the new DM comes out on the same communicator as the old DM, just with a different partition.<br>
> <br>
> This has lots of follow-on consequences, for example, one can't easily load on P processes and then compute on Q.<br>
> <br>
> Unfortunately, collectiveness over MPI_Comm(old) is baked into the redistribution routines everywhere, and I didn't manage to finish things.<br>
> <br>
> Yes, I remember thinking this out. I believe the conclusion was that redistribution should happen on the large comm, which<br>
> some fraction of processes getting no cells. Then at the end we call one new function which copies that DM onto the smaller comm.<br>
<br>
I think this kind of works, but I worry that it makes the following usecase really painful.<br>
<br>
1. Parallel load on P processes<br>
2. Redistribute onto Q > P processes<br>
... Compute<br>
<br>
3. (Occasionally), checkpoint by mapping onto P processes and dumping<br>
<br>
Perhaps I don't understand your proposal properly, but I think you're saying that the _input_ DM in redistribute defines the processes that participate in the redistribution (and the _output_ DM just controls the size).<br>
<br>
This would then require me in my example to load on Q processes (not P) or else do some futzing around.<br>
<br>
Or do I have the wrong end of the stick?<br>
<br>
I think I would prefer an interface where I go from<br>
<br>
commOld -> commNew<br>
<br>
with the number of partitions determined by size(commNew)<br>
<br>
Then the restriction you have on the two communicators is one of:<br>
<br>
i. commOld \subseteq commNew<br>
<br>
ii. commOld \supseteq commNew<br>
<br>
I guess if you wanted to be really fancy you could make an intercommunicator of commOld \cap commNew, but now you have to carry another communicator around with you for data transfer etc...<br>
<br>
Perhaps internally you do this by making a DM on whichever the smaller of the two communicators is that is empty on a bunch of ranks and then copying in (or copying out) appropriately.<br>
<br>
I think a first step here is to break the assumption everywhere in the redistribution interface that the size of commOld determines the number of partitions you want, and instead move to using size(commNew). Then for now we just pass commNew=commOld.<br></blockquote><div><br></div><div>I almost agree. I still think we do not change Distribute(), since it is really convenient, but we do check sizes on input as you say.</div><div>We either,</div><div><br></div><div> 1) Copy the DM to a larger comm with empty slots on input</div><div><br></div><div>or</div><div><br></div><div> 2) Copy the DM to a smaller comm eliminating empty slots on output</div><div><br></div><div>depending on P_in < P_out, or the reverse.</div><div><br></div><div> THanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Lawrence<br></blockquote></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>