<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Yes I saw that, I am assuming you are referring to DMPlexCreateNumbering_Plex? <div class=""><br class=""></div><div class="">Since you are already building the IS it can be done in a more stream-lined way, since the way it is done in <span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">DMPlexCreateNumbering_Plex </span>has to loop over all the local points again, then build a layout, and finally all_gather. My implementation just gathers to root instead of all_gather, and uses cached min and max values already computed during the IS construction.</div><div class=""><div class=""><br class=""><div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div>Best regards,<br class=""><br class="">Jacob Faibussowitsch<br class="">(Jacob Fai - booss - oh - vitch)<br class="">Cell: (312) 694-3391</div></div>
</div>
<div><br class=""><blockquote type="cite" class=""><div class="">On Apr 16, 2020, at 8:56 PM, Matthew Knepley <<a href="mailto:knepley@gmail.com" class="">knepley@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><div dir="ltr" class="">On Thu, Apr 16, 2020 at 9:51 PM Jacob Faibussowitsch <<a href="mailto:jacob.fai@gmail.com" class="">jacob.fai@gmail.com</a>> wrote:<br class=""></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div style="overflow-wrap: break-word;" class=""><div class=""><blockquote type="cite" class=""><div dir="ltr" class=""><div class="gmail_quote"><div class="">This algorithm does not work for any stratum</div></div></div></blockquote></div>Shouldn’t it? The IS generated are local to global conversions. Here is my current approach.<div class=""><br class=""></div><div class="">Each Iocal IS will have local indices [0, 4) but the<span class="Apple-converted-space"> </span><u class="">value</u><span class="Apple-converted-space"> </span>of the index will be the global ID of the vertex, e.g. for 2 quads in a line (assuming as usual lowest rank owns points):<div class=""><div class=""><br class=""></div><div class="">0——1——2<br class=""><div class="">| P0 | P1 |</div><div class="">|. |. |</div><div class="">3——4——5</div><div class=""><br class=""><div class=""><div class=""><u class="">VERTICES:</u></div><div class="">IS Object: 2 MPI processes</div><div class=""> type: general</div><div class="">[0] Number of indices in set 4</div><div class="">[0] 0 0 </div><div class="">[0] 1 1</div><div class="">[0] 2 3</div><div class="">[0] 3 4</div><div class=""><div style="" class="">[1] Number of indices in set 4</div><div style="" class="">[1] 0 -1</div><div style="" class="">[1] 1 2</div><div style="" class="">[1] 2 -4</div><div style="" class="">[1] 3 5</div><div class=""><div class=""><div class=""><br class=""></div><div class="">So for total number of vertices the algorithm is:</div><div class=""><br class=""></div><div class="">1. Find max local IS value</div><div class="">2. Add 1 to it</div><div class="">3. All_reduce to proc 0</div><div class="">4. Find max of all_reduced values</div></div></div></div></div></div></div></div></div></div></blockquote><div class=""><br class=""></div><div class="">The code that calculates the L2G _already_ computed the global size, it just did not put it anywhere. If you are willing</div><div class="">to compute the L2G, you have already done what you want.</div><div class=""><br class=""></div><div class=""> Matt</div><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div style="overflow-wrap: break-word;" class=""><div class=""><div class=""><div class=""><div class=""><div class=""><div class=""><div class=""><div class=""><div class=""><div class=""><div dir="auto" style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class=""><div class="">Best regards,<br class=""><br class="">Jacob Faibussowitsch<br class="">(Jacob Fai - booss - oh - vitch)<br class="">Cell: (312) 694-3391</div></div></div><div class=""><br class=""><blockquote type="cite" class=""><div class="">On Apr 16, 2020, at 8:37 PM, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank" class="">knepley@gmail.com</a>> wrote:</div><br class=""><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class=""><div dir="ltr" class="">On Thu, Apr 16, 2020 at 9:28 PM Jacob Faibussowitsch <<a href="mailto:jacob.fai@gmail.com" target="_blank" class="">jacob.fai@gmail.com</a>> wrote:<br class=""></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div class=""><div class=""><div dir="auto" class=""><div class=""><blockquote type="cite" class=""><div dir="ltr" class=""><div class="gmail_quote"><div class="">What do you want to do? </div></div></div></blockquote>Count the global number of points per depth based on all_reduce’ing the maximum positive value returned from the IS’s listed below. This works as intended for anything but cells since global number of points = max(IS) + 1. For cells this breaks since 1 rank reports 3 as max, the next reports 6, etc.</div></div></div></div></blockquote><div class=""><br class=""></div><div class="">I do not understand this at all. This algorithm does not work for any stratum. For example, suppose that we have a straight line of quad cells,</div><div class="">1 per process. The vertices would be [1, 5) on all processes but there would be 2*(P + 1) vertices.</div><div class=""><br class=""></div><div class=""> <span class="Apple-converted-space"> </span>Thanks,</div><div class=""><br class=""></div><div class=""> Matt</div><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div class=""><div class=""><div dir="auto" class=""><div class=""><blockquote type="cite" class=""><div dir="ltr" class=""><div class="gmail_quote"><div class=""> The system should be flexible enough to distribute whatever you want</div></div></div></blockquote>What is the best way to check that a non-standard distribute has been done? </div><div class=""><br class=""></div></div></div><div class=""><div dir="auto" style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class=""><div class="">Best regards,<br class=""><br class=""></div><div class="">Jacob Faibussowitsch<br class="">(Jacob Fai - booss - oh - vitch)<br class="">Cell: (312) 694-3391</div></div></div><div class=""><br class=""><blockquote type="cite" class=""><div class="">On Apr 16, 2020, at 8:17 PM, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank" class="">knepley@gmail.com</a>> wrote:</div><br class=""><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class=""><div dir="ltr" class="">On Thu, Apr 16, 2020 at 9:04 PM Jacob Faibussowitsch <<a href="mailto:jacob.fai@gmail.com" target="_blank" class="">jacob.fai@gmail.com</a>> wrote:<br class=""></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div class="">Hello All,<div class=""><br class=""></div><div class="">TL;DR: Is it possible now, or is it a planned feature, for plex to distribute over anything but points with height = 0? </div><div class=""><br class=""></div><div class="">If I understand this correctly when plex currently partitions a graph, points with height 0 are only owned by a single process, but all other points can be co-owned by multiple procs. For example for a 2D plex with 8 vertices, 12 edges, and 6 cells over 2 procs these are the global-local IS’s for all points on processes (negative values indicate ownership by another proc) the final IS corresponding to cells will always have positive values as each proc is the sole owner of its cells.</div></div></blockquote><div class=""><br class=""></div><div class="">What do you want to do? The system should be flexible enough to distribute whatever you want, but the current guarantee</div><div class="">is that the cone of any points is always available. So if you decide to distribute something else, like faces, then it ends up</div><div class="">looking just like an overlapping mesh with some custom overlap. Moreover, the dual mesh only really makes sense for cells.</div><div class="">For faces/edges you would need a hypergraph partitioner.</div><div class=""><br class=""></div><div class=""> <span class=""> </span>Thanks,</div><div class=""><br class=""></div><div class=""> Matt</div><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div class=""><div class=""><u class="">VERTICES</u></div>IS Object: 2 MPI processes<br class=""> type: general<br class="">[0] Number of indices in set 7<br class="">[0] 0 -2<br class="">[0] 1 0<br class="">[0] 2 -3<br class="">[0] 3 -4<br class="">[0] 4 -5<br class="">[0] 5 -6<br class="">[0] 6 -8<br class="">[1] Number of indices in set 7<br class="">[1] 0 1<br class="">[1] 1 2<br class="">[1] 2 3<br class="">[1] 3 4<br class="">[1] 4 5<br class="">[1] 5 6<br class="">[1] 6 7<div class=""><br class=""></div><div class=""><u class="">EDGES</u><br class="">IS Object: 2 MPI processes<br class=""> type: general<br class="">[0] Number of indices in set 9<br class="">[0] 0 0<br class="">[0] 1 1<br class="">[0] 2 -4<br class="">[0] 3 -5<br class="">[0] 4 -6<br class="">[0] 5 2<br class="">[0] 6 -7<br class="">[0] 7 -9<br class="">[0] 8 -11<br class="">[1] Number of indices in set 9<br class="">[1] 0 3<br class="">[1] 1 4<br class="">[1] 2 5<br class="">[1] 3 6<br class="">[1] 4 7<br class="">[1] 5 8<br class="">[1] 6 9<br class="">[1] 7 10<br class="">[1] 8 11</div><div class=""><br class=""></div><div class=""><u class="">CELLS</u><br class="">IS Object: 2 MPI processes<br class=""> type: general<br class="">[0] Number of indices in set 3<br class="">[0] 0 0<br class="">[0] 1 1<br class="">[0] 2 2<br class="">[1] Number of indices in set 3<br class="">[1] 0 3<br class="">[1] 1 4<br class=""><div class="">[1] 2 5 </div><div class=""><br class=""><div class=""><div dir="auto" style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class=""><div class="">Best regards,<br class=""><br class="">Jacob Faibussowitsch<br class="">(Jacob Fai - booss - oh - vitch)<br class="">Cell: (312) 694-3391</div></div></div><br class=""></div></div></div></blockquote></div><br clear="all" class=""><div class=""><br class=""></div>--<span class=""> </span><br class=""><div dir="ltr" class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">-- Norbert Wiener</div><div class=""><br class=""></div><div class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" class="">https://www.cse.buffalo.edu/~knepley/</a></div></div></div></div></div></div></div></div></div></blockquote></div><br class=""></div></blockquote></div><br clear="all" class=""><div class=""><br class=""></div>--<span class=""> </span><br class=""><div dir="ltr" class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">-- Norbert Wiener</div><div class=""><br class=""></div><div class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" class="">https://www.cse.buffalo.edu/~knepley/</a></div></div></div></div></div></div></div></div></div></blockquote></div><br class=""></div></div></div></div></div></div></div></div></div></div></blockquote></div><br clear="all" class=""><div class=""><br class=""></div>--<span class="Apple-converted-space"> </span><br class=""><div dir="ltr" class="gmail_signature"><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">-- Norbert Wiener</div><div class=""><br class=""></div><div class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" class="">https://www.cse.buffalo.edu/~knepley/</a></div></div></div></div></div></div></div></div></div></blockquote></div><br class=""></div></div></body></html>