<div dir="ltr">On Tue, Sep 10, 2013 at 11:25 PM, Barry Smith <span dir="ltr"><<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>></span> wrote:<br><div class="gmail_extra"><div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im"><br>
On Sep 10, 2013, at 8:55 PM, Jed Brown <<a href="mailto:jedbrown@mcs.anl.gov">jedbrown@mcs.anl.gov</a>> wrote:<br>
<br>
> Matthew Knepley <<a href="mailto:knepley@gmail.com">knepley@gmail.com</a>> writes:<br>
>>> PetscSF cannot currently scatter individual bytes (4 bytes minimum), and<br>
>>> even if it could, it's a horribly inefficient representation (4 or 8<br>
>>> bytes of metadata for each byte of payload). The quick fix at that<br>
>>> moment was to send in units of size PetscInt (the struct was always<br>
>>> going to be divisible by that size).<br>
<br>
</div> Since when are PETSc people interested in quick fixes, we are only concerned with doing things right; quick fixes lead to bad code and bad ideas.<br>
<div class="im"><br>
>>><br>
>><br>
>> The way I understand it, he never had a single MPI_BYTE, but always a bunch.<br>
>> Shouldn't that packing handle that case?<br>
><br>
> In DMPlexDistributeData, your fieldSF is blown up so that every unit has<br>
> its own metadata (rank, offset). That's a terrible mechanism for moving<br>
> structs with hundreds of bytes.<br>
><br>
> I would rather not make PetscSF deal with fully-heterogeneous data<br>
> (where each node can have a different size) because it makes indexing a<br>
> nightmare (you need a PetscSection or similar just to get in the door; I<br>
> don't want PetscSF to depend on PetscSection).<br>
<br>
</div> Having PetscSF depend on PetscSection would be insane (after all PetscSection has information about constrained variables and other silly nonsense) but having PetscSF depend on XX makes perfect sense.<br>
<br>
A review:<br>
<br>
(current situation; with unimportant stuff removed)<br>
<br>
+ sf - star forest<br>
. nroots - number of root vertices on the current process (these are possible targets for other process to attach leaves)<br>
. nleaves - number of leaf vertices on the current process, each of these references a root on any process<br>
. ilocal - locations of leaves in leafdata buffers, pass NULL for contiguous storage<br>
. iremote - remote locations of root vertices for each leaf on the current process<br>
<br>
PetscSFSetGraph(PetscSF sf,PetscInt nroots,PetscInt nleaves,PetscInt *ilocal,,PetscSFNode *iremote,)<br>
<br>
typedef struct {<br>
PetscInt rank; /* Rank of owner */ (why is this not PetscMPIInt?)<br>
PetscInt index; /* Index of node on rank */<br>
} PetscSFNode;<br>
<br>
abstractly nleaves, ilocal are the indices (into an array) you are copying too; iremote are the indices (into an array) you are copying from<br>
<br>
Now if PETSc had an abstract concept of indexing (funny no one ever put that in PETSc decades ago) it would look like<br>
<br>
PetscSFSetGraph(PetscSF sf,PetscInt nroots, toIndices , fromIndices)<br>
<br>
but wait; PETSc does have have an abstraction for indexing (into regular arrays) called IS, come to think of it, PETSc has another abstraction for indexing (into arrays with different sized items) called XX. Actually thinking a tiny bit more one realizes that there is a third: PetscSFNode is a way of indexing (into regular arrays) on a bunch of different processes. We come up with a couple more related, but inconsistent syntaxes for indexing, we'll have code has good as Trilinos.<br>
<br>
So let's fix this up! One abstraction for indexing that handles both regular arrays, arrays with different size items and both kinds of arrays on a bunch of different processes; with simple enough syntax so it can be used whenever indexing is needed including passing to PetscSF! (We can ignore the need for MPI datatypes in communication for now).<br>
<br>
Concrete examples to demonstrate this need not be so difficult. A completely general set of indices for this situation could be handled by<br>
<br>
typedef struct {<br>
PetscInt nindices;<br>
PetscMPIInt *ranks;<br>
PetscInt *offsets;<br>
PetscInt *sizes;<br>
} XXData; Hardly more than PetscSFNode. Or could be handled as an array of structs.<br>
<br>
special cases would include:<br>
all on one process so ranks is not needed<br>
all items the same size so sizes not needed<br>
contiquous memory (or strided) so offsets not needed<br>
<br>
If we decided not to go with an abstract object but instead just a concrete data structure that could handle all cases it could look something like<br>
<br>
typedef struct {<br>
PetscInt nindices;<br>
PetscMPIInt *ranks;<br>
PetscInt *offsets;<br>
PetscInt *sizes;<br>
PetscInt chunksize,strideoffset;<br>
} XXData;<br>
<br>
For completeness I note with IS (for example in VecScatter creating with parallel vectors) we don't use rank,offset notation we use global offset == rstart[rank] + offset but that is something I'm sure we can deal with.<br>
<br>
<br>
QED<br>
<br>
Comments?<br>
</blockquote></div><br>This sounds like a good idea. So</div><div class="gmail_extra"><br></div><div class="gmail_extra"> - we always store indices in blocks (get rid of regular IS)</div><div class="gmail_extra"> - we can optimize for strided (get rid of strided IS)</div>
<div class="gmail_extra"> - they can be local or nonlocal (get rid of PetscSFNode)</div><div class="gmail_extra"><br></div><div class="gmail_extra">Right now, I have a GlobalSection, which uses XX but puts in different values. It has negative offsets</div>
<div class="gmail_extra">and sizes for nonlocal indices. Now I could just allocate the ranks array and check for different rank</div><div class="gmail_extra">rather than negative offset.</div><div class="gmail_extra"><br>
</div><div class="gmail_extra">This would increase the size of GlobalSection, but then I could shared indices with PetscSF, so overall</div><div class="gmail_extra">we could have a decrease. So it seems like PetscSection and PetscSF could share a substantial</div>
<div class="gmail_extra">amount of code if we go to a common data structure.</div><div class="gmail_extra"><br></div><div class="gmail_extra">I believe all this is orthogonal to the packing issue, which I think can be done before we unify indices.</div>
<div class="gmail_extra">In PetscSectionCreateFieldSF() I choose a granularity for communication. It would be easy to check</div><div class="gmail_extra">for a common multiple of sizes here, and only check if MPI_Type is small. This is likely to be what we</div>
<div class="gmail_extra">want to do in general, even creating our own packed types for communication. I will try and code it up.</div><div class="gmail_extra"><br></div><div class="gmail_extra"> Matt<br clear="all"><div>
<br></div>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener
</div></div>