[petsc-dev] programming model for PETSc

[Dmitry Karpeev]

Is O(commSize) storage considered unscalable in this model?

Dmitry.

On Thu, Nov 24, 2011 at 5:54 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> On Thu, Nov 24, 2011 at 17:45, Matthew Knepley <knepley at gmail.com> wrote:
>
>> This not model my current model, but it is a possible one. I do not
>> "ghost" the layout, only
>> the points.
>>
>
> I wasn't asking that. How do you represent a ghosted point? I thought you
> do it by (rank, index). I have everything I need if the ghoster has (owner
> rank, index), or equivalently (modulo MPI_Scan), the global index of the
> point it is ghosting.
>
>
>> In that case, the fact that remote processes have (owner rank, offset)
>>> means that I can broadcast (local_offset, size) with purely local setup
>>> (this is the first primitive which can be implemented using MPI_Get()).
>>>
>>
>> Okay, I would really like this coded up. We can do a 1-D mesh  of
>> Lagrangian elements just to show me what is going on.
>>
>
> Sure, it's simple.
>
>
>> I am still not sure how much this buys you since you had to communicate
>> that offset info somehow.
>>
>
> It buys you that the next operation is non-synchronizing, has no
> user-visible packing, and does not have memory scalability issues if every
> process needs data from one point.
>
>
>>
>>
>>> Alternatively, given (owner rank, offset, size), we can literally call
>>> VecScatterCreate() after just an MPI_Scan(), which is logarithmic, and
>>> local setup. but VecScatterCreate() does lots of unnecessary setup to build
>>> the two-way representation.
>>>
>>>
>>> Redistributing a mesh after partitioning is slightly more demanding.
>>> First, senders are enumerated using a fetch-and-add initiated by the
>>> sending process which has the side-effect of counting the number of nodes
>>> that will be in the new partition and informing the senders of the offsets
>>> at which to deposit those nodes. Then we broadcast the (rank, offset) of
>>> each node on the sender to the receiver. Then we send connectivity using a
>>> non-uniform broadcast. Now, before moving data, we can reorder locally,
>>> inform the sender of the new ordering, and then move all the data.
>>>
>>
>> Great. This is exactly what i hate about this crap. It always seems
>> specially coded for the problem. I think we can use exactly the
>> primitive above to do your non-uniform broadcast step above. Mesh
>> topolgoy is jsut another function over the mesh points.
>>
>
> Yeah, there is still a question of what to store (e.g. FV and DG can throw
> away vertices and edges, lowest-order FE can throw away faces and edges),
> but that is orthogonal to communication which I agree is just a (usually
> variable-sized) function on the mesh points.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20111124/c2ba8bec/attachment.html>