[petsc-dev] programming model for PETSc

[Dmitry Karpeev]

On Thu, Nov 24, 2011 at 6:35 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> On Nov 24, 2011, at 6:30 PM, Dmitry Karpeev wrote:
>
> > The MPICH paper anticipates O(1B) cores in an exascale machine.
> > The number of cores, however, needn't be the number of nodes (with one
> MPI process per node),
> > if each node has O(1000) cores that are programmed using threads or
> whatever.  If we can limit the number
> > of nodes to O(1M) (at the expense of "fattening" the nodes to whatever
> crazy core counts people envisage in
> > the future), then the current PetscLayout may be an acceptable price to
> pay.
> > Otherwise some sort of hierarchical PetscLayout or something entirely
> different is necessary.
> >
> > In any event, this business of storing explicit ranks/offsets for the
> ghost indices is sort of antithetical to the current
> > PetscLayout-based approach of indices having parallel semantics from the
> outset.  A lot of conceptual difficulties
> > in Sieve and MOAB (and the need for the extra objects such as Overlap
> and ParallelComm (sic!)) stem from a lack of
> > a global address space.  If we are planning to do away with PetscLayout
> in its current form, I think
> > the whole PETSc parallelization model requires rethinking.  That's
> definitely not an incremental change.
>
>    Everything is on the table. I could envision a model/implementation of
> PETSc that has no concept of global numbering and always works with
> rank,local indexing.
>
>   Being able to make a user friendly system with any model/implementation,
> of course, is crucial to whether the proposed system makes any sense.
>

I would start then by defining the API for PetscLayout (or Overlap, or
whatever equivalent object), which defines the local view of the global
address space.  That's the only way, it seems to me, to insulate the API
exposed to PETSc from these fundamental choices made underneath.

Dmitry.

>
>   Barry
>
> >
> > Dmitry.
> >
> > On Thu, Nov 24, 2011 at 6:17 PM, Dmitry Karpeev <karpeev at mcs.anl.gov>
> wrote:
> > So then are you planning to re-engineer PetscLayout?
> > Or is it acceptable to have a few PetscLayouts, in their current form,
> lying around?
> >
> > On Thu, Nov 24, 2011 at 6:14 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
> > On Thu, Nov 24, 2011 at 18:05, Dmitry Karpeev <karpeev at mcs.anl.gov>
> wrote:
> > Is O(commSize) storage considered unscalable in this model?
> >
> > I would like to avoid it where possible. If we have million-way
> parallelism at the network level (e.g. one million MPI processes), then we
> have to justify the memory usage. One 64-bit integer per process is 8MB. If
> that is needed once for the entire application, it is probably not an
> issue, but if it's needed by every object, then it could add up to a
> substantial amount (considering the low-memory estimates).
> >
> > The MPICH guys are thinking about this:
> >
> > http://www.mcs.anl.gov/~goodell/pdfs/mpi-mem-final.pdf
> >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20111124/a5dc61fe/attachment.html>