AO scalability issue

[Matthew Knepley]

On Wed, Aug 12, 2009 at 1:01 PM, Lianjun An <alianjun at us.ibm.com> wrote:

> We currently use Petsc routine "AOCreateBasic" to create application order
> --- a mapping from used id to petsc id. Then we setup a connection graph for
> our mesh and assemble vector and matrix based on the Petsc order. When total
> number of elements in our mesh is about couple million and 64 computer nodes
> are used, AO is created in a reasonable time. But when we have 700 million
> elements in the mesh and use more then 256 computer nodes, the time to
> create AO grows significantly (9000 secs).
>
> It seems that each process has whole permutation table about the mapping
> and quite expensive in time and buffer size.
> a) Memory requirement is quite high (AO creation failed for 1.3 billion
> elements on 1024 nodes due to not enough memory).
> b) If user id numbering on each processor is partially continuous, then
> time for creation AO is reduced.
> c) In my understanding, AOApplicationToPetsc and AOPetscToApplication
> routines don't use communication since AO has whole mapping on each
> processor.
>
> In our application, we might not need the whole mapping in each processor.
> We only need to ids of vertices that reside in the processor (limited ghost
> vertices). We wonder whether there is an efficient way to create AO and to
> cut its creation time and buffer size. Let us know your thought on that.
>

The AO was not designed to be scalable (unfortunately). Any scalable version
would segment the renumbering, so
that each process was responsible for a range of entries. You would first
communicate the indices to  correct process
(using a VecScatter), renumber, and then communicate the result back to the
original process. It does not sound all
that hard to me (using contiguous indices), however we are unlikely to
implement it right now. If you would like to
try yourself, we would be willing to help.

   Matt
-- 
What most experimenters take for granted before they begin their experiments
is infinitely more interesting than any results to which their experiments
lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20090812/45c066e9/attachment.htm>