AO scalability issue

[Lianjun An]

We currently use Petsc routine "AOCreateBasic" to create application order
--- a mapping from used id to petsc id. Then we setup a connection graph
for our mesh and assemble vector and matrix based on the Petsc order. When
total number of elements in our mesh is about couple million and 64
computer nodes are used, AO is created in a reasonable time. But when we
have 700 million elements in the mesh and use more then 256 computer nodes,
the time to create AO grows significantly (9000 secs).

It seems that each process has whole permutation table about the mapping
and quite expensive in time and buffer size.
a) Memory requirement is quite high (AO creation failed for 1.3 billion
elements on 1024 nodes due to not enough memory).
b) If user id numbering on each processor is partially continuous, then
time for creation AO is reduced.
c) In my understanding, AOApplicationToPetsc and AOPetscToApplication
routines don't use communication since AO has whole mapping on each
processor.

In our application, we might not need the whole mapping in each processor.
We only need to ids of vertices that reside in the processor (limited ghost
vertices). We wonder whether there is an efficient way to create AO and to
cut its creation time and buffer size. Let us know your thought on that.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20090812/88d8424f/attachment.htm>