AO scalability issue

[Aron Ahmadia]

Hi All,

I can take charge of the development of this.  Ulisses, we can take this
discussion offline.  The PETSc team only takes patches against their current
working version, but I can work against an earlier PETSc if that is
preferred.

Thanks,
Aron

On Wed, Aug 12, 2009 at 2:19 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Wed, Aug 12, 2009 at 1:01 PM, Lianjun An <alianjun at us.ibm.com> wrote:
>
>>  We currently use Petsc routine "AOCreateBasic" to create application
>> order --- a mapping from used id to petsc id. Then we setup a connection
>> graph for our mesh and assemble vector and matrix based on the Petsc order.
>> When total number of elements in our mesh is about couple million and 64
>> computer nodes are used, AO is created in a reasonable time. But when we
>> have 700 million elements in the mesh and use more then 256 computer nodes,
>> the time to create AO grows significantly (9000 secs).
>>
>> It seems that each process has whole permutation table about the mapping
>> and quite expensive in time and buffer size.
>> a) Memory requirement is quite high (AO creation failed for 1.3 billion
>> elements on 1024 nodes due to not enough memory).
>> b) If user id numbering on each processor is partially continuous, then
>> time for creation AO is reduced.
>> c) In my understanding, AOApplicationToPetsc and AOPetscToApplication
>> routines don't use communication since AO has whole mapping on each
>> processor.
>>
>> In our application, we might not need the whole mapping in each processor.
>> We only need to ids of vertices that reside in the processor (limited ghost
>> vertices). We wonder whether there is an efficient way to create AO and to
>> cut its creation time and buffer size. Let us know your thought on that.
>>
>
> The AO was not designed to be scalable (unfortunately). Any scalable
> version would segment the renumbering, so
> that each process was responsible for a range of entries. You would first
> communicate the indices to  correct process
> (using a VecScatter), renumber, and then communicate the result back to the
> original process. It does not sound all
> that hard to me (using contiguous indices), however we are unlikely to
> implement it right now. If you would like to
> try yourself, we would be willing to help.
>
>    Matt
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>



-- 
Aron Jamil Ahmadia
Assistant Research Scientist
King Abdullah University of Science and Technology
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20090812/537ad738/attachment.htm>