[petsc-users] AOCreateBasic is not scaling

[Matthew Knepley]

On Wed, Mar 2, 2011 at 4:52 PM, M. Scot Breitenfeld <brtnfld at uiuc.edu>wrote:

> I don't number my global degree's of freedom from low-high continuously
> per processor as PETSc uses for ordering, but I use the natural ordering
> of the application, I then use AOcreateBasic to obtain the mapping
> between the PETSc and my ordering.
>

I would suggest using the LocalToGlobalMapping functions, which are
scalable.
AO is designed for complete global permutations.

  Thanks,

     Matt


>  CALL VecGetOwnershipRange(b, low, high, ierr)
>
>  icnt = 0
>
>  DO mi = 1, mctr ! these are the nodes local to processor
>        mi_global = myglobal(mi)
>
>        irowx = 3*mi_global-2
>        irowy = 3*mi_global-1
>        irowz = 3*mi_global
>
>        mappings(icnt+1:icnt+3) = (/            &
>             nrow_global(row_from_dof(1,mi))-1, &
>             nrow_global(row_from_dof(2,mi))-1, &
>             nrow_global(row_from_dof(3,mi))-1  &
>             /)
>
>        petscOrdering(icnt+1:icnt+3) = (/low+icnt,low+icnt+1,low+icnt+2/)
>
>        icnt = icnt + 3
>  END DO
>
>  CALL AOCreateBasic(PETSC_COMM_WORLD, icnt, mappings, petscOrdering,
> toao, ierr)
>
>  DO mi = mctr+1, myn ! these are the ghost nodes not on this processor
>
>     mi_global = myglobal(mi)
>
>     mappings(icnt+1:icnt+3) = (/            &
>          nrow_global(row_from_dof(1,mi))-1, &
>          nrow_global(row_from_dof(2,mi))-1, &
>          nrow_global(row_from_dof(3,mi))-1  &
>          /)
>
>     icnt = icnt + 3
>  ENDDO
>  CALL AOApplicationToPetsc(toao, 3*myn, mappings, ierr)
>
>  CALL AODestroy(toao, ierr)
>
> I then use  mapping to input the values into the correct row as wanted
> by PETSc
>
>
> On 03/02/2011 04:29 PM, Matthew Knepley wrote:
> > On Wed, Mar 2, 2011 at 4:25 PM, M. Scot Breitenfeld <brtnfld at uiuc.edu
> > <mailto:brtnfld at uiuc.edu>> wrote:
> >
> >     Hi,
> >
> >     First, thanks for the suggestion on using MPISBAIJ for my A matrix,
> it
> >     seems to have cut down on my memory and assembly time. For a 1.5
> >     million
> >     dof problem:
> >
> >        # procs:                  2        4        8           16
> >     ----------------------------------------------------------------
> >     Assembly (sec):        245     124      63       86
> >     Solver (sec):              924     578     326     680
> >
> >     Memory (GB):          2.5        1.4     .877    .565
> >
> >     The problem I have is the amount of time it's taking in
> AOCreateBasic,
> >     it takes longer then assembling,
> >
> >        # procs:                            2        4        8         16
> >     ---------------------------------------------------------------------
> >     AOCreateBasic (sec):         .6      347      170     197
> >
> >     Is there something that I can change or something I can look for that
> >     might be causing this increase in time as I go from 2 to 4 processors
> >     (at least it scales from 4 to 8 processors). I read in the archive
> >     that
> >     AOCreateBasic is not meant to be scalable so maybe there is nothing I
> >     can do.
> >
> >
> > Yes, this is non-scalable. What are you using it for?
> >
> >    Matt
> >
> >
> >     Thanks,
> >     Scot
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> > experiments is infinitely more interesting than any results to which
> > their experiments lead.
> > -- Norbert Wiener
>
>


-- 
What most experimenters take for granted before they begin their experiments
is infinitely more interesting than any results to which their experiments
lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20110303/0246deae/attachment.htm>