[petsc-users] AOCreateBasic is not scaling

[M. Scot Breitenfeld]

I don't number my global degree's of freedom from low-high continuously
per processor as PETSc uses for ordering, but I use the natural ordering
of the application, I then use AOcreateBasic to obtain the mapping
between the PETSc and my ordering.

 CALL VecGetOwnershipRange(b, low, high, ierr)

  icnt = 0

  DO mi = 1, mctr ! these are the nodes local to processor
        mi_global = myglobal(mi)

        irowx = 3*mi_global-2
        irowy = 3*mi_global-1
        irowz = 3*mi_global
       
        mappings(icnt+1:icnt+3) = (/            &
             nrow_global(row_from_dof(1,mi))-1, &
             nrow_global(row_from_dof(2,mi))-1, &
             nrow_global(row_from_dof(3,mi))-1  &
             /)

        petscOrdering(icnt+1:icnt+3) = (/low+icnt,low+icnt+1,low+icnt+2/)

        icnt = icnt + 3
 END DO

  CALL AOCreateBasic(PETSC_COMM_WORLD, icnt, mappings, petscOrdering,
toao, ierr)

  DO mi = mctr+1, myn ! these are the ghost nodes not on this processor

     mi_global = myglobal(mi)

     mappings(icnt+1:icnt+3) = (/            &
          nrow_global(row_from_dof(1,mi))-1, &
          nrow_global(row_from_dof(2,mi))-1, &
          nrow_global(row_from_dof(3,mi))-1  &
          /)

     icnt = icnt + 3
  ENDDO
  CALL AOApplicationToPetsc(toao, 3*myn, mappings, ierr)

  CALL AODestroy(toao, ierr)

I then use  mapping to input the values into the correct row as wanted
by PETSc


On 03/02/2011 04:29 PM, Matthew Knepley wrote:
> On Wed, Mar 2, 2011 at 4:25 PM, M. Scot Breitenfeld <brtnfld at uiuc.edu
> <mailto:brtnfld at uiuc.edu>> wrote:
>
>     Hi,
>
>     First, thanks for the suggestion on using MPISBAIJ for my A matrix, it
>     seems to have cut down on my memory and assembly time. For a 1.5
>     million
>     dof problem:
>
>        # procs:                  2        4        8           16
>     ----------------------------------------------------------------
>     Assembly (sec):        245     124      63       86
>     Solver (sec):              924     578     326     680
>
>     Memory (GB):          2.5        1.4     .877    .565
>
>     The problem I have is the amount of time it's taking in AOCreateBasic,
>     it takes longer then assembling,
>
>        # procs:                            2        4        8         16
>     ---------------------------------------------------------------------
>     AOCreateBasic (sec):         .6      347      170     197
>
>     Is there something that I can change or something I can look for that
>     might be causing this increase in time as I go from 2 to 4 processors
>     (at least it scales from 4 to 8 processors). I read in the archive
>     that
>     AOCreateBasic is not meant to be scalable so maybe there is nothing I
>     can do.
>
>
> Yes, this is non-scalable. What are you using it for?
>
>    Matt
>  
>
>     Thanks,
>     Scot
>
>
>
>
>
>
>
>
>
> -- 
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which
> their experiments lead.
> -- Norbert Wiener