[petsc-users] MatSetValues is expensive

[M. Scot Breitenfeld]

On 02/24/2011 10:20 AM, Jed Brown wrote:
> On Thu, Feb 24, 2011 at 16:33, M. Scot Breitenfeld <brtnfld at uiuc.edu
> <mailto:brtnfld at uiuc.edu>> wrote:
>
>     I'm working on a particle type method and I'm using MatSetValues, to
>     insert values (add_values) into my matrix. Currently I:
>
>     do  i, loop over number of particles
>         do j, loop over particles in i's family
>
>
> How big is a typical family?
300- 900  particles


>  
>
>             ...
>              in row i's dof; insert values in columns of j's (x,y,z) dofs
>     (3 calls to MatSetValues for i's x,y,z dof)
>
>
> Why make three calls here instead of one?
I split the x-y-z row entries up depending on if the dof is prescribed,
I'll combine them and see if that helps.


>  
>
>              in row j's dof; insert values in columns of i's (x,y,z) dofs
>     (3 calls to MatSetValues for j's x,y,z dof)
>
>
> Again, why call these separately? Also, is the matrix symmetric?
I split them depending on if the dof of particle j is prescribed. The
matrix is symmetric and the percentage of non-zeros in the matrix has a
range of 3%-7% depending on the number of particles in the family.

>  
>
>             ...
>         enddo
>     enddo
>
>     Running serially, using MatSetValues it takes 294.8 sec. to
>     assemble the
>     matrix,
>
>
> Are you sure that it was preallocated correctly? Is the cost to
> compute the entries essentially zero?
Assuming you preallocate by:

  CALL MatCreateMPIAIJ(PETSC_COMM_WORLD, &
       3*mctr, 3*mctr, &
       total_global_nodes*3, total_global_nodes*3, &
       0, d_nnz, 0, o_nnz, A, ierr)

and I tried, as suggested,

  CALL MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_TRUE, ierr)

and it does not report any problems.

I would not say it's zero to compute the entries (I guess it takes about
3.5ms per particle for the calculations). This is a fairly small case,
only 8000 particles.
>  
>
>
>
> FEM has a simple quadrature loop that builds a dense element matrix,
> MatSetValues() is called once per element.
That is what I meant, once per element.

I do have a simpler formulation that allows me to enter an entire row
all at once per particle:

do  i, loop over number of particles
    do j, loop over particles in i's family
       ... fill the rows of i's dofs (x,y,z)
    enddo
call MatSetValues...
enddo

For the same case, it takes 6 seconds to assemble. If I remove the
MatSetValues call it takes 0.66 seconds (for this case the calculations
are REALLY simple), the family is also always smaller then the previous
method.