[petsc-users] MatSetValues is expensive

[Jed Brown]

On Thu, Feb 24, 2011 at 16:33, M. Scot Breitenfeld <brtnfld at uiuc.edu> wrote:

> I'm working on a particle type method and I'm using MatSetValues, to
> insert values (add_values) into my matrix. Currently I:
>
> do  i, loop over number of particles
>     do j, loop over particles in i's family
>

How big is a typical family?


>         ...
>          in row i's dof; insert values in columns of j's (x,y,z) dofs
> (3 calls to MatSetValues for i's x,y,z dof)
>

Why make three calls here instead of one?


>          in row j's dof; insert values in columns of i's (x,y,z) dofs
> (3 calls to MatSetValues for j's x,y,z dof)
>

Again, why call these separately? Also, is the matrix symmetric?


>         ...
>     enddo
> enddo
>
> Running serially, using MatSetValues it takes 294.8 sec. to assemble the
> matrix,
>

Are you sure that it was preallocated correctly? Is the cost to compute the
entries essentially zero?


> if I remove the calls to MatSetValues it takes 29.5 sec. to run
> through the same loops, so the MatSetValues calls take up 90% of the
> assembling time. I'm preallocating the A matrix specifying d_nnz and o_nnz.
>
> I guess I need to add extra storage so I can call the MatSetValues with
> more values so that I can call it less, or just do a lot of
> recalculating of values so that I can add an entire row at once. I just
> want to make sure this is expected behavior and not something that I'm
> doing wrong before I start to rewrite my assembling routine. Probably a
> hash table would be better but I don't want to store that and then
> convert that to a CRS matrix, I'm already running into memory issues as
> it is.
>
> Just out of curiosity, wouldn't a finite element code have a similar
> situation, in that case you would form the local stiffness matrix and
> then insert that into the global stiffness matrix, so you would be
> calling MatSetValues "number of elements" times.
>

FEM has a simple quadrature loop that builds a dense element matrix,
MatSetValues() is called once per element.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20110224/fa3d0f36/attachment.htm>