[petsc-users] Slow MatAssemblyBegin MAT_FINAL_ASSEMBLY

[Jose A. Abell M.]

Hello dear PETSc users,

This is a problem that pops up often, from what I see, in the mailing list.
My program takes a long time assembling the matrix.

What I know:


   - Matrix Size is (MatMPIAIJ) 2670402
   - Number of processes running PETSc: 95
   - Not going to virtual memory (no swapping, used mem well withing each
   node's capacity)
   - System is partitioned with ParMETIS for load balancing
   - I see memory moving around in each node (total used memory changes a
   bit, grows and then frees)
   - Matrix is filled in blocks of size 81x81 (FEM code, so this ends up
   being a sparse matrix)
   - I don't do flushes at all. Only MAT_FINAL_ASSEMBLY when all the
   MatSetValues are done.

Should I do MAT_FLUSH_ASSEMBLY even though I have enough memory to store
the buffers? If so, how often? Every 100 blocks?

What else could it be?

Its taking several hours to asseble this matrix. I re-use the sparsity
pattern, so subsequent assemblies are fast. Does this mean that my
preallocation is wrong?

Regards,




--

José Abell
*PhD Candidate*
Computational Geomechanics Group
Dept. of Civil and Environmental Engineering
UC Davis
www.joseabell.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151216/dd55f08b/attachment.html>