[petsc-users] Reducing cost of MatSetValues

[Jed Brown]

On Wed, Jun 1, 2011 at 22:43, John Fettig <john.fettig at gmail.com> wrote:

> Unless I'm reading it wrong, there are 89M insertions.  This is from
> 10 timesteps with nonlinear iteration on each timestep,
>

Needing 27 linear solves per time step would normally be a better place to
direct your effort, but strangely, very little of the total run time is
currently in the solve.



> with ~1.5M
> elements.
>

Is this 1.5M finite elements or dofs or nonzeros in the matrix? Doing more
than 1000 SOR sweeps per second would be weird if the matrix is big.


>
> We had thought about constructing the matrix outside of PETSc and then
> passing PETSc pointers to the matrix, but maybe matrix-free would be
> better for some of the equations where jacobi preconditioning is
> sufficient.
>

Have you considered solving the coupled system instead of a bunch of
separate systems. Solving the coupled thing allows you to use block matrix
formats and gets better data locality. It also reduces the number of
synchronization points (parallel reductions and scatters). And it should
converge faster on the coupled system. 27 separate solves per time step is
really high, and you currently have over 800 reductions per time step. That
is going to hurt a lot in parallel, especially for strong scaling.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20110601/1dfbc0f8/attachment.htm>