[petsc-users] solving multiple linear systems with same matrix (sequentially, not simultaneously)

[Daniel Goldberg]

Hi Barry

Thank you for the reply. From a test I did with a linear system of about
400 000 on 128 cores, i clocked that the setup takes about 2% of the time
of the solve -- but it will be a larger fraction with smaller systems.

If you mean the differential operator -- it is elliptic. The equations are
based on those of a 3D shear-thinning stokes fluid, but simplified by
various approximations including small aspect ratio, so the equations are
essentially two-dimensional. (sometimes called the Shallow Shelf
Approximation and similar to the equations solved by
http://onlinelibrary.wiley.com/doi/10.1029/2008JF001179/abstract). The
linear system I solve is a step in an iterative solution to the nonlinear
equations.

I will try gamg as I know it can reduce the number of CG iterations
required. (I'm guessing you mean algebraic, not geometric?)

Am I correct in thinking that, for subsequent solves with the same KSP
object, the time of the solve should scale with the number of conj grad
iterations? Will this time be relatively independent of the preconditioner
type?

Thanks
Dan

On Sun, Feb 22, 2015 at 5:54 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>    You should definitely keep the matrix, PC, and KSP for all 40 of the
> solves. This will eliminate the matrix and preconditioner setup time for
> last 39 solves.
>
>     What kind of operator do you have? ILU(k) is a general purpose
> preconditioner but not particularly good. I would recommend trying -pc_type
> gamg next.
>
>   Barry
>
>
> > On Feb 22, 2015, at 11:00 AM, Daniel Goldberg <dngoldberg at gmail.com>
> wrote:
> >
> > Hello
> >
> > In the code I am developing, there is a point where I will need to solve
> a linear system with the same matrix but different right hand sides. It is
> a time-dependent model and the matrix in question will change each
> timestep, but for a single timestep ~30-40 linear solves will be required,
> with the same matrix but differing right hand sides. Each right hand side
> depends on the previous, so they cannot all be solved simultaneously.
> >
> > Generally I solve the matrix via CG with  Block Jacobi / ILU(0)
> preconditioning. I don't persist the matrix in between solves (i.e. I
> destroy the Mat, KSP and Vec objects after each linear solve) because I do
> not get the feeling this makes a large difference to performance and it is
> easier in dealing with the rest of the model code, which does not use
> PETSc. The matrix is positive definite with sparsity of 18 nonzero elements
> per row, and generally the linear system is smaller than 1 million degrees
> of freedom.
> >
> > If my matrix was dense, then likely retaining LU factors for
> foward/backward substitution would be a good strategy (though I'm not sure
> if this is easily doable using PETSc direct solvers). But given the
> sparsity I am unsure of whether I can take advantage of using the same
> matrix 30-40 times in a row.
> >
> > The comments in ex16.c state that when a KSP object is used multiple
> times for a linear solve, the preconditioner operations are not done each
> time -- and so I figured that if I changed my preconditioning parameters I
> might be able to make subsequent solves with the same KSP object faster. I
> tried an experment in which I increased pc_factor_levels, and then created
> a number of random vectors, e.g. vec_1, vec_2, vec_3... and called
> >
> > KSPSolve(ksp, vec_i, solution_i)
> >
> > sequentially with the same ksp object. I did see a decrease in the
> number of CG iterations required as pc_factor_levels was increased, as well
> as an increase in time for the first linear solve, but no decrease in time
> required for subsequent solves. *Should* there be a timing decrease? But
> more generally is there a way to optimize the fact I'm solving 40 linear
> systems with the same matrix?
> >
> > Apologies for not providing a copy of the relevant bits of code -- this
> is my first email to the group, and a rather long one already -- happy to
> be more concrete about anything I've said above.
> >
> > Thanks
> > Dan
> >
> > --
> >
> > Daniel Goldberg, PhD
> > Lecturer in Glaciology
> > School of Geosciences, University of Edinburgh
> > Geography Building, Drummond Street, Edinburgh EH8 9XP
> >
> >
> > em: Dan.Goldberg at ed.ac.uk
> > web: http://ocean.mit.edu/~dgoldberg
>
>


-- 

Daniel Goldberg, PhD
Lecturer in Glaciology
School of Geosciences, University of Edinburgh
Geography Building, Drummond Street, Edinburgh EH8 9XP


em: D <dgoldber at mit.edu>an.Goldberg at ed.ac.uk
web: http://ocean.mit.edu/~dgoldberg
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150222/f4787f47/attachment.html>