[petsc-dev] Question About Petsc ILU
Dave Nystrom
Dave.Nystrom at tachyonlogic.com
Thu Dec 29 19:30:17 CST 2011
Mark F. Adams writes:
>
> On Dec 29, 2011, at 12:45 PM, Dave Nystrom wrote:
>
> > Jed Brown writes:
> >> On Thu, Dec 29, 2011 at 11:01, Dave Nystrom <dnystrom1 at comcast.net> wrote:
> >>
> >>> I have recently added the capability to have a separate preconditioning
> >>> matrix in the petsc interface for the code I am working with. I have two
> >>> types of preconditioning matrices that I have questions about. One is
> >>> tridiagonal and the other is 7 diagonals. In both cases, the the diagonals
> >>> are all lexically adjacent. Or phrased differently, the tridiagonal matrix
> >>> has a bandwidth of 3 and the 7 diagonal matrix has a bandwidth of 7 and so
> >>> they are compact or dense band systems.
> >>>
> >>> I was wondering what petsc ilu will do for preconditioning matrices like
> >>> these. Will it produce an exact lu factorization or a nearly exact
> >>> factorization?
> >>
> >> Yes
> >>
> >>> I'm interested in the answer to this question because I am thinking I
> >>> might be able to run this preconditioner on the gpu using the txpetscgpu
> >>> package.
> >>
> >> Likely pointless because this solve is probably not a big part of run-time.
> >> The bigger concern is the convergence rate that you lose by using this
> >> approximation. Matt and I mentioned this the last time you brought it up,
> >> but I recommend getting familiar with the literature. Then, get the math
> >> straight before looking for ways to transform into problems that you can
> >> run fast on a GPU or whatever. If you just optimize kernels, you're likely
> >> to optimize something that takes a small part of run-time and isn't really
> >> helping you anyway.
> >
> > So the application code is one that I am not the primary author on and it is
> > a 2d resistive MHD with extensions code for magnetized fusion problems.
> > Magnetized fusion problems tend to be highly anisotropic with qualitatively
> > different physics parallel and perpendicular to the magnetic field. The code
> > has 3 approaches to solving the linear systems. One is a suite of native cg
> > solvers that use a cholesky solver for the inner band of the matrix as the
> > preconditioner. As noted, this preconditioner ignores the coupling to the
> > second coordinate. However, this preconditioner produces the lowest
> > iteration count of the various preconditioner approaches that I have tried.
> >
>
> This sounds like it is and SPD matrix. I would think GAMG would be OK.
> If you can set -pc_gamg_verbose -ksp_monitor' and send me the output I can
> see of anything jumps out.
Sure. I'll try this as soon as I can.
> > The second solver approach is use of the agmg package which works well on all
> > of the linear solves except for the Hall matrix. For the Hall matrix, agmg
> > fails to produce a solution.
> >
>
> Are you using CG and Cholesky for the Hall matrix? If so it is probably a
> div(alpha(x)grad(u)).
Generally I use CG and LU from petsc. Cholesky runs slower than LU and I was
told in a previous email that it was because of the extra data movement for
Cholesky versus LU. I have also tried minres instead of CG and tend to get
reduced iteration counts. But I think minres may have an extra dot product
and so is a little more expensive than cg. But minres is usually cheaper
over all because of the reduction in iterations.
> Mark
>
> > The third approach is an interface to petsc that I have added in the last few
> > months. So far, I have been able to get really good results with petsc on
> > all of the linear systems except for the Hall matrix. In general, the
> > fastest solution on these other linear systems has been petsc using jacobi
> > preconditioning running on the gpu. However, I think I may be able to do
> > even better on these other systems using petsc with the separate
> > preconditioning matrix and using ilu plus txpetscgpu.
> >
> > But, having invested the time to interface to petsc, I am now trying to
> > explore the vast set of options available to me to see what works best for my
> > problems from the petsc toolbox. But solving the Hall matrix, especially for
> > larger mesh sizes, still remains problematic. The lead author of this code
> > is investigating the Hall matrix formulation to see if there are better ways
> > to either formulate it or solve it and I am exploring what petsc has to offer
> > for solving it. The matrix is very ill conditioned. The formulation is
> > complicated enough that perhaps there is just a bug somewhere. But these
> > sort of questions are being investigated by the main author of the code.
> >
> > Anyway, I have so far gotten good performance gains using the gpu with petsc
> > but am totally open to and am trying to explore the whole petsc toolbox.
> >
>
More information about the petsc-dev
mailing list