[petsc-dev] Question About Petsc ILU

[Dave Nystrom]

Mark F. Adams writes:
 > 
 > On Dec 29, 2011, at 12:45 PM, Dave Nystrom wrote:
 > 
 > > Jed Brown writes:
 > >> On Thu, Dec 29, 2011 at 11:01, Dave Nystrom <dnystrom1 at comcast.net> wrote:
 > >> 
 > >>> I have recently added the capability to have a separate preconditioning
 > >>> matrix in the petsc interface for the code I am working with.  I have two
 > >>> types of preconditioning matrices that I have questions about.  One is
 > >>> tridiagonal and the other is 7 diagonals.  In both cases, the the diagonals
 > >>> are all lexically adjacent.  Or phrased differently, the tridiagonal matrix
 > >>> has a bandwidth of 3 and the 7 diagonal matrix has a bandwidth of 7 and so
 > >>> they are compact or dense band systems.
 > >>> 
 > >>> I was wondering what petsc ilu will do for preconditioning matrices like
 > >>> these.  Will it produce an exact lu factorization or a nearly exact
 > >>> factorization?
 > >> 
 > >> Yes
 > >> 
 > >>> I'm interested in the answer to this question because I am thinking I
 > >>> might be able to run this preconditioner on the gpu using the txpetscgpu
 > >>> package.
 > >> 
 > >> Likely pointless because this solve is probably not a big part of run-time.
 > >> The bigger concern is the convergence rate that you lose by using this
 > >> approximation. Matt and I mentioned this the last time you brought it up,
 > >> but I recommend getting familiar with the literature. Then, get the math
 > >> straight before looking for ways to transform into problems that you can
 > >> run fast on a GPU or whatever. If you just optimize kernels, you're likely
 > >> to optimize something that takes a small part of run-time and isn't really
 > >> helping you anyway.
 > > 
 > > So the application code is one that I am not the primary author on and it is
 > > a 2d resistive MHD with extensions code for magnetized fusion problems.
 > > Magnetized fusion problems tend to be highly anisotropic with qualitatively
 > > different physics parallel and perpendicular to the magnetic field.  The code
 > > has 3 approaches to solving the linear systems.  One is a suite of native cg
 > > solvers that use a cholesky solver for the inner band of the matrix as the
 > > preconditioner.  As noted, this preconditioner ignores the coupling to the
 > > second coordinate.  However, this preconditioner produces the lowest
 > > iteration count of the various preconditioner approaches that I have tried.
 > > 
 > 
 > This sounds like it is and SPD matrix.  I would think GAMG would be OK.
 > If you can set -pc_gamg_verbose -ksp_monitor' and send me the output I can
 > see of anything jumps out.

Sure.  I'll try this as soon as I can.

 > > The second solver approach is use of the agmg package which works well on all
 > > of the linear solves except for the Hall matrix.  For the Hall matrix, agmg
 > > fails to produce a solution.
 > > 
 > 
 > Are you using CG and Cholesky for the Hall matrix?  If so it is probably a
 > div(alpha(x)grad(u)).

Generally I use CG and LU from petsc.  Cholesky runs slower than LU and I was
told in a previous email that it was because of the extra data movement for
Cholesky versus LU.  I have also tried minres instead of CG and tend to get
reduced iteration counts.  But I think minres may have an extra dot product
and so is a little more expensive than cg.  But minres is usually cheaper
over all because of the reduction in iterations.

 > Mark
 > 
 > > The third approach is an interface to petsc that I have added in the last few
 > > months.  So far, I have been able to get really good results with petsc on
 > > all of the linear systems except for the Hall matrix.  In general, the
 > > fastest solution on these other linear systems has been petsc using jacobi
 > > preconditioning running on the gpu.  However, I think I may be able to do
 > > even better on these other systems using petsc with the separate
 > > preconditioning matrix and using ilu plus txpetscgpu.
 > > 
 > > But, having invested the time to interface to petsc, I am now trying to
 > > explore the vast set of options available to me to see what works best for my
 > > problems from the petsc toolbox.  But solving the Hall matrix, especially for
 > > larger mesh sizes, still remains problematic.  The lead author of this code
 > > is investigating the Hall matrix formulation to see if there are better ways
 > > to either formulate it or solve it and I am exploring what petsc has to offer
 > > for solving it.  The matrix is very ill conditioned.  The formulation is
 > > complicated enough that perhaps there is just a bug somewhere.  But these
 > > sort of questions are being investigated by the main author of the code.
 > > 
 > > Anyway, I have so far gotten good performance gains using the gpu with petsc
 > > but am totally open to and am trying to explore the whole petsc toolbox.
 > > 
 >