[petsc-users] Issues with subsequent GMRES Solves

Wed May 12 13:00:17 CDT 2010

See responses in text below:
On Wed, 2010-05-12 at 12:48 -0500, Barry Smith wrote:
> Is it the same matrix or a different matrix that is solved with the "second adjoint system"?  Or is it just a different right hand side?
It is a different matrix. In this case I solve the CFD at a given flow
condition, then solve the adjoint twice (two different RHS) both
converge. Then I solve the CFD case for a second flow condition,
followed by two more adjoint solutions. Neither of which solve. However,
if I solver directly for the second flow condition, the adojint system
will solve...
> 
>    If it is a different system how do you know it will even converge for the new matrix? Maybe the new matrix is much more difficult?
see above answer, if the same system will solve under one set of
conditions but not under another...
> 
>    Is it possible the "second adjoint system" is singular?
The second system will solve if I solve it on its own.

>    Given that this is a very strong preconditioner the convergence for the first adjoint system is pretty poor.
We use an approximate preconditioner to reduce the memory requirements
of the solver. I do not have this problem if I use the exact matrix as
the preconditioner but it limits my problem size....
> 
>    Does this happen also for much smaller problems? Recommend running much smaller problem to see what happens convergence wise, then run that much smaller problem with -pc_type lu to see if the direct solver handles it happily.
I have tried it on a smaller case and this problem does not occur...
> 
> 
> 
> On May 12, 2010, at 12:39 PM, C.A.(Sandy) Mader wrote:
> 
> > I am using PETSc to solve an adjoint system for a CFD code. 
> > The issue I am having is that if I try to run successive cfd/adjoint
> > solutions, the adjoint fails to converge after the first cfd/adjoint
> > solution cycle. i.e.
> > Solve CFD (ok)
> > Solve adjoint system with multiple RHS's (ok)
> > Solve CFD at new point (ok)
> > Solve adjoint system at new point with multiple RHS's (GMRES iterations
> > stall)
> > 
> > PETSc is not used to solve the CFD system, only the adjoint system.
> > 
> > I am currently using the following solver options:
> > 'Adjoint solver type': 'GMRES',\
> > 'adjoint relative tolerance':1e-6,\
> > 'adjoint absolute tolerance':1e-16,\
> > 'adjoint max iterations': 1500,\
> > 'adjoint restart iteration' : 80,\
> > 'adjoint monitor step': 10,\
> > 'Preconditioner Side': 'right',\
> > 'Matrix Ordering': 'NestedDissection',\
> > 'Global Preconditioner Type': 'Additive Schwartz',\
> > 'Local Preconditioner Type' : 'ILU',\
> > 'ILU Fill Levels': 3,\
> > 'ASM Overlap' : 5,\
> > 
> > Also, I am using the fortran interface to PETSc.
> > 
> > As a sample, I have included two convergence histories. Both are at the
> > same converged CFD point(this case is ~1.2million cell so the PETSc
> > system is approx 6million degrees of freedom). The first is an example
> > where the adjoint system is solved the first time through the cycle. The
> > second is an example of where the adjoint is solved the second time
> > through the cycle:
> > 1) as first point in cycle
> > # ... KSP properties:
> > #       type        : gmres     
> > #       tolerances  : rel = 1.0E-10
> > #                     abs = 1.0E-16
> > #                     div = 1.0E+05
> > #       max.iter.   : 1500
> > #       precond.type: asm   
> > Solving ADjoint Transpose with PETSc...
> >   0 KSP Residual norm 0.1392E+00
> >  10 KSP Residual norm 0.6394E-01
> >  20 KSP Residual norm 0.6106E-01
> >  30 KSP Residual norm 0.6019E-01
> >  40 KSP Residual norm 0.5941E-01
> >  50 KSP Residual norm 0.5876E-01
> >  60 KSP Residual norm 0.5602E-01
> >  70 KSP Residual norm 0.4915E-01
> >  80 KSP Residual norm 0.3994E-01
> >  90 KSP Residual norm 0.3892E-01
> > 100 KSP Residual norm 0.3854E-01
> > 110 KSP Residual norm 0.3794E-01
> > 120 KSP Residual norm 0.3717E-01
> > 130 KSP Residual norm 0.3630E-01
> > 140 KSP Residual norm 0.3415E-01
> > ...
> > 900 KSP Residual norm 0.2437E-09
> > 910 KSP Residual norm 0.1452E-09
> > 920 KSP Residual norm 0.1025E-09
> > 930 KSP Residual norm 0.6875E-10
> > 940 KSP Residual norm 0.4141E-10
> > 950 KSP Residual norm 0.2317E-10
> > 960 KSP Residual norm 0.1559E-10
> > Solving ADjoint Transpose with PETSc time (s) =   391.43
> > Norm of error = 0.1366E-10    Iterations =  965
> > ------------------------------------------------
> > PETSc solver converged after   965 iterations.
> > ------------------------------------------------ 
> > 
> > 2) As second point in cycle
> > # ... KSP properties:
> > #       type        : gmres     
> > #       tolerances  : rel = 1.0E-10
> > #                     abs = 1.0E-16
> > #                     div = 1.0E+05
> > #       max.iter.   : 1500
> > #       precond.type: asm   
> > Solving ADjoint Transpose with PETSc...
> >   0 KSP Residual norm 0.1392E+00
> >  10 KSP Residual norm 0.6400E-01
> >  20 KSP Residual norm 0.6140E-01
> >  30 KSP Residual norm 0.6060E-01
> >  40 KSP Residual norm 0.5995E-01
> >  50 KSP Residual norm 0.5974E-01
> >  60 KSP Residual norm 0.5971E-01
> >  70 KSP Residual norm 0.5957E-01
> >  80 KSP Residual norm 0.5906E-01
> >  90 KSP Residual norm 0.5906E-01
> > 100 KSP Residual norm 0.5906E-01
> > 110 KSP Residual norm 0.5903E-01
> > 120 KSP Residual norm 0.5901E-01
> > 130 KSP Residual norm 0.5901E-01
> > 140 KSP Residual norm 0.5901E-01
> > ...
> > 1400 KSP Residual norm 0.5895E-01
> > 1410 KSP Residual norm 0.5895E-01
> > 1420 KSP Residual norm 0.5895E-01
> > 1430 KSP Residual norm 0.5895E-01
> > 1440 KSP Residual norm 0.5895E-01
> > 1450 KSP Residual norm 0.5895E-01
> > 1460 KSP Residual norm 0.5895E-01
> > 1470 KSP Residual norm 0.5895E-01
> > 1480 KSP Residual norm 0.5895E-01
> > 1490 KSP Residual norm 0.5895E-01
> > 1500 KSP Residual norm 0.5895E-01
> > Solving ADjoint Transpose with PETSc time (s) =   516.59
> > Norm of error = 0.5895E-01    Iterations = 1500
> > ------------------------------------------------
> > PETSc solver converged after  1500 iterations.
> > ------------------------------------------------
> > I have tried both resetting the operators between cycles and completely
> > destroying the KSP object between cycles. Both give the same result. Is
> > there a step I am missing to properly reset the system for the
> > subsequent solves?
> > 
> > Thanks in advance for your input...
> > 
> > Sandy 
> > -- 
> > C.A.(Sandy) Mader
> > Ph.D Candidate
> > Multidisciplinary Design Optimization Laboratory
> > University of Toronto Institute for Aerospace Studies
> > mader at utias.utoronto.ca
> > 
> 
>