[petsc-users] Issues with subsequent GMRES Solves

[Barry Smith]

   Is it the same matrix or a different matrix that is solved with the "second adjoint system"?  Or is it just a different right hand side?

   If it is a different system how do you know it will even converge for the new matrix? Maybe the new matrix is much more difficult?

   Is it possible the "second adjoint system" is singular?

   Given that this is a very strong preconditioner the convergence for the first adjoint system is pretty poor.

   Does this happen also for much smaller problems? Recommend running much smaller problem to see what happens convergence wise, then run that much smaller problem with -pc_type lu to see if the direct solver handles it happily.



On May 12, 2010, at 12:39 PM, C.A.(Sandy) Mader wrote:

> I am using PETSc to solve an adjoint system for a CFD code. 
> The issue I am having is that if I try to run successive cfd/adjoint
> solutions, the adjoint fails to converge after the first cfd/adjoint
> solution cycle. i.e.
> Solve CFD (ok)
> Solve adjoint system with multiple RHS's (ok)
> Solve CFD at new point (ok)
> Solve adjoint system at new point with multiple RHS's (GMRES iterations
> stall)
> 
> PETSc is not used to solve the CFD system, only the adjoint system.
> 
> I am currently using the following solver options:
> 'Adjoint solver type': 'GMRES',\
> 'adjoint relative tolerance':1e-6,\
> 'adjoint absolute tolerance':1e-16,\
> 'adjoint max iterations': 1500,\
> 'adjoint restart iteration' : 80,\
> 'adjoint monitor step': 10,\
> 'Preconditioner Side': 'right',\
> 'Matrix Ordering': 'NestedDissection',\
> 'Global Preconditioner Type': 'Additive Schwartz',\
> 'Local Preconditioner Type' : 'ILU',\
> 'ILU Fill Levels': 3,\
> 'ASM Overlap' : 5,\
> 
> Also, I am using the fortran interface to PETSc.
> 
> As a sample, I have included two convergence histories. Both are at the
> same converged CFD point(this case is ~1.2million cell so the PETSc
> system is approx 6million degrees of freedom). The first is an example
> where the adjoint system is solved the first time through the cycle. The
> second is an example of where the adjoint is solved the second time
> through the cycle:
> 1) as first point in cycle
> # ... KSP properties:
> #       type        : gmres     
> #       tolerances  : rel = 1.0E-10
> #                     abs = 1.0E-16
> #                     div = 1.0E+05
> #       max.iter.   : 1500
> #       precond.type: asm   
> Solving ADjoint Transpose with PETSc...
>   0 KSP Residual norm 0.1392E+00
>  10 KSP Residual norm 0.6394E-01
>  20 KSP Residual norm 0.6106E-01
>  30 KSP Residual norm 0.6019E-01
>  40 KSP Residual norm 0.5941E-01
>  50 KSP Residual norm 0.5876E-01
>  60 KSP Residual norm 0.5602E-01
>  70 KSP Residual norm 0.4915E-01
>  80 KSP Residual norm 0.3994E-01
>  90 KSP Residual norm 0.3892E-01
> 100 KSP Residual norm 0.3854E-01
> 110 KSP Residual norm 0.3794E-01
> 120 KSP Residual norm 0.3717E-01
> 130 KSP Residual norm 0.3630E-01
> 140 KSP Residual norm 0.3415E-01
> ...
> 900 KSP Residual norm 0.2437E-09
> 910 KSP Residual norm 0.1452E-09
> 920 KSP Residual norm 0.1025E-09
> 930 KSP Residual norm 0.6875E-10
> 940 KSP Residual norm 0.4141E-10
> 950 KSP Residual norm 0.2317E-10
> 960 KSP Residual norm 0.1559E-10
> Solving ADjoint Transpose with PETSc time (s) =   391.43
> Norm of error = 0.1366E-10    Iterations =  965
> ------------------------------------------------
> PETSc solver converged after   965 iterations.
> ------------------------------------------------ 
> 
> 2) As second point in cycle
> # ... KSP properties:
> #       type        : gmres     
> #       tolerances  : rel = 1.0E-10
> #                     abs = 1.0E-16
> #                     div = 1.0E+05
> #       max.iter.   : 1500
> #       precond.type: asm   
> Solving ADjoint Transpose with PETSc...
>   0 KSP Residual norm 0.1392E+00
>  10 KSP Residual norm 0.6400E-01
>  20 KSP Residual norm 0.6140E-01
>  30 KSP Residual norm 0.6060E-01
>  40 KSP Residual norm 0.5995E-01
>  50 KSP Residual norm 0.5974E-01
>  60 KSP Residual norm 0.5971E-01
>  70 KSP Residual norm 0.5957E-01
>  80 KSP Residual norm 0.5906E-01
>  90 KSP Residual norm 0.5906E-01
> 100 KSP Residual norm 0.5906E-01
> 110 KSP Residual norm 0.5903E-01
> 120 KSP Residual norm 0.5901E-01
> 130 KSP Residual norm 0.5901E-01
> 140 KSP Residual norm 0.5901E-01
> ...
> 1400 KSP Residual norm 0.5895E-01
> 1410 KSP Residual norm 0.5895E-01
> 1420 KSP Residual norm 0.5895E-01
> 1430 KSP Residual norm 0.5895E-01
> 1440 KSP Residual norm 0.5895E-01
> 1450 KSP Residual norm 0.5895E-01
> 1460 KSP Residual norm 0.5895E-01
> 1470 KSP Residual norm 0.5895E-01
> 1480 KSP Residual norm 0.5895E-01
> 1490 KSP Residual norm 0.5895E-01
> 1500 KSP Residual norm 0.5895E-01
> Solving ADjoint Transpose with PETSc time (s) =   516.59
> Norm of error = 0.5895E-01    Iterations = 1500
> ------------------------------------------------
> PETSc solver converged after  1500 iterations.
> ------------------------------------------------
> I have tried both resetting the operators between cycles and completely
> destroying the KSP object between cycles. Both give the same result. Is
> there a step I am missing to properly reset the system for the
> subsequent solves?
> 
> Thanks in advance for your input...
> 
> Sandy 
> -- 
> C.A.(Sandy) Mader
> Ph.D Candidate
> Multidisciplinary Design Optimization Laboratory
> University of Toronto Institute for Aerospace Studies
> mader at utias.utoronto.ca
>