[petsc-users] Issues with subsequent GMRES Solves

[C.A.(Sandy) Mader]

I am using PETSc to solve an adjoint system for a CFD code. 
The issue I am having is that if I try to run successive cfd/adjoint
solutions, the adjoint fails to converge after the first cfd/adjoint
solution cycle. i.e.
Solve CFD (ok)
Solve adjoint system with multiple RHS's (ok)
Solve CFD at new point (ok)
Solve adjoint system at new point with multiple RHS's (GMRES iterations
stall)

PETSc is not used to solve the CFD system, only the adjoint system.

I am currently using the following solver options:
'Adjoint solver type': 'GMRES',\
'adjoint relative tolerance':1e-6,\
'adjoint absolute tolerance':1e-16,\
'adjoint max iterations': 1500,\
'adjoint restart iteration' : 80,\
'adjoint monitor step': 10,\
'Preconditioner Side': 'right',\
'Matrix Ordering': 'NestedDissection',\
'Global Preconditioner Type': 'Additive Schwartz',\
'Local Preconditioner Type' : 'ILU',\
'ILU Fill Levels': 3,\
'ASM Overlap' : 5,\

Also, I am using the fortran interface to PETSc.

As a sample, I have included two convergence histories. Both are at the
same converged CFD point(this case is ~1.2million cell so the PETSc
system is approx 6million degrees of freedom). The first is an example
where the adjoint system is solved the first time through the cycle. The
second is an example of where the adjoint is solved the second time
through the cycle:
1) as first point in cycle
# ... KSP properties:
#       type        : gmres     
#       tolerances  : rel = 1.0E-10
#                     abs = 1.0E-16
#                     div = 1.0E+05
#       max.iter.   : 1500
#       precond.type: asm   
 Solving ADjoint Transpose with PETSc...
   0 KSP Residual norm 0.1392E+00
  10 KSP Residual norm 0.6394E-01
  20 KSP Residual norm 0.6106E-01
  30 KSP Residual norm 0.6019E-01
  40 KSP Residual norm 0.5941E-01
  50 KSP Residual norm 0.5876E-01
  60 KSP Residual norm 0.5602E-01
  70 KSP Residual norm 0.4915E-01
  80 KSP Residual norm 0.3994E-01
  90 KSP Residual norm 0.3892E-01
 100 KSP Residual norm 0.3854E-01
 110 KSP Residual norm 0.3794E-01
 120 KSP Residual norm 0.3717E-01
 130 KSP Residual norm 0.3630E-01
 140 KSP Residual norm 0.3415E-01
 ...
 900 KSP Residual norm 0.2437E-09
 910 KSP Residual norm 0.1452E-09
 920 KSP Residual norm 0.1025E-09
 930 KSP Residual norm 0.6875E-10
 940 KSP Residual norm 0.4141E-10
 950 KSP Residual norm 0.2317E-10
 960 KSP Residual norm 0.1559E-10
Solving ADjoint Transpose with PETSc time (s) =   391.43
 Norm of error = 0.1366E-10    Iterations =  965
 ------------------------------------------------
 PETSc solver converged after   965 iterations.
 ------------------------------------------------ 

2) As second point in cycle
# ... KSP properties:
#       type        : gmres     
#       tolerances  : rel = 1.0E-10
#                     abs = 1.0E-16
#                     div = 1.0E+05
#       max.iter.   : 1500
#       precond.type: asm   
Solving ADjoint Transpose with PETSc...
   0 KSP Residual norm 0.1392E+00
  10 KSP Residual norm 0.6400E-01
  20 KSP Residual norm 0.6140E-01
  30 KSP Residual norm 0.6060E-01
  40 KSP Residual norm 0.5995E-01
  50 KSP Residual norm 0.5974E-01
  60 KSP Residual norm 0.5971E-01
  70 KSP Residual norm 0.5957E-01
  80 KSP Residual norm 0.5906E-01
  90 KSP Residual norm 0.5906E-01
 100 KSP Residual norm 0.5906E-01
 110 KSP Residual norm 0.5903E-01
 120 KSP Residual norm 0.5901E-01
 130 KSP Residual norm 0.5901E-01
 140 KSP Residual norm 0.5901E-01
 ...
1400 KSP Residual norm 0.5895E-01
1410 KSP Residual norm 0.5895E-01
1420 KSP Residual norm 0.5895E-01
1430 KSP Residual norm 0.5895E-01
1440 KSP Residual norm 0.5895E-01
1450 KSP Residual norm 0.5895E-01
1460 KSP Residual norm 0.5895E-01
1470 KSP Residual norm 0.5895E-01
1480 KSP Residual norm 0.5895E-01
1490 KSP Residual norm 0.5895E-01
1500 KSP Residual norm 0.5895E-01
Solving ADjoint Transpose with PETSc time (s) =   516.59
 Norm of error = 0.5895E-01    Iterations = 1500
 ------------------------------------------------
 PETSc solver converged after  1500 iterations.
 ------------------------------------------------
I have tried both resetting the operators between cycles and completely
destroying the KSP object between cycles. Both give the same result. Is
there a step I am missing to properly reset the system for the
subsequent solves?

Thanks in advance for your input...

Sandy 
-- 
C.A.(Sandy) Mader
Ph.D Candidate
Multidisciplinary Design Optimization Laboratory
University of Toronto Institute for Aerospace Studies
mader at utias.utoronto.ca