[petsc-users] Issues with subsequent GMRES Solves

Wed May 12 13:05:45 CDT 2010

   Are you calling KSPSetOperators(ksp,A,B,DIFFERENT_NONZERO_PATTERN); before the second adjoint solve? Or are you trying to reuse a preconditioner from the first set of solves?

   When the "second adjoint" converges and when it doesn't converge are you using the exact same matrix A, matrix B and right hand side?

   Barry

On May 12, 2010, at 1:00 PM, C.A.(Sandy) Mader wrote:

> See responses in text below:
> On Wed, 2010-05-12 at 12:48 -0500, Barry Smith wrote:
>> Is it the same matrix or a different matrix that is solved with the "second adjoint system"?  Or is it just a different right hand side?
> It is a different matrix. In this case I solve the CFD at a given flow
> condition, then solve the adjoint twice (two different RHS) both
> converge. Then I solve the CFD case for a second flow condition,
> followed by two more adjoint solutions. Neither of which solve. However,
> if I solver directly for the second flow condition, the adojint system
> will solve...
>> 
>>   If it is a different system how do you know it will even converge for the new matrix? Maybe the new matrix is much more difficult?
> see above answer, if the same system will solve under one set of
> conditions but not under another...
>> 
>>   Is it possible the "second adjoint system" is singular?
> The second system will solve if I solve it on its own.
> 
>>   Given that this is a very strong preconditioner the convergence for the first adjoint system is pretty poor.
> We use an approximate preconditioner to reduce the memory requirements
> of the solver. I do not have this problem if I use the exact matrix as
> the preconditioner but it limits my problem size....
>> 
>>   Does this happen also for much smaller problems? Recommend running much smaller problem to see what happens convergence wise, then run that much smaller problem with -pc_type lu to see if the direct solver handles it happily.
> I have tried it on a smaller case and this problem does not occur...
>> 
>> 
>> 
>> On May 12, 2010, at 12:39 PM, C.A.(Sandy) Mader wrote:
>> 
>>> I am using PETSc to solve an adjoint system for a CFD code. 
>>> The issue I am having is that if I try to run successive cfd/adjoint
>>> solutions, the adjoint fails to converge after the first cfd/adjoint
>>> solution cycle. i.e.
>>> Solve CFD (ok)
>>> Solve adjoint system with multiple RHS's (ok)
>>> Solve CFD at new point (ok)
>>> Solve adjoint system at new point with multiple RHS's (GMRES iterations
>>> stall)
>>> 
>>> PETSc is not used to solve the CFD system, only the adjoint system.
>>> 
>>> I am currently using the following solver options:
>>> 'Adjoint solver type': 'GMRES',\
>>> 'adjoint relative tolerance':1e-6,\
>>> 'adjoint absolute tolerance':1e-16,\
>>> 'adjoint max iterations': 1500,\
>>> 'adjoint restart iteration' : 80,\
>>> 'adjoint monitor step': 10,\
>>> 'Preconditioner Side': 'right',\
>>> 'Matrix Ordering': 'NestedDissection',\
>>> 'Global Preconditioner Type': 'Additive Schwartz',\
>>> 'Local Preconditioner Type' : 'ILU',\
>>> 'ILU Fill Levels': 3,\
>>> 'ASM Overlap' : 5,\
>>> 
>>> Also, I am using the fortran interface to PETSc.
>>> 
>>> As a sample, I have included two convergence histories. Both are at the
>>> same converged CFD point(this case is ~1.2million cell so the PETSc
>>> system is approx 6million degrees of freedom). The first is an example
>>> where the adjoint system is solved the first time through the cycle. The
>>> second is an example of where the adjoint is solved the second time
>>> through the cycle:
>>> 1) as first point in cycle
>>> # ... KSP properties:
>>> #       type        : gmres     
>>> #       tolerances  : rel = 1.0E-10
>>> #                     abs = 1.0E-16
>>> #                     div = 1.0E+05
>>> #       max.iter.   : 1500
>>> #       precond.type: asm   
>>> Solving ADjoint Transpose with PETSc...
>>>  0 KSP Residual norm 0.1392E+00
>>> 10 KSP Residual norm 0.6394E-01
>>> 20 KSP Residual norm 0.6106E-01
>>> 30 KSP Residual norm 0.6019E-01
>>> 40 KSP Residual norm 0.5941E-01
>>> 50 KSP Residual norm 0.5876E-01
>>> 60 KSP Residual norm 0.5602E-01
>>> 70 KSP Residual norm 0.4915E-01
>>> 80 KSP Residual norm 0.3994E-01
>>> 90 KSP Residual norm 0.3892E-01
>>> 100 KSP Residual norm 0.3854E-01
>>> 110 KSP Residual norm 0.3794E-01
>>> 120 KSP Residual norm 0.3717E-01
>>> 130 KSP Residual norm 0.3630E-01
>>> 140 KSP Residual norm 0.3415E-01
>>> ...
>>> 900 KSP Residual norm 0.2437E-09
>>> 910 KSP Residual norm 0.1452E-09
>>> 920 KSP Residual norm 0.1025E-09
>>> 930 KSP Residual norm 0.6875E-10
>>> 940 KSP Residual norm 0.4141E-10
>>> 950 KSP Residual norm 0.2317E-10
>>> 960 KSP Residual norm 0.1559E-10
>>> Solving ADjoint Transpose with PETSc time (s) =   391.43
>>> Norm of error = 0.1366E-10    Iterations =  965
>>> ------------------------------------------------
>>> PETSc solver converged after   965 iterations.
>>> ------------------------------------------------ 
>>> 
>>> 2) As second point in cycle
>>> # ... KSP properties:
>>> #       type        : gmres     
>>> #       tolerances  : rel = 1.0E-10
>>> #                     abs = 1.0E-16
>>> #                     div = 1.0E+05
>>> #       max.iter.   : 1500
>>> #       precond.type: asm   
>>> Solving ADjoint Transpose with PETSc...
>>>  0 KSP Residual norm 0.1392E+00
>>> 10 KSP Residual norm 0.6400E-01
>>> 20 KSP Residual norm 0.6140E-01
>>> 30 KSP Residual norm 0.6060E-01
>>> 40 KSP Residual norm 0.5995E-01
>>> 50 KSP Residual norm 0.5974E-01
>>> 60 KSP Residual norm 0.5971E-01
>>> 70 KSP Residual norm 0.5957E-01
>>> 80 KSP Residual norm 0.5906E-01
>>> 90 KSP Residual norm 0.5906E-01
>>> 100 KSP Residual norm 0.5906E-01
>>> 110 KSP Residual norm 0.5903E-01
>>> 120 KSP Residual norm 0.5901E-01
>>> 130 KSP Residual norm 0.5901E-01
>>> 140 KSP Residual norm 0.5901E-01
>>> ...
>>> 1400 KSP Residual norm 0.5895E-01
>>> 1410 KSP Residual norm 0.5895E-01
>>> 1420 KSP Residual norm 0.5895E-01
>>> 1430 KSP Residual norm 0.5895E-01
>>> 1440 KSP Residual norm 0.5895E-01
>>> 1450 KSP Residual norm 0.5895E-01
>>> 1460 KSP Residual norm 0.5895E-01
>>> 1470 KSP Residual norm 0.5895E-01
>>> 1480 KSP Residual norm 0.5895E-01
>>> 1490 KSP Residual norm 0.5895E-01
>>> 1500 KSP Residual norm 0.5895E-01
>>> Solving ADjoint Transpose with PETSc time (s) =   516.59
>>> Norm of error = 0.5895E-01    Iterations = 1500
>>> ------------------------------------------------
>>> PETSc solver converged after  1500 iterations.
>>> ------------------------------------------------
>>> I have tried both resetting the operators between cycles and completely
>>> destroying the KSP object between cycles. Both give the same result. Is
>>> there a step I am missing to properly reset the system for the
>>> subsequent solves?
>>> 
>>> Thanks in advance for your input...
>>> 
>>> Sandy 
>>> -- 
>>> C.A.(Sandy) Mader
>>> Ph.D Candidate
>>> Multidisciplinary Design Optimization Laboratory
>>> University of Toronto Institute for Aerospace Studies
>>> mader at utias.utoronto.ca
>>> 
>> 
>> 
> 