Problem in convergence after upgrade to petsc 3.0.0

Barry Smith bsmith at mcs.anl.gov
Sat Jan 31 21:40:19 CST 2009 

    I have found the problem. Both BiCG-stab and BiCG-stab lhad been  
optimized to
use MDots in some place instead of Dot(). Unfortunately the  
PetscConj() was not handled
properly for complex numbers.

    This will be fixed in the next patch. You can also use the two  
attached files
src/vec/vec/utils/vinv.c and src/ksp/ksp/impls/bcgsl/bcgsl.c  
(recompile in those two
directories).

    Barry

    I'd be very nervous about trusting any linear solution results if  
the residual norms are large.

On Jan 30, 2009, at 11:34 PM, Randall Mackie wrote:

> 	Barry, trust me, these are *NOT* garbage. The reason you're seeing  
> such
> a jump in true residual is because in the case of ex10, I assume  
> you're starting
> with zero values. The nature of the physical system I'm solving has  
> air layers with
> electrical resistivity of 1e10 up against earth layers of  
> resistivity 1.
> A small change causes a large increase in the true residual. In my  
> program,
> I start with 1D boundary values which are much closer to the truth,  
> and you
> see only a slight increase in residual.
>
> 	I know from a numerical perspective you don't like these systems,
> but that's what we deal with and they have been validated with direct
> solvers and against 1D and analytic solutions for appropriate models.
>
>
> 	I will send the binaryoutput to petsc-maint. Regardless of whether or
> not you like my numerical systems, there is obviously something wrong
> in 3.0.0-p2 with the BCGSL and BCGS implementations.
>
> 	Randy
> 	
> 	
>
> Barry Smith wrote:
>>   The convergence of both of those is GARBAGE. The true residual  
>> gets hugely worse in the first
>> step and then stays bad in both cases. The fact that the  
>> preconditioned residual norm gets small
>> better with the old PETSc then the new is kind of irrelevent. The  
>> solution that KSP claims to give in
>> both cases is frankly crap.
>>  Send me the output for both with -ksp_monitor_true_residual with - 
>> ksp_type gmres with old and
>> new PETSc.
>>   Barry
>> On Jan 30, 2009, at 11:11 PM, Randall Mackie wrote:
>>>    Barry,
>>>
>>>    I've run ex10 from 2.3.3-p11 and from 3.0.0-p2 on the same  
>>> binaryoutput
>>> using the same parameters input to ex10 (ksp=bcgsl and pc=bjacobi,  
>>> sub_pc=ilu)
>>> and the convergence is different. Note, this is the binaryoutput  
>>> from the
>>> v 2.3.3-p11.
>>>
>>> Here is the convergence for ex10 from 2.3.3-p11:
>>>
>>> [rmackie ~/SPARSE/PETsc/petsc-2.3.3-p11/src/ksp/ksp/examples/ 
>>> tutorials] ./cmd_test
>>> 0 KSP preconditioned resid norm 2.434324107505e-05 true resid norm  
>>> 3.334159901684e-02 ||Ae||/||Ax|| 1.000000000000e+00
>>> 2 KSP preconditioned resid norm 4.170258940641e-07 true resid norm  
>>> 1.624814448703e+04 ||Ae||/||Ax|| 4.873234927583e+05
>>> 4 KSP preconditioned resid norm 8.984963207977e-08 true resid norm  
>>> 2.848180153976e+04 ||Ae||/||Ax|| 8.542422193181e+05
>>> 6 KSP preconditioned resid norm 3.526062648105e-08 true resid norm  
>>> 1.884591980560e+04 ||Ae||/||Ax|| 5.652374319564e+05
>>> 8 KSP preconditioned resid norm 1.241153849592e-08 true resid norm  
>>> 1.045401827970e+04 ||Ae||/||Ax|| 3.135427990247e+05
>>> 10 KSP preconditioned resid norm 5.141701606491e-09 true resid  
>>> norm 5.371087738119e+03 ||Ae||/||Ax|| 1.610926859088e+05
>>> 12 KSP preconditioned resid norm 2.591522726108e-09 true resid  
>>> norm 3.519633363751e+03 ||Ae||/||Ax|| 1.055628244456e+05
>>> 14 KSP preconditioned resid norm 1.452300847705e-09 true resid  
>>> norm 2.509415501645e+03 ||Ae||/||Ax|| 7.526380184636e+04
>>> 16 KSP preconditioned resid norm 8.978785776773e-10 true resid  
>>> norm 1.872610647707e+03 ||Ae||/||Ax|| 5.616439231845e+04
>>> 18 KSP preconditioned resid norm 5.743601707920e-10 true resid  
>>> norm 1.266421550645e+03 ||Ae||/||Ax|| 3.798322779917e+04
>>> 20 KSP preconditioned resid norm 3.678705188041e-10 true resid  
>>> norm 7.536064149571e+02 ||Ae||/||Ax|| 2.260258767363e+04
>>> 22 KSP preconditioned resid norm 2.687340247327e-10 true resid  
>>> norm 5.533061905955e+02 ||Ae||/||Ax|| 1.659507062981e+04
>>> 24 KSP preconditioned resid norm 2.142070779181e-10 true resid  
>>> norm 4.684485692902e+02 ||Ae||/||Ax|| 1.404997309978e+04
>>> 26 KSP preconditioned resid norm 1.927583902818e-10 true resid  
>>> norm 6.146993326148e+02 ||Ae||/||Ax|| 1.843640829296e+04
>>> 28 KSP preconditioned resid norm 1.330050553611e-10 true resid  
>>> norm 4.217422387701e+02 ||Ae||/||Ax|| 1.264913055181e+04
>>> 30 KSP preconditioned resid norm 1.125778226987e-10 true resid  
>>> norm 3.058702983977e+02 ||Ae||/||Ax|| 9.173834111652e+03
>>> 32 KSP preconditioned resid norm 8.786382414490e-11 true resid  
>>> norm 2.001904013306e+02 ||Ae||/||Ax|| 6.004223169665e+03
>>> 34 KSP preconditioned resid norm 6.396029789854e-11 true resid  
>>> norm 1.761414838880e+02 ||Ae||/||Ax|| 5.282934504701e+03
>>> 36 KSP preconditioned resid norm 5.462116905743e-11 true resid  
>>> norm 1.540860953921e+02 ||Ae||/||Ax|| 4.621436881724e+03
>>> 38 KSP preconditioned resid norm 5.410616131680e-11 true resid  
>>> norm 1.910613970561e+02 ||Ae||/||Ax|| 5.730420936308e+03
>>> 40 KSP preconditioned resid norm 7.946054076400e-11 true resid  
>>> norm 2.743761797072e+02 ||Ae||/||Ax|| 8.229244781230e+03
>>> 42 KSP preconditioned resid norm 3.519925335531e-11 true resid  
>>> norm 1.022381658728e+02 ||Ae||/||Ax|| 3.066384603244e+03
>>> 42 KSP preconditioned resid norm 3.519925335531e-11 true resid  
>>> norm 1.022381658728e+02 ||Ae||/||Ax|| 3.066384603244e+03
>>> Number of iterations =  42
>>>
>>>
>>> Here is the convergence for ex10 from 3.0.0-p2:
>>>
>>> rmackie ~/SPARSE/PETsc/petsc-3.0.0-p2/src/ksp/ksp/examples/ 
>>> tutorials] ./cmd_test
>>> 0 KSP preconditioned resid norm 2.434324107505e-05 true resid norm  
>>> 3.334159901684e-02 ||Ae||/||Ax|| 1.000000000000e+00
>>> 2 KSP preconditioned resid norm 4.798562862178e-07 true resid norm  
>>> 1.642225014976e+04 ||Ae||/||Ax|| 4.925453677690e+05
>>> 4 KSP preconditioned resid norm 1.587355624039e-07 true resid norm  
>>> 3.502938376404e+04 ||Ae||/||Ax|| 1.050620989904e+06
>>> 6 KSP preconditioned resid norm 8.015528577103e-08 true resid norm  
>>> 2.536595627800e+04 ||Ae||/||Ax|| 7.607900348506e+05
>>> 8 KSP preconditioned resid norm 3.039306325625e-08 true resid norm  
>>> 1.253662335500e+04 ||Ae||/||Ax|| 3.760054623856e+05
>>> 10 KSP preconditioned resid norm 1.729327127129e-08 true resid  
>>> norm 8.080934369743e+03 ||Ae||/||Ax|| 2.423679309940e+05
>>> 12 KSP preconditioned resid norm 1.039787850500e-08 true resid  
>>> norm 4.777791224801e+03 ||Ae||/||Ax|| 1.432982030162e+05
>>> 14 KSP preconditioned resid norm 5.025780191774e-09 true resid  
>>> norm 3.147651686153e+03 ||Ae||/||Ax|| 9.440614064618e+04
>>> 16 KSP preconditioned resid norm 3.311781967950e-09 true resid  
>>> norm 2.688053658600e+03 ||Ae||/||Ax|| 8.062161797467e+04
>>> 18 KSP preconditioned resid norm 5.621276662229e-09 true resid  
>>> norm 3.098810918425e+03 ||Ae||/||Ax|| 9.294128085637e+04
>>> 20 KSP preconditioned resid norm 1.184533040734e-08 true resid  
>>> norm 5.469874887175e+03 ||Ae||/||Ax|| 1.640555656737e+05
>>> 22 KSP preconditioned resid norm 2.494642590524e-08 true resid  
>>> norm 1.003335643955e+04 ||Ae||/||Ax|| 3.009260723962e+05
>>> 24 KSP preconditioned resid norm 5.136091311727e-08 true resid  
>>> norm 1.828432513826e+04 ||Ae||/||Ax|| 5.483937686680e+05
>>> 26 KSP preconditioned resid norm 9.627430082715e-08 true resid  
>>> norm 1.175348501769e+04 ||Ae||/||Ax|| 3.525171366783e+05
>>> 28 KSP preconditioned resid norm 6.409712928943e-08 true resid  
>>> norm 5.524687582334e+03 ||Ae||/||Ax|| 1.656995388716e+05
>>> 30 KSP preconditioned resid norm 6.013091685526e-07 true resid  
>>> norm 1.371019320496e+04 ||Ae||/||Ax|| 4.112038297274e+05
>>> 32 KSP preconditioned resid norm 7.026562454712e-07 true resid  
>>> norm 1.053982255306e+04 ||Ae||/||Ax|| 3.161162890759e+05
>>> 34 KSP preconditioned resid norm 4.086784421188e-07 true resid  
>>> norm 5.503180350963e+03 ||Ae||/||Ax|| 1.650544818856e+05
>>> 36 KSP preconditioned resid norm 1.651444280250e-06 true resid  
>>> norm 1.984011183420e+04 ||Ae||/||Ax|| 5.950557987388e+05
>>> 38 KSP preconditioned resid norm 1.058319572456e-06 true resid  
>>> norm 1.403784173466e+04 ||Ae||/||Ax|| 4.210308488073e+05
>>> 40 KSP preconditioned resid norm 4.341084013969e-05 true resid  
>>> norm 3.837773616917e+05 ||Ae||/||Ax|| 1.151046659453e+07
>>> 42 KSP preconditioned resid norm 4.190225826231e-07 true resid  
>>> norm 6.768382935039e+03 ||Ae||/||Ax|| 2.030011497535e+05
>>> 44 KSP preconditioned resid norm 1.054511038261e-06 true resid  
>>> norm 4.966771429542e+03 ||Ae||/||Ax|| 1.489662036615e+05
>>> 46 KSP preconditioned resid norm 5.351004248086e-07 true resid  
>>> norm 5.112611747101e+03 ||Ae||/||Ax|| 1.533403285343e+05
>>> 48 KSP preconditioned resid norm 7.104477128923e-07 true resid  
>>> norm 5.478002736962e+03 ||Ae||/||Ax|| 1.642993407183e+05
>>> 50 KSP preconditioned resid norm 1.162050733932e-06 true resid  
>>> norm 5.395393687747e+03 ||Ae||/||Ax|| 1.618216836278e+05
>>>
>>>
>>> When I output binaryoutput using my code compiled under 3.0.0-p2  
>>> and run ex10, I get exactly
>>> the same output as above. In other words, the matrix/rhs are  
>>> exactly the same (as they
>>> should be since I didn't change anything), and something about  
>>> 3.0.0-p2 using BCGSL and ILU
>>> is not working correctly.
>>>
>>> I changed ksptype to gmres, and I get the same convergence for  
>>> both 2.3.3-p11 and for
>>> 3.0.0-p2. Therefore, I conclude that bcgsl (PCBCGSL) is not  
>>> working correctly under 3.0.0-p2.
>>>
>>> Randy
>>>
>>>
>>> Barry Smith wrote:
>>>> Run the old code with -ksp_view_binary this will create a file  
>>>> called binaryoutput; you can then
>>>> run src/ksp/ksp/examples/tutorials/ex10.c using that input file.  
>>>> Use the ex10 from the old version
>>>> of PETSc and then the ex10 from the new version. Do they have  
>>>> they same convergence?
>>>> Now run the new code (that is build your code with petsc-3.0.)  
>>>> with -ksp_view_binary and run that
>>>> binaryoutput file with the old and new ex10 to see what happens.
>>>>  Basically there are two possible changes with the change in the  
>>>> version:
>>>> 1) the matrix/right hand side has changed or
>>>> 2) the solver has changed to behave differently.
>>>>  By running the four cases you can start to get a handle on what  
>>>> has actually changed, this will
>>>> lead you to what needs to be investigated next.
>>>>  Barry
>>>> On Jan 30, 2009, at 8:01 PM, Randall Mackie wrote:
>>>>> I just downloaded and compiled Petsc 3.0.0-p2, and after making  
>>>>> some changes
>>>>> in my code to specify the correct location of the include files,  
>>>>> finally
>>>>> got everything to compile okay.
>>>>>
>>>>> Now, I'm trying to run my test problem, and it's not converging.  
>>>>> When I say
>>>>> not converging, the first line (with ksp_monitor_true_residual)  
>>>>> shows that
>>>>> the true and preconditioned residuals are the same as before,  
>>>>> but immediately
>>>>> thereafter, the preconditioned residual fails to go below 1e-8  
>>>>> whereas before
>>>>> it quickly went down to 1e-15.
>>>>>
>>>>> The options in my command file are:
>>>>>
>>>>> -ksp_type bcgsl
>>>>> -pc_type bjacobi
>>>>> -sub_pc_type ilu
>>>>> -sub_pc_factor_levels 3
>>>>> -sub_pc_factor_fill 6
>>>>>
>>>>>
>>>>> The only thing I see in the Change notes are that the ILU  
>>>>> defaults to shifting
>>>>> so that it's p.d. but I don't see an easy way to turn this off  
>>>>> by the command
>>>>> line to see if that's the problem. I tried to do it in my  
>>>>> program, but it's unclear
>>>>> if I did that correctly.
>>>>>
>>>>> Any suggestions?
>>>>>
>>>>> Thanks, Randy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: vinv.c
Type: application/octet-stream
Size: 40678 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20090131/0efed259/attachment-0002.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: bcgsl.c
Type: application/octet-stream
Size: 17709 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20090131/0efed259/attachment-0003.obj>

More information about the petsc-users mailing list