Problem in convergence after upgrade to petsc 3.0.0
Randall Mackie
rlmackie862 at gmail.com
Sat Jan 31 22:08:27 CST 2009
Thanks. I know you're nervous, but trust me, it's because I have these
air layers with resistivity 1e10. A 1% error in the air causes a huge
residual. The residual reduction I'm getting is fantastic for these
problems - I could send you lots of papers from the geophysics community
dealing with low frequency EM propagation in the earth - a nasty
numerical problem I've been dealing with for 20 years. The residual error
reduction I'm getting now is fantastic, and the solutions at the earth's
surface are practically indistinguishable from direct solution results
(for smaller models).
Actually, it was *YOU* several years ago who forced me to rethink these
solutions. Remember, I was beginning to use DA's, and because my
equations were so strongly coupled, the performance was terrible in
parallel. I had to go back and rethink the physics, and I wound up
adding in another equation, div H = 0,l and that stabilized everything
and now the convergence is great (even if you don't think so!).
Thanks again, I'll give these a try.
Randy
Barry Smith wrote:
>
> I have found the problem. Both BiCG-stab and BiCG-stab lhad been
> optimized to
> use MDots in some place instead of Dot(). Unfortunately the PetscConj()
> was not handled
> properly for complex numbers.
>
> This will be fixed in the next patch. You can also use the two
> attached files
> src/vec/vec/utils/vinv.c and src/ksp/ksp/impls/bcgsl/bcgsl.c (recompile
> in those two
> directories).
>
> Barry
>
> I'd be very nervous about trusting any linear solution results if the
> residual norms are large.
>
> On Jan 30, 2009, at 11:34 PM, Randall Mackie wrote:
>
>> Barry, trust me, these are *NOT* garbage. The reason you're seeing
>> such
>> a jump in true residual is because in the case of ex10, I assume
>> you're starting
>> with zero values. The nature of the physical system I'm solving has
>> air layers with
>> electrical resistivity of 1e10 up against earth layers of resistivity 1.
>> A small change causes a large increase in the true residual. In my
>> program,
>> I start with 1D boundary values which are much closer to the truth,
>> and you
>> see only a slight increase in residual.
>>
>> I know from a numerical perspective you don't like these systems,
>> but that's what we deal with and they have been validated with direct
>> solvers and against 1D and analytic solutions for appropriate models.
>>
>>
>> I will send the binaryoutput to petsc-maint. Regardless of whether or
>> not you like my numerical systems, there is obviously something wrong
>> in 3.0.0-p2 with the BCGSL and BCGS implementations.
>>
>> Randy
>>
>>
>>
>> Barry Smith wrote:
>>> The convergence of both of those is GARBAGE. The true residual gets
>>> hugely worse in the first
>>> step and then stays bad in both cases. The fact that the
>>> preconditioned residual norm gets small
>>> better with the old PETSc then the new is kind of irrelevent. The
>>> solution that KSP claims to give in
>>> both cases is frankly crap.
>>> Send me the output for both with -ksp_monitor_true_residual with
>>> -ksp_type gmres with old and
>>> new PETSc.
>>> Barry
>>> On Jan 30, 2009, at 11:11 PM, Randall Mackie wrote:
>>>> Barry,
>>>>
>>>> I've run ex10 from 2.3.3-p11 and from 3.0.0-p2 on the same
>>>> binaryoutput
>>>> using the same parameters input to ex10 (ksp=bcgsl and pc=bjacobi,
>>>> sub_pc=ilu)
>>>> and the convergence is different. Note, this is the binaryoutput
>>>> from the
>>>> v 2.3.3-p11.
>>>>
>>>> Here is the convergence for ex10 from 2.3.3-p11:
>>>>
>>>> [rmackie
>>>> ~/SPARSE/PETsc/petsc-2.3.3-p11/src/ksp/ksp/examples/tutorials]
>>>> ./cmd_test
>>>> 0 KSP preconditioned resid norm 2.434324107505e-05 true resid norm
>>>> 3.334159901684e-02 ||Ae||/||Ax|| 1.000000000000e+00
>>>> 2 KSP preconditioned resid norm 4.170258940641e-07 true resid norm
>>>> 1.624814448703e+04 ||Ae||/||Ax|| 4.873234927583e+05
>>>> 4 KSP preconditioned resid norm 8.984963207977e-08 true resid norm
>>>> 2.848180153976e+04 ||Ae||/||Ax|| 8.542422193181e+05
>>>> 6 KSP preconditioned resid norm 3.526062648105e-08 true resid norm
>>>> 1.884591980560e+04 ||Ae||/||Ax|| 5.652374319564e+05
>>>> 8 KSP preconditioned resid norm 1.241153849592e-08 true resid norm
>>>> 1.045401827970e+04 ||Ae||/||Ax|| 3.135427990247e+05
>>>> 10 KSP preconditioned resid norm 5.141701606491e-09 true resid norm
>>>> 5.371087738119e+03 ||Ae||/||Ax|| 1.610926859088e+05
>>>> 12 KSP preconditioned resid norm 2.591522726108e-09 true resid norm
>>>> 3.519633363751e+03 ||Ae||/||Ax|| 1.055628244456e+05
>>>> 14 KSP preconditioned resid norm 1.452300847705e-09 true resid norm
>>>> 2.509415501645e+03 ||Ae||/||Ax|| 7.526380184636e+04
>>>> 16 KSP preconditioned resid norm 8.978785776773e-10 true resid norm
>>>> 1.872610647707e+03 ||Ae||/||Ax|| 5.616439231845e+04
>>>> 18 KSP preconditioned resid norm 5.743601707920e-10 true resid norm
>>>> 1.266421550645e+03 ||Ae||/||Ax|| 3.798322779917e+04
>>>> 20 KSP preconditioned resid norm 3.678705188041e-10 true resid norm
>>>> 7.536064149571e+02 ||Ae||/||Ax|| 2.260258767363e+04
>>>> 22 KSP preconditioned resid norm 2.687340247327e-10 true resid norm
>>>> 5.533061905955e+02 ||Ae||/||Ax|| 1.659507062981e+04
>>>> 24 KSP preconditioned resid norm 2.142070779181e-10 true resid norm
>>>> 4.684485692902e+02 ||Ae||/||Ax|| 1.404997309978e+04
>>>> 26 KSP preconditioned resid norm 1.927583902818e-10 true resid norm
>>>> 6.146993326148e+02 ||Ae||/||Ax|| 1.843640829296e+04
>>>> 28 KSP preconditioned resid norm 1.330050553611e-10 true resid norm
>>>> 4.217422387701e+02 ||Ae||/||Ax|| 1.264913055181e+04
>>>> 30 KSP preconditioned resid norm 1.125778226987e-10 true resid norm
>>>> 3.058702983977e+02 ||Ae||/||Ax|| 9.173834111652e+03
>>>> 32 KSP preconditioned resid norm 8.786382414490e-11 true resid norm
>>>> 2.001904013306e+02 ||Ae||/||Ax|| 6.004223169665e+03
>>>> 34 KSP preconditioned resid norm 6.396029789854e-11 true resid norm
>>>> 1.761414838880e+02 ||Ae||/||Ax|| 5.282934504701e+03
>>>> 36 KSP preconditioned resid norm 5.462116905743e-11 true resid norm
>>>> 1.540860953921e+02 ||Ae||/||Ax|| 4.621436881724e+03
>>>> 38 KSP preconditioned resid norm 5.410616131680e-11 true resid norm
>>>> 1.910613970561e+02 ||Ae||/||Ax|| 5.730420936308e+03
>>>> 40 KSP preconditioned resid norm 7.946054076400e-11 true resid norm
>>>> 2.743761797072e+02 ||Ae||/||Ax|| 8.229244781230e+03
>>>> 42 KSP preconditioned resid norm 3.519925335531e-11 true resid norm
>>>> 1.022381658728e+02 ||Ae||/||Ax|| 3.066384603244e+03
>>>> 42 KSP preconditioned resid norm 3.519925335531e-11 true resid norm
>>>> 1.022381658728e+02 ||Ae||/||Ax|| 3.066384603244e+03
>>>> Number of iterations = 42
>>>>
>>>>
>>>> Here is the convergence for ex10 from 3.0.0-p2:
>>>>
>>>> rmackie
>>>> ~/SPARSE/PETsc/petsc-3.0.0-p2/src/ksp/ksp/examples/tutorials]
>>>> ./cmd_test
>>>> 0 KSP preconditioned resid norm 2.434324107505e-05 true resid norm
>>>> 3.334159901684e-02 ||Ae||/||Ax|| 1.000000000000e+00
>>>> 2 KSP preconditioned resid norm 4.798562862178e-07 true resid norm
>>>> 1.642225014976e+04 ||Ae||/||Ax|| 4.925453677690e+05
>>>> 4 KSP preconditioned resid norm 1.587355624039e-07 true resid norm
>>>> 3.502938376404e+04 ||Ae||/||Ax|| 1.050620989904e+06
>>>> 6 KSP preconditioned resid norm 8.015528577103e-08 true resid norm
>>>> 2.536595627800e+04 ||Ae||/||Ax|| 7.607900348506e+05
>>>> 8 KSP preconditioned resid norm 3.039306325625e-08 true resid norm
>>>> 1.253662335500e+04 ||Ae||/||Ax|| 3.760054623856e+05
>>>> 10 KSP preconditioned resid norm 1.729327127129e-08 true resid norm
>>>> 8.080934369743e+03 ||Ae||/||Ax|| 2.423679309940e+05
>>>> 12 KSP preconditioned resid norm 1.039787850500e-08 true resid norm
>>>> 4.777791224801e+03 ||Ae||/||Ax|| 1.432982030162e+05
>>>> 14 KSP preconditioned resid norm 5.025780191774e-09 true resid norm
>>>> 3.147651686153e+03 ||Ae||/||Ax|| 9.440614064618e+04
>>>> 16 KSP preconditioned resid norm 3.311781967950e-09 true resid norm
>>>> 2.688053658600e+03 ||Ae||/||Ax|| 8.062161797467e+04
>>>> 18 KSP preconditioned resid norm 5.621276662229e-09 true resid norm
>>>> 3.098810918425e+03 ||Ae||/||Ax|| 9.294128085637e+04
>>>> 20 KSP preconditioned resid norm 1.184533040734e-08 true resid norm
>>>> 5.469874887175e+03 ||Ae||/||Ax|| 1.640555656737e+05
>>>> 22 KSP preconditioned resid norm 2.494642590524e-08 true resid norm
>>>> 1.003335643955e+04 ||Ae||/||Ax|| 3.009260723962e+05
>>>> 24 KSP preconditioned resid norm 5.136091311727e-08 true resid norm
>>>> 1.828432513826e+04 ||Ae||/||Ax|| 5.483937686680e+05
>>>> 26 KSP preconditioned resid norm 9.627430082715e-08 true resid norm
>>>> 1.175348501769e+04 ||Ae||/||Ax|| 3.525171366783e+05
>>>> 28 KSP preconditioned resid norm 6.409712928943e-08 true resid norm
>>>> 5.524687582334e+03 ||Ae||/||Ax|| 1.656995388716e+05
>>>> 30 KSP preconditioned resid norm 6.013091685526e-07 true resid norm
>>>> 1.371019320496e+04 ||Ae||/||Ax|| 4.112038297274e+05
>>>> 32 KSP preconditioned resid norm 7.026562454712e-07 true resid norm
>>>> 1.053982255306e+04 ||Ae||/||Ax|| 3.161162890759e+05
>>>> 34 KSP preconditioned resid norm 4.086784421188e-07 true resid norm
>>>> 5.503180350963e+03 ||Ae||/||Ax|| 1.650544818856e+05
>>>> 36 KSP preconditioned resid norm 1.651444280250e-06 true resid norm
>>>> 1.984011183420e+04 ||Ae||/||Ax|| 5.950557987388e+05
>>>> 38 KSP preconditioned resid norm 1.058319572456e-06 true resid norm
>>>> 1.403784173466e+04 ||Ae||/||Ax|| 4.210308488073e+05
>>>> 40 KSP preconditioned resid norm 4.341084013969e-05 true resid norm
>>>> 3.837773616917e+05 ||Ae||/||Ax|| 1.151046659453e+07
>>>> 42 KSP preconditioned resid norm 4.190225826231e-07 true resid norm
>>>> 6.768382935039e+03 ||Ae||/||Ax|| 2.030011497535e+05
>>>> 44 KSP preconditioned resid norm 1.054511038261e-06 true resid norm
>>>> 4.966771429542e+03 ||Ae||/||Ax|| 1.489662036615e+05
>>>> 46 KSP preconditioned resid norm 5.351004248086e-07 true resid norm
>>>> 5.112611747101e+03 ||Ae||/||Ax|| 1.533403285343e+05
>>>> 48 KSP preconditioned resid norm 7.104477128923e-07 true resid norm
>>>> 5.478002736962e+03 ||Ae||/||Ax|| 1.642993407183e+05
>>>> 50 KSP preconditioned resid norm 1.162050733932e-06 true resid norm
>>>> 5.395393687747e+03 ||Ae||/||Ax|| 1.618216836278e+05
>>>>
>>>>
>>>> When I output binaryoutput using my code compiled under 3.0.0-p2 and
>>>> run ex10, I get exactly
>>>> the same output as above. In other words, the matrix/rhs are exactly
>>>> the same (as they
>>>> should be since I didn't change anything), and something about
>>>> 3.0.0-p2 using BCGSL and ILU
>>>> is not working correctly.
>>>>
>>>> I changed ksptype to gmres, and I get the same convergence for both
>>>> 2.3.3-p11 and for
>>>> 3.0.0-p2. Therefore, I conclude that bcgsl (PCBCGSL) is not working
>>>> correctly under 3.0.0-p2.
>>>>
>>>> Randy
>>>>
>>>>
>>>> Barry Smith wrote:
>>>>> Run the old code with -ksp_view_binary this will create a file
>>>>> called binaryoutput; you can then
>>>>> run src/ksp/ksp/examples/tutorials/ex10.c using that input file.
>>>>> Use the ex10 from the old version
>>>>> of PETSc and then the ex10 from the new version. Do they have they
>>>>> same convergence?
>>>>> Now run the new code (that is build your code with petsc-3.0.) with
>>>>> -ksp_view_binary and run that
>>>>> binaryoutput file with the old and new ex10 to see what happens.
>>>>> Basically there are two possible changes with the change in the
>>>>> version:
>>>>> 1) the matrix/right hand side has changed or
>>>>> 2) the solver has changed to behave differently.
>>>>> By running the four cases you can start to get a handle on what
>>>>> has actually changed, this will
>>>>> lead you to what needs to be investigated next.
>>>>> Barry
>>>>> On Jan 30, 2009, at 8:01 PM, Randall Mackie wrote:
>>>>>> I just downloaded and compiled Petsc 3.0.0-p2, and after making
>>>>>> some changes
>>>>>> in my code to specify the correct location of the include files,
>>>>>> finally
>>>>>> got everything to compile okay.
>>>>>>
>>>>>> Now, I'm trying to run my test problem, and it's not converging.
>>>>>> When I say
>>>>>> not converging, the first line (with ksp_monitor_true_residual)
>>>>>> shows that
>>>>>> the true and preconditioned residuals are the same as before, but
>>>>>> immediately
>>>>>> thereafter, the preconditioned residual fails to go below 1e-8
>>>>>> whereas before
>>>>>> it quickly went down to 1e-15.
>>>>>>
>>>>>> The options in my command file are:
>>>>>>
>>>>>> -ksp_type bcgsl
>>>>>> -pc_type bjacobi
>>>>>> -sub_pc_type ilu
>>>>>> -sub_pc_factor_levels 3
>>>>>> -sub_pc_factor_fill 6
>>>>>>
>>>>>>
>>>>>> The only thing I see in the Change notes are that the ILU defaults
>>>>>> to shifting
>>>>>> so that it's p.d. but I don't see an easy way to turn this off by
>>>>>> the command
>>>>>> line to see if that's the problem. I tried to do it in my program,
>>>>>> but it's unclear
>>>>>> if I did that correctly.
>>>>>>
>>>>>> Any suggestions?
>>>>>>
>>>>>> Thanks, Randy
More information about the petsc-users
mailing list