[petsc-users] Fwd: SNES with Fieldsplit + MatNest in parallel

[barnafi]

I have checked the matrix in a very small problem and they are the 
same, so the problem is definitely somewhere in the ordering/ghost 
interface from fenics, so PETSc is doing everything fine.

Thanks for your help, I'm moving to the fenics forum.

Best,
NB

On Tue, Nov 9, 2021 at 07:01, Matthew Knepley <knepley at gmail.com> wrote:
> On Tue, Nov 9, 2021 at 6:52 AM Nicolas Barnafi <nabw91 at gmail.com> 
> wrote:
>> Thanks for the help Matthew.
>>> 
>>> The next step would be to verify that the operators you get in 
>>> those slots are correct in parallel. I normally just print out the 
>>> matrix
>>> for a small problem. This is the only way I know to debug assembly.
>>> 
>>> To check the solver, you set it to an exact solver and see that it 
>>> takes 1 iterate. That means using LU as the subsolver
>>> for the first block, using the full Schur complement, and solving 
>>> the Schur system with a residual tolerance of something like 1e-10.
>>> 
>> 
>>  There is certainly something wrong in the nested matrix, as this is 
>> the output in serial (LU + full Schur LU):
> 
> Yes, this is why I recommend going to just 2 or 4 cells, so that you 
> can see explicitly what is wrong with the matrix.
> 
>   Thanks,
> 
>      Matt
> 
>>   0 SNES Function norm 3.240309155660e+03
>>      0 KSP Residual norm 3.240309155660e+03
>>      1 KSP Residual norm 3.219893365194e-08
>>    1 SNES Function norm 3.999872813081e+03
>>      0 KSP Residual norm 1.214145680994e+04
>>      1 KSP Residual norm 2.742817643080e-04
>>    2 SNES Function norm 3.663148395735e+01
>>      0 KSP Residual norm 3.980103473350e+03
>>      1 KSP Residual norm 1.814754406828e-05
>>    3 SNES Function norm 2.136252100504e+01
>>      0 KSP Residual norm 6.970609974245e+01
>>      1 KSP Residual norm 1.982821753925e-05
>>    4 SNES Function norm 1.059694009734e-02
>>      0 KSP Residual norm 2.136131725219e+01
>>      1 KSP Residual norm 4.545878835451e-06
>>      2 KSP Residual norm 8.841427426699e-08
>>      3 KSP Residual norm 3.305898584949e-09
>>    5 SNES Function norm 3.729120062850e-03
>>      0 KSP Residual norm 3.846880232871e-03
>>      1 KSP Residual norm 1.740268533385e-06
>>      2 KSP Residual norm 3.013108797112e-08
>>      3 KSP Residual norm 1.105602265959e-09
>>    6 SNES Function norm 1.117697448247e-09
>> 
>>  whereas in parallel it is this (two processes):
>> 
>>    0 SNES Function norm 3.240309155660e+03
>>      0 KSP Residual norm 3.240309155660e+03
>>      1 KSP Residual norm 2.158162432432e-01
>>      2 KSP Residual norm 2.156121452805e-01
>>      3 KSP Residual norm 2.148639482595e-01
>>      4 KSP Residual norm 2.148248193325e-01
>>      5 KSP Residual norm 1.305140166188e-03
>>    Linear solve converged due to CONVERGED_RTOL iterations 5
>>    1 SNES Function norm 4.774611203034e+42
>> 
>> 
>>  Best,
>>  NB
> 
> 
> --
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/