[petsc-users] Suggestions for solver and pc

[Smith, Barry F.]

-ksp_type gmres -ksp_gmres_restart 200  -pc_type asm -sub_pc_type lu -pc_asm_overlap 3 -ksp_monitor

It will run like molasses but may converge 

Good luck


> On May 17, 2019, at 3:40 AM, Sal Am via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Hello, 
> 
> So I am trying to solve this problem in the frequency domain by solving a Ax=b. It is apparently very ill conditioned. RF wave plasma interaction simulation. There are 6M finite elements the matrix is of the size:
> 
> nnx: 1257303210 (1B non-zero elements)
> n: 20347817 (size of matrix so 20M x 20M) 
> 
> What I have tried so far: 
> 
>  -ksp_type bcgs -pc_type gamg
> 
>  -ksp_type gmres -ksp_gmres_restart 150 -pc_type bjacobi -sub_pc_type ilu -sub_pc_factor_levels 5 -sub_ksp_type bcgs -mattransposematmult_via scalable -build_twosided allreduce
> 
> -ksp_type bcgs -pc_type gamg -mattransposematmult_via scalable -build_twosided allreduce 
> 
> -ksp_type bcgs -pc_type asm -sub_pc_type lu
> 
> None of the above has really shown any convergence after 2 days of running the simulation. The farthest I got was using gmres + bjacobi which gave me a ||r||/||b|| of the order 1e-2 but it got stuck between 1e-2 and 1e-1 after a week of having left it running.
> 
> What I have available in terms of computational resources:
> select=25:ncpus=16:mpiprocs=16:mem=230GB
> 
> So 25 nodes with around 6TB of total memory.
> 
> Thank you.
>