[petsc-users] Suggestions for solver and pc

Fri May 17 09:20:31 CDT 2019

   As well as trying Mark's suggestion, you can try with just an overlap of 2 for the ASM and use an ILU preconditioner on each domain -sub_pc_type ilu (in combination or one at a time) There is no doubt that when using ASM for difficult problems it does require a great deal of memory, you could try smaller problems to determine how an effective solver it is or not; it is is the most effective solver then at least you know this, and if needed, purse use of larger machines.

   Barry

> On May 17, 2019, at 5:32 AM, Sal Am <tempohoper at gmail.com> wrote:
> 
> Thank you Barry for quick response, I tried that, but I get several errors of which the first one is:
> 
> [230]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [230]PETSC ERROR: Out of memory. This could be due to allocating
> [230]PETSC ERROR: too large an object or bleeding by not properly
> [230]PETSC ERROR: destroying unneeded objects.
> [230]PETSC ERROR: Memory allocated 0 Memory used by process 5116485632
> [230]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info.
> [230]PETSC ERROR: Memory requested 186645063808
> [230]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [230]PETSC ERROR: Petsc Release Version 3.10.2, unknown
> [230]PETSC ERROR: ./solveCSys on a linux-cumulus-x64 named r03n04 by vef002 Fri May 17 04:23:38 2019
> [230]PETSC ERROR: Configure options PETSC_ARCH=linux-cumulus-x64 --with-cc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicc --with-fc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpifort --with-cxx=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicxx --download-parmetis --download-metis --download-ptscotch --download-superlu_dist --with-64-bit-indices --with-scalar-type=complex --with-debugging=no --download-scalapack --download-fblaslapack=1 --download-cmake
> [230]PETSC ERROR: #1 PetscFreeSpaceGet() line 11 in /lustre/home/vef002/petsc/src/mat/utils/freespace.c
> [230]PETSC ERROR: #2 PetscMallocA() line 390 in /lustre/home/vef002/petsc/src/sys/memory/mal.c
> [230]PETSC ERROR: #3 PetscFreeSpaceGet() line 11 in /lustre/home/vef002/petsc/src/mat/utils/freespace.c
> [230]PETSC ERROR: #4 MatLUFactorSymbolic_SeqAIJ() line 349 in /lustre/home/vef002/petsc/src/mat/impls/aij/seq/aijfact.c
> [230]PETSC ERROR: #5 MatLUFactorSymbolic() line 3015 in /lustre/home/vef002/petsc/src/mat/interface/matrix.c
> [230]PETSC ERROR: #6 PCSetUp_LU() line 95 in /lustre/home/vef002/petsc/src/ksp/pc/impls/factor/lu/lu.c
> [230]PETSC ERROR: #7 PCSetUp() line 932 in /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
> [230]PETSC ERROR: #8 KSPSetUp() line 391 in /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
> [230]PETSC ERROR: #9 PCSetUpOnBlocks_ASM() line 450 in /lustre/home/vef002/petsc/src/ksp/pc/impls/asm/asm.c
> [230]PETSC ERROR: #10 PCSetUpOnBlocks() line 963 in /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
> [230]PETSC ERROR: #11 KSPSetUpOnBlocks() line 223 in /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
> [230]PETSC ERROR: #12 KSPSolve() line 724 in /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
> 
> So apparently ~6TB is not enough for the suggested routine... 
> 
> On Fri, May 17, 2019 at 9:45 AM Smith, Barry F. <bsmith at mcs.anl.gov> wrote:
> 
> -ksp_type gmres -ksp_gmres_restart 200  -pc_type asm -sub_pc_type lu -pc_asm_overlap 3 -ksp_monitor
> 
> It will run like molasses but may converge 
> 
> Good luck
> 
> 
> > On May 17, 2019, at 3:40 AM, Sal Am via petsc-users <petsc-users at mcs.anl.gov> wrote:
> > 
> > Hello, 
> > 
> > So I am trying to solve this problem in the frequency domain by solving a Ax=b. It is apparently very ill conditioned. RF wave plasma interaction simulation. There are 6M finite elements the matrix is of the size:
> > 
> > nnx: 1257303210 (1B non-zero elements)
> > n: 20347817 (size of matrix so 20M x 20M) 
> > 
> > What I have tried so far: 
> > 
> >  -ksp_type bcgs -pc_type gamg
> > 
> >  -ksp_type gmres -ksp_gmres_restart 150 -pc_type bjacobi -sub_pc_type ilu -sub_pc_factor_levels 5 -sub_ksp_type bcgs -mattransposematmult_via scalable -build_twosided allreduce
> > 
> > -ksp_type bcgs -pc_type gamg -mattransposematmult_via scalable -build_twosided allreduce 
> > 
> > -ksp_type bcgs -pc_type asm -sub_pc_type lu
> > 
> > None of the above has really shown any convergence after 2 days of running the simulation. The farthest I got was using gmres + bjacobi which gave me a ||r||/||b|| of the order 1e-2 but it got stuck between 1e-2 and 1e-1 after a week of having left it running.
> > 
> > What I have available in terms of computational resources:
> > select=25:ncpus=16:mpiprocs=16:mem=230GB
> > 
> > So 25 nodes with around 6TB of total memory.
> > 
> > Thank you.
> > 
> 