[petsc-users] Suggestions for solver and pc

[Sal Am]

Thank you Barry for quick response, I tried that, but I get several errors
of which the first one is:

[230]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[230]PETSC ERROR: Out of memory. This could be due to allocating
[230]PETSC ERROR: too large an object or bleeding by not properly
[230]PETSC ERROR: destroying unneeded objects.
[230]PETSC ERROR: Memory allocated 0 Memory used by process 5116485632
[230]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info.
[230]PETSC ERROR: Memory requested 186645063808
[230]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
for trouble shooting.
[230]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[230]PETSC ERROR: ./solveCSys on a linux-cumulus-x64 named r03n04 by vef002
Fri May 17 04:23:38 2019
[230]PETSC ERROR: Configure options PETSC_ARCH=linux-cumulus-x64
--with-cc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicc
--with-fc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpifort
--with-cxx=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicxx
--download-parmetis --download-metis --download-ptscotch
--download-superlu_dist --with-64-bit-indices --with-scalar-type=complex
--with-debugging=no --download-scalapack --download-fblaslapack=1
--download-cmake
[230]PETSC ERROR: #1 PetscFreeSpaceGet() line 11 in
/lustre/home/vef002/petsc/src/mat/utils/freespace.c
[230]PETSC ERROR: #2 PetscMallocA() line 390 in
/lustre/home/vef002/petsc/src/sys/memory/mal.c
[230]PETSC ERROR: #3 PetscFreeSpaceGet() line 11 in
/lustre/home/vef002/petsc/src/mat/utils/freespace.c
[230]PETSC ERROR: #4 MatLUFactorSymbolic_SeqAIJ() line 349 in
/lustre/home/vef002/petsc/src/mat/impls/aij/seq/aijfact.c
[230]PETSC ERROR: #5 MatLUFactorSymbolic() line 3015 in
/lustre/home/vef002/petsc/src/mat/interface/matrix.c
[230]PETSC ERROR: #6 PCSetUp_LU() line 95 in
/lustre/home/vef002/petsc/src/ksp/pc/impls/factor/lu/lu.c
[230]PETSC ERROR: #7 PCSetUp() line 932 in
/lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
[230]PETSC ERROR: #8 KSPSetUp() line 391 in
/lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
[230]PETSC ERROR: #9 PCSetUpOnBlocks_ASM() line 450 in
/lustre/home/vef002/petsc/src/ksp/pc/impls/asm/asm.c
[230]PETSC ERROR: #10 PCSetUpOnBlocks() line 963 in
/lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
[230]PETSC ERROR: #11 KSPSetUpOnBlocks() line 223 in
/lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
[230]PETSC ERROR: #12 KSPSolve() line 724 in
/lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c

So apparently ~6TB is not enough for the suggested routine...

On Fri, May 17, 2019 at 9:45 AM Smith, Barry F. <bsmith at mcs.anl.gov> wrote:

>
> -ksp_type gmres -ksp_gmres_restart 200  -pc_type asm -sub_pc_type lu
> -pc_asm_overlap 3 -ksp_monitor
>
> It will run like molasses but may converge
>
> Good luck
>
>
> > On May 17, 2019, at 3:40 AM, Sal Am via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
> >
> > Hello,
> >
> > So I am trying to solve this problem in the frequency domain by solving
> a Ax=b. It is apparently very ill conditioned. RF wave plasma interaction
> simulation. There are 6M finite elements the matrix is of the size:
> >
> > nnx: 1257303210 (1B non-zero elements)
> > n: 20347817 (size of matrix so 20M x 20M)
> >
> > What I have tried so far:
> >
> >  -ksp_type bcgs -pc_type gamg
> >
> >  -ksp_type gmres -ksp_gmres_restart 150 -pc_type bjacobi -sub_pc_type
> ilu -sub_pc_factor_levels 5 -sub_ksp_type bcgs -mattransposematmult_via
> scalable -build_twosided allreduce
> >
> > -ksp_type bcgs -pc_type gamg -mattransposematmult_via scalable
> -build_twosided allreduce
> >
> > -ksp_type bcgs -pc_type asm -sub_pc_type lu
> >
> > None of the above has really shown any convergence after 2 days of
> running the simulation. The farthest I got was using gmres + bjacobi which
> gave me a ||r||/||b|| of the order 1e-2 but it got stuck between 1e-2 and
> 1e-1 after a week of having left it running.
> >
> > What I have available in terms of computational resources:
> > select=25:ncpus=16:mpiprocs=16:mem=230GB
> >
> > So 25 nodes with around 6TB of total memory.
> >
> > Thank you.
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190517/bc592128/attachment.html>