[petsc-users] Suggestions for solver and pc

[Mark Adams]

Are you shifting into high frequency? If not you can try tricks in this
paper. Using a large coarse grid in AMG that captures the frequency
response of interest and using a parallel direct solver is a good option if
it is doable (not too deep a shift).

@Article{Adams-03a,
  author =   {Adams M.F.},
  title =   {Algebraic multigrid techniques for strongly indefinite linear
systems from direct frequency response analysis in solid mechanics},
  journal =   {Computational Mechanics},
  year =   {2007},
  volume =   {39},
  number =   {4},
  pages =   {497-507}
}

On Fri, May 17, 2019 at 6:33 AM Sal Am via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Thank you Barry for quick response, I tried that, but I get several errors
> of which the first one is:
>
> [230]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [230]PETSC ERROR: Out of memory. This could be due to allocating
> [230]PETSC ERROR: too large an object or bleeding by not properly
> [230]PETSC ERROR: destroying unneeded objects.
> [230]PETSC ERROR: Memory allocated 0 Memory used by process 5116485632
> [230]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info.
> [230]PETSC ERROR: Memory requested 186645063808
> [230]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [230]PETSC ERROR: Petsc Release Version 3.10.2, unknown
> [230]PETSC ERROR: ./solveCSys on a linux-cumulus-x64 named r03n04 by
> vef002 Fri May 17 04:23:38 2019
> [230]PETSC ERROR: Configure options PETSC_ARCH=linux-cumulus-x64
> --with-cc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicc
> --with-fc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpifort
> --with-cxx=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicxx
> --download-parmetis --download-metis --download-ptscotch
> --download-superlu_dist --with-64-bit-indices --with-scalar-type=complex
> --with-debugging=no --download-scalapack --download-fblaslapack=1
> --download-cmake
> [230]PETSC ERROR: #1 PetscFreeSpaceGet() line 11 in
> /lustre/home/vef002/petsc/src/mat/utils/freespace.c
> [230]PETSC ERROR: #2 PetscMallocA() line 390 in
> /lustre/home/vef002/petsc/src/sys/memory/mal.c
> [230]PETSC ERROR: #3 PetscFreeSpaceGet() line 11 in
> /lustre/home/vef002/petsc/src/mat/utils/freespace.c
> [230]PETSC ERROR: #4 MatLUFactorSymbolic_SeqAIJ() line 349 in
> /lustre/home/vef002/petsc/src/mat/impls/aij/seq/aijfact.c
> [230]PETSC ERROR: #5 MatLUFactorSymbolic() line 3015 in
> /lustre/home/vef002/petsc/src/mat/interface/matrix.c
> [230]PETSC ERROR: #6 PCSetUp_LU() line 95 in
> /lustre/home/vef002/petsc/src/ksp/pc/impls/factor/lu/lu.c
> [230]PETSC ERROR: #7 PCSetUp() line 932 in
> /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
> [230]PETSC ERROR: #8 KSPSetUp() line 391 in
> /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
> [230]PETSC ERROR: #9 PCSetUpOnBlocks_ASM() line 450 in
> /lustre/home/vef002/petsc/src/ksp/pc/impls/asm/asm.c
> [230]PETSC ERROR: #10 PCSetUpOnBlocks() line 963 in
> /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
> [230]PETSC ERROR: #11 KSPSetUpOnBlocks() line 223 in
> /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
> [230]PETSC ERROR: #12 KSPSolve() line 724 in
> /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
>
> So apparently ~6TB is not enough for the suggested routine...
>
> On Fri, May 17, 2019 at 9:45 AM Smith, Barry F. <bsmith at mcs.anl.gov>
> wrote:
>
>>
>> -ksp_type gmres -ksp_gmres_restart 200  -pc_type asm -sub_pc_type lu
>> -pc_asm_overlap 3 -ksp_monitor
>>
>> It will run like molasses but may converge
>>
>> Good luck
>>
>>
>> > On May 17, 2019, at 3:40 AM, Sal Am via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>> >
>> > Hello,
>> >
>> > So I am trying to solve this problem in the frequency domain by solving
>> a Ax=b. It is apparently very ill conditioned. RF wave plasma interaction
>> simulation. There are 6M finite elements the matrix is of the size:
>> >
>> > nnx: 1257303210 (1B non-zero elements)
>> > n: 20347817 (size of matrix so 20M x 20M)
>> >
>> > What I have tried so far:
>> >
>> >  -ksp_type bcgs -pc_type gamg
>> >
>> >  -ksp_type gmres -ksp_gmres_restart 150 -pc_type bjacobi -sub_pc_type
>> ilu -sub_pc_factor_levels 5 -sub_ksp_type bcgs -mattransposematmult_via
>> scalable -build_twosided allreduce
>> >
>> > -ksp_type bcgs -pc_type gamg -mattransposematmult_via scalable
>> -build_twosided allreduce
>> >
>> > -ksp_type bcgs -pc_type asm -sub_pc_type lu
>> >
>> > None of the above has really shown any convergence after 2 days of
>> running the simulation. The farthest I got was using gmres + bjacobi which
>> gave me a ||r||/||b|| of the order 1e-2 but it got stuck between 1e-2 and
>> 1e-1 after a week of having left it running.
>> >
>> > What I have available in terms of computational resources:
>> > select=25:ncpus=16:mpiprocs=16:mem=230GB
>> >
>> > So 25 nodes with around 6TB of total memory.
>> >
>> > Thank you.
>> >
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190517/82f37d54/attachment-0001.html>