[petsc-users] Weak scaling test with geometric multigrid

[Dave May]

Henrik,


On Wed, 10 Jan 2018 at 16:39, Smith, Barry F. <bsmith at mcs.anl.gov> wrote:

>
>    DMDA requires that there be at least 1 grid points in each direction on
> each process (this simplified the implementation a huge amount but gives up
> flexibility). In your case you have 16 processes in a particular direction
> but a total of only 12 grid points hence not enough grid points to have at
> least 1 per process.


This particular implementation limitation can be overcome using
PCTELESCOPE. It allows you to repartition the coarse levels onto fewer
ranks.

Cheers,
  Dave



>
>   Barry
>
>
> > On Jan 10, 2018, at 5:25 AM, Buesing, Henrik <
> hbuesing at eonerc.rwth-aachen.de> wrote:
> >
> > Dear all,
> >
> > I am doing a weak scaling test using geometric multigrid. With
> increasing the number of cells, and the number of processes, I also
> increase the number of multigrid levels. With 64 cores, 12288 cells in
> x-direction and 11 multigrid levels, I see error message [1].
> >
> > Could you help me understand what is happening here?
> >
> > The characteristics of the weak scaling test are summarized in table
> [2]. Refinement level 7-9 went through fine.
> >
> > Thank you!
> > Henrik
> >
> > [1]
> >
> > [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> > [0]PETSC ERROR: Argument out of range
> > [0]PETSC ERROR: Partition in x direction is too fine! 12 16
> > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> > [0]PETSC ERROR: Petsc Development GIT revision: v3.8.2-48-g851ec02  GIT
> Date: 2017-12-05 09:52:17 -0600
> > [0]PETSC ERROR: shem_fw64gnu_const.x on a gnu_openmpi named
> linuxihfc033.rz.RWTH-Aachen.DE by hb111949 Wed Jan 10 11:48:09 2018
> > [0]PETSC ERROR: Configure options --download-fblaslapack --with-cc=mpicc
> -with-fc=mpif90 --with-cxx=mpicxx --download-hypre --download-superlu_dist
> --download-suitesparse --download-scalapack --download-blacs
> --download-hdf5 --download-parmetis --download-metis --with-debugging=0
> --download-mumps
> > [0]PETSC ERROR: #1 DMSetUp_DA_3D() line 299 in
> /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/impls/da/da3.c
> > [0]PETSC ERROR: #2 DMSetUp_DA() line 25 in
> /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/impls/da/dareg.c
> > [0]PETSC ERROR: #3 DMSetUp() line 720 in
> /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/interface/dm.c
> > [0]PETSC ERROR: #4 DMCoarsen_DA() line 1203 in
> /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/impls/da/da.c
> > [0]PETSC ERROR: #5 DMCoarsen() line 2427 in
> /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/interface/dm.c
> > [0]PETSC ERROR: #6 PCSetUp_MG() line 618 in
> /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/pc/impls/mg/mg.c
> > [0]PETSC ERROR: #7 PCSetUp() line 924 in
> /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/pc/interface/precon.c
> > [0]PETSC ERROR: #8 KSPSetUp() line 381 in
> /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/ksp/interface/itfunc.c
> > [0]PETSC ERROR: #9 KSPSolve() line 612 in
> /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/ksp/interface/itfunc.c
> > [0]PETSC ERROR: #10 SNESSolve_NEWTONLS() line 224 in
> /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/snes/impls/ls/ls.c
> > [0]PETSC ERROR: #11 SNESSolve() line 4179 in
> /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/snes/interface/snes.c
> >
> >
> > [2]
> >
> > # refinement level
> > # cores
> > # cells in x
> > # cells in y
> > # cells in z
> > # mg levels
> > 7
> > 1
> > 1536
> > 1
> > 256
> > 8
> > 8
> > 4
> > 3072
> > 1
> > 512
> > 9
> > 9
> > 16
> > 6144
> > 1
> > 1024
> > 10
> > 10
> > 64
> > 12288
> > 1
> > 2048
> > 11
> >
> > --
> > Dipl.-Math. Henrik Büsing
> > Institute for Applied Geoph
> <https://maps.google.com/?q=Applied+Geoph&entry=gmail&source=g>ysics and
> Geothermal Energy
> > E.ON Energy Research Center
> > RWTH Aachen University
> > ------------------------------------------------------
> > Mathieustr. 10            |    Tel +49 (0)241 80 49907
> > 52074 Aachen, Germany     |    Fax +49 (0)241 80 49889
> > ------------------------------------------------------
> > http://www.eonerc.rwth-aachen.de/GGE
> > hbuesing at eonerc.rwth-aachen.de
> > ------------------------------------------------------
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20180111/bc196f0d/attachment.html>