<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
Mark,<br>
The problem is a 2D plane elasticity problem, definitely SPD.<br>
If I add -pc_gamg_esteig_ksp_type cg, it works perfectly. <br>
<br>
I also tried adding -info but that caused some other type of
weird crash; the options I used were<br>
<blockquote>-ksp_type cg -ksp_monitor -log_view -pc_type gamg
-pc_gamg_type agg -pc_gamg_agg_nsmooths 1
-pc_gamg_esteig_ksp_type cg -info -options_left<br>
<br>
</blockquote>
The resulting error was:<br>
<br>
<blockquote>[0] petscinitialize_internal(): (Fortran):PETSc
successfully started: procs 1<br>
[0] petscinitialize_internal(): Running on machine:
vulture.ce.berkeley.edu<br>
--> ERRORS OCCURRED: For details see file: Ofo<br>
<br>
[0] PetscFinalize(): PetscFinalize() called<br>
[0] PetscCommDuplicate(): Duplicating a communicator
140367846615616 28408096 max tags = 2147483647<br>
************************************************************************************************************************<br>
*** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use
'enscript -r -fCourier9' to print this document ***<br>
************************************************************************************************************************<br>
<br>
---------------------------------------------- PETSc Performance
Summary: ----------------------------------------------<br>
<br>
<br>
<br>
##########################################################<br>
# #<br>
# WARNING!!! #<br>
# #<br>
# This code was compiled with a debugging option. #<br>
# To get timing results run ./configure #<br>
# using --with-debugging=no, the performance will #<br>
# be generally two or three times faster. #<br>
# #<br>
##########################################################<br>
<br>
<br>
/home/sg/Feap/ver86/parfeap/feap on a gnu named
vulture.ce.berkeley.edu with 1 processor, by sg Sat Jun 13
12:47:06 2020<br>
Using Petsc Release Version 3.13.2, unknown <br>
<br>
Max Max/Min Avg Total<br>
Time (sec): 1.516e-03 1.000 1.516e-03<br>
Objects: 1.000e+00 1.000 1.000e+00<br>
Flop: 0.000e+00 0.000 0.000e+00 0.000e+00<br>
Flop/sec: 0.000e+00 0.000 0.000e+00 0.000e+00<br>
Memory: 5.122e+04 1.000 5.122e+04 5.122e+04<br>
MPI Messages: 0.000e+00 0.000 0.000e+00 0.000e+00<br>
MPI Message Lengths: 0.000e+00 0.000 0.000e+00 0.000e+00<br>
MPI Reductions: 0.000e+00 0.000<br>
<br>
Flop counting convention: 1 flop = 1 real number operation of type
(multiply/divide/add/subtract)<br>
e.g., VecAXPY() for real vectors of
length N --> 2N flop<br>
and VecAXPY() for complex vectors of
length N --> 8N flop<br>
<br>
Summary of Stages: ----- Time ------ ----- Flop ------ ---
Messages --- -- Message Lengths -- -- Reductions --<br>
Avg %Total Avg %Total
Count %Total Avg %Total Count %Total<br>
0: Main Stage: 1.5061e-03 99.3% 0.0000e+00 0.0%
0.000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0%<br>
<br>
------------------------------------------------------------------------------------------------------------------------<br>
See the 'Profiling' chapter of the users' manual for details on
interpreting output.<br>
Phase summary info:<br>
Count: number of times phase was executed<br>
Time and Flop: Max - maximum over all processors<br>
Ratio - ratio of maximum to minimum over all
processors<br>
Mess: number of messages sent<br>
AvgLen: average message length (bytes)<br>
Reduct: number of global reductions<br>
Global: entire computation<br>
Stage: stages of a computation. Set stages with
PetscLogStagePush() and PetscLogStagePop().<br>
%T - percent time in this phase %F - percent flop in
this phase<br>
%M - percent messages in this phase %L - percent message
lengths in this phase<br>
%R - percent reductions in this phase<br>
Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max
time over all processors)<br>
------------------------------------------------------------------------------------------------------------------------<br>
<br>
<br>
##########################################################<br>
# #<br>
# WARNING!!! #<br>
# #<br>
# This code was compiled with a debugging option. #<br>
# To get timing results run ./configure #<br>
# using --with-debugging=no, the performance will #<br>
# be generally two or three times faster. #<br>
# #<br>
##########################################################<br>
<br>
<br>
Event Count Time (sec)
Flop --- Global --- --- Stage ----
Total<br>
Max Ratio Max Ratio Max Ratio Mess
AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s<br>
------------------------------------------------------------------------------------------------------------------------<br>
<br>
--- Event Stage 0: Main Stage<br>
<br>
------------------------------------------------------------------------------------------------------------------------<br>
<br>
Memory usage is given in bytes:<br>
<br>
Object Type Creations Destructions Memory
Descendants' Mem.<br>
Reports information only for process 0.<br>
<br>
--- Event Stage 0: Main Stage<br>
<br>
Viewer 1 0 0 0.<br>
========================================================================================================================<br>
Average time to get PetscTime(): 4.38071e-08<br>
#PETSc Option Table entries:<br>
-info<br>
-ksp_monitor<br>
-ksp_type cg<br>
-log_view<br>
-options_left<br>
-pc_gamg_agg_nsmooths 1<br>
-pc_gamg_esteig_ksp_type cg<br>
-pc_gamg_type agg<br>
-pc_type gamg<br>
#End of PETSc Option Table entries<br>
Compiled without FORTRAN kernels<br>
Compiled with full precision matrices (default)<br>
sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8
sizeof(PetscScalar) 8 sizeof(PetscInt) 4<br>
Configure options: --download-parmetis --download-superlu_dist
--download-openmpi --download-ml --download-hypre --download-metis
--download-mumps --download-scalapack --download-blacs<br>
-----------------------------------------<br>
Libraries compiled on 2020-06-13 05:13:15 on
vulture.ce.berkeley.edu <br>
Machine characteristics:
Linux-5.6.13-100.fc30.x86_64-x86_64-with-fedora-30-Thirty<br>
Using PETSc directory: /home/sg/petsc-3.13.2<br>
Using PETSc arch: gnu<br>
-----------------------------------------<br>
<br>
Using C compiler: /home/sg/petsc-3.13.2/gnu/bin/mpicc -fPIC -Wall
-Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas
-fstack-protector -fvisibility=hidden -g3 <br>
Using Fortran compiler: /home/sg/petsc-3.13.2/gnu/bin/mpif90
-fPIC -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -g <br>
-----------------------------------------<br>
<br>
Using include paths: -I/home/sg/petsc-3.13.2/include
-I/home/sg/petsc-3.13.2/gnu/include<br>
-----------------------------------------<br>
<br>
Using C linker: /home/sg/petsc-3.13.2/gnu/bin/mpicc<br>
Using Fortran linker: /home/sg/petsc-3.13.2/gnu/bin/mpif90<br>
Using libraries: -Wl,-rpath,/home/sg/petsc-3.13.2/gnu/lib
-L/home/sg/petsc-3.13.2/gnu/lib -lpetsc
-Wl,-rpath,/home/sg/petsc-3.13.2/gnu/lib
-L/home/sg/petsc-3.13.2/gnu/lib
-Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/9
-L/usr/lib/gcc/x86_64-redhat-linux/9 -lHYPRE -lcmumps -ldmumps
-lsmumps -lzmumps -lmumps_common -lpord -lscalapack -lsuperlu_dist
-lml -llapack -lblas -lX11 -lparmetis -lmetis -lm -lstdc++ -ldl
-lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi
-lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lpthread
-lstdc++ -ldl<br>
-----------------------------------------<br>
<br>
<br>
<br>
##########################################################<br>
# #<br>
# WARNING!!! #<br>
# #<br>
# This code was compiled with a debugging option. #<br>
# To get timing results run ./configure #<br>
# using --with-debugging=no, the performance will #<br>
# be generally two or three times faster. #<br>
# #<br>
##########################################################<br>
<br>
<br>
[0] Petsc_DelViewer(): Removing viewer data attribute in an
MPI_Comm 28408096<br>
[0] Petsc_OuterComm_Attr_Delete_Fn(): Removing reference to PETSc
communicator embedded in a user MPI_Comm 28408096<br>
[0] Petsc_InnerComm_Attr_Delete_Fn(): User MPI_Comm
140367846615616 is being unlinked from inner PETSc comm 28408096<br>
[0] PetscCommDestroy(): Deleting PETSc MPI_Comm 28408096<br>
[0] Petsc_Counter_Attr_Delete_Fn(): Deleting counter data in an
MPI_Comm 28408096<br>
[0] PetscCommDuplicate(): Duplicating a communicator
140367846615616 28408096 max tags = 2147483647<br>
#PETSc Option Table entries:<br>
-info<br>
-ksp_monitor<br>
-ksp_type cg<br>
-log_view<br>
-options_left<br>
-pc_gamg_agg_nsmooths 1<br>
-pc_gamg_esteig_ksp_type cg<br>
-pc_gamg_type agg<br>
-pc_type gamg<br>
#End of PETSc Option Table entries<br>
[0] Petsc_OuterComm_Attr_Delete_Fn(): Removing reference to PETSc
communicator embedded in a user MPI_Comm 28408096<br>
[0] Petsc_InnerComm_Attr_Delete_Fn(): User MPI_Comm
140367846615616 is being unlinked from inner PETSc comm 28408096<br>
[0] PetscCommDestroy(): Deleting PETSc MPI_Comm 28408096<br>
[0] Petsc_Counter_Attr_Delete_Fn(): Deleting counter data in an
MPI_Comm 28408096<br>
WARNING! There are options you set that were not used!<br>
WARNING! could be spelling mistake, etc!<br>
There are 6 unused database options. They are:<br>
Option left: name:-ksp_monitor (no value)<br>
Option left: name:-ksp_type value: cg<br>
Option left: name:-pc_gamg_agg_nsmooths value: 1<br>
Option left: name:-pc_gamg_esteig_ksp_type value: cg<br>
Option left: name:-pc_gamg_type value: agg<br>
Option left: name:-pc_type value: gamg<br>
[sg@vulture main]$ <br>
<br>
</blockquote>
<blockquote><br>
</blockquote>
<br>
<pre class="moz-signature" cols="72">
</pre>
<div class="moz-cite-prefix">On 6/13/20 5:10 AM, Mark Adams wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CADOhEh4ECmKvnYG6C+pH3HTFT_M1=xa+1RK3De-LM4D7Tgonmw@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">That is odd. Are these problems
symmetric positive definite?
<div><br>
</div>
<div>Eigen estimates are a pain in practice but I've never seen
this. Hypre has (better) smoothers that don't need this and
the AMG algorithm does not need them either. I think ML does
pretty much the same thing as me.
<div><br>
</div>
<div>If SPD then you definitely want '-pc_gamg_esteig_ksp_type
cg'. CG converges faster and is more robust. Even if not
strictly SPD like with large deformation or plasticity CG is
probably better. You can run with -info and grep on GAMG to
see what eigen (high) estimates it came up with for each
level. They should be >1 and < 4, or so.</div>
<div><br>
</div>
<div>I wonder if your LAPACK is funky. You might try a fresh
build (delete the "arch" directory) and download LAPACK, but
this problem will go away with CG.</div>
<div><br>
</div>
<div>Note, if you use Chebyshev smoothing then it needs eigen
estimates also. THis is usually where eigen estimate
problems come up. If you use jacobi PC in the smoother, GAMG
will use the eigen estimate from the smoothed aggregation
smoothing (what is failing for you here).
-pc_gamg_use_sa_esteig <T,F> will force it to use, or
not use, it. For instance, if you sor/ilu/asm then the eigen
estimates are probably fine to reuse but GAMG does not by
default. If the cheby eigen estimates are low then the
solver fails. THere is a safety factor to increase the eigen
estimate in chychev to avoid this failure.</div>
<div><br>
</div>
<div>And, -pc_gamg_esteig_ksp_max_it X sets the number of
iterations in the estimator. You can increase this and you
should see your eigen estimates go up and converge. You
should see that CG converges much faster than GMRES (the
default).</div>
<div><br>
</div>
<div><br>
<div><br>
</div>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Sat, Jun 13, 2020 at 2:32
AM Sanjay Govindjee <<a href="mailto:s_g@berkeley.edu"
moz-do-not-send="true">s_g@berkeley.edu</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div> I have a FEA problem that I am trying to solve with
GAMG. The problem solves<br>
just fine with direct solvers (mumps, superlu) and iterative
solvers (gmres, ml, hypre-boomer) etc.<br>
<br>
However with GAMG I am getting a divide by zero that I am
having trouble tracking down. Below<br>
is the gdb stack trace and the source lines going up the
stack. <br>
<br>
When I run in valgrind the problem runs fine (and gets the
correct answer).<br>
Valgrind reports nothing of note (just lots of indirectly
lost blocks related to PMP_INIT).<br>
<br>
I'm only running on one processor.<br>
<br>
Any suggestions on where to start to trace the problem?<br>
<br>
-sanjay<br>
<br>
<blockquote>#0 0x00007fb262dc5be1 in ieeeck_ () from
/lib64/liblapack.so.3<br>
#1 0x00007fb262dc5332 in ilaenv_ () from
/lib64/liblapack.so.3<br>
#2 0x00007fb262dbbcef in dlasq2_ () from
/lib64/liblapack.so.3<br>
#3 0x00007fb262dbb78c in dlasq1_ () from
/lib64/liblapack.so.3<br>
#4 0x00007fb262da1e2e in dbdsqr_ () from
/lib64/liblapack.so.3<br>
#5 0x00007fb262960110 in dgesvd_ () from
/lib64/liblapack.so.3<br>
#6 0x00007fb264e74b66 in
KSPComputeExtremeSingularValues_GMRES (ksp=0x1816560,
emax=0x7ffc5010e7c8, emin=0x7ffc5010e7d0) at
/home/sg/petsc-3.13.2/src/ksp/ksp/impls/gmres/gmreig.c:32<br>
#7 0x00007fb264dfe69a in KSPComputeExtremeSingularValues
(ksp=0x1816560, emax=0x7ffc5010e7c8, emin=0x7ffc5010e7d0)
at /home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:64<br>
#8 0x00007fb264b44a1f in PCGAMGOptProlongator_AGG
(pc=0x12f3d30, Amat=0x11a2630, a_P=0x7ffc5010ebe0) at
/home/sg/petsc-3.13.2/src/ksp/pc/impls/gamg/agg.c:1145<br>
#9 0x00007fb264b248a1 in PCSetUp_GAMG (pc=0x12f3d30) at
/home/sg/petsc-3.13.2/src/ksp/pc/impls/gamg/gamg.c:557<br>
#10 0x00007fb264d8535b in PCSetUp (pc=0x12f3d30) at
/home/sg/petsc-3.13.2/src/ksp/pc/interface/precon.c:898<br>
#11 0x00007fb264e01a93 in KSPSetUp (ksp=0x128dd80) at
/home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:376<br>
#12 0x00007fb264e057af in KSPSolve_Private (ksp=0x128dd80,
b=0x1259f30, x=0x125d910) at
/home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:633<br>
#13 0x00007fb264e086b9 in KSPSolve (ksp=0x128dd80,
b=0x1259f30, x=0x125d910) at
/home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:853<br>
#14 0x00007fb264e46216 in kspsolve_ (ksp=0x832670
<__pfeapc_MOD_kspsol>, b=0x832698
<__pfeapc_MOD_rhs>, x=0x8326a0
<__pfeapc_MOD_sol>, __ierr=0x7ffc5010f358)<br>
at
/home/sg/petsc-3.13.2/src/ksp/ksp/interface/ftn-auto/itfuncf.c:266<br>
#15 0x000000000043298d in usolve (flags=..., b=...) at
usolve.F:313<br>
#16 0x000000000044afba in psolve (stype=-3, b=..., fp=...,
factor=.TRUE., solve=.TRUE., cfr=.FALSE., prnt=.TRUE.) at
psolve.f:212<br>
#17 0x00000000006b7393 in pmacr1 (lct=..., ct=..., j=3,
_lct=_lct@entry=15) at pmacr1.f:578<br>
#18 0x00000000005c247b in pmacr (initf=.FALSE.) at
pmacr.f:578<br>
#19 0x000000000044ff20 in pcontr () at pcontr.f:1307<br>
#20 0x0000000000404d9b in feap () at feap86.f:162<br>
#21 main (argc=<optimized out>, argv=<optimized
out>) at feap86.f:168<br>
#22 0x00007fb261aaef43 in __libc_start_main () from
/lib64/libc.so.6<br>
#23 0x0000000000404dde in _start ()<br>
<br>
(gdb) list<br>
1 <built-in>: No such file or directory.<br>
(gdb) up<br>
#1 0x00007fb262dc5332 in ilaenv_ () from
/lib64/liblapack.so.3<br>
(gdb) up<br>
#2 0x00007fb262dbbcef in dlasq2_ () from
/lib64/liblapack.so.3<br>
(gdb) up<br>
#3 0x00007fb262dbb78c in dlasq1_ () from
/lib64/liblapack.so.3<br>
(gdb) up<br>
#4 0x00007fb262da1e2e in dbdsqr_ () from
/lib64/liblapack.so.3<br>
(gdb) up<br>
#5 0x00007fb262960110 in dgesvd_ () from
/lib64/liblapack.so.3<br>
(gdb) up<br>
#6 0x00007fb264e74b66 in
KSPComputeExtremeSingularValues_GMRES (ksp=0x1816560,
emax=0x7ffc5010e7c8, emin=0x7ffc5010e7d0) at
/home/sg/petsc-3.13.2/src/ksp/ksp/impls/gmres/gmreig.c:32<br>
32
PetscStackCallBLAS("LAPACKgesvd",LAPACKgesvd_("N","N",&bn,&bn,R,&bN,realpart,&sdummy,&idummy,&sdummy,&idummy,work,&lwork,&lierr));<br>
(gdb) up<br>
#7 0x00007fb264dfe69a in KSPComputeExtremeSingularValues
(ksp=0x1816560, emax=0x7ffc5010e7c8, emin=0x7ffc5010e7d0)
at /home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:64<br>
64 ierr =
(*ksp->ops->computeextremesingularvalues)(ksp,emax,emin);CHKERRQ(ierr);<br>
(gdb) up<br>
#8 0x00007fb264b44a1f in PCGAMGOptProlongator_AGG
(pc=0x12f3d30, Amat=0x11a2630, a_P=0x7ffc5010ebe0) at
/home/sg/petsc-3.13.2/src/ksp/pc/impls/gamg/agg.c:1145<br>
1145 ierr = KSPComputeExtremeSingularValues(eksp,
&emax, &emin);CHKERRQ(ierr);<br>
(gdb) up<br>
#9 0x00007fb264b248a1 in PCSetUp_GAMG (pc=0x12f3d30) at
/home/sg/petsc-3.13.2/src/ksp/pc/impls/gamg/gamg.c:557<br>
557 ierr =
pc_gamg->ops->optprolongator(pc, Aarr[level],
&Prol11);CHKERRQ(ierr);<br>
(gdb) up<br>
#10 0x00007fb264d8535b in PCSetUp (pc=0x12f3d30) at
/home/sg/petsc-3.13.2/src/ksp/pc/interface/precon.c:898<br>
898 ierr =
(*pc->ops->setup)(pc);CHKERRQ(ierr);<br>
(gdb) up<br>
#11 0x00007fb264e01a93 in KSPSetUp (ksp=0x128dd80) at
/home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:376<br>
376 ierr = PCSetUp(ksp->pc);CHKERRQ(ierr);<br>
(gdb) up<br>
#12 0x00007fb264e057af in KSPSolve_Private (ksp=0x128dd80,
b=0x1259f30, x=0x125d910) at
/home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:633<br>
633 ierr = KSPSetUp(ksp);CHKERRQ(ierr);<br>
(gdb) up<br>
#13 0x00007fb264e086b9 in KSPSolve (ksp=0x128dd80,
b=0x1259f30, x=0x125d910) at
/home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:853<br>
853 ierr = KSPSolve_Private(ksp,b,x);CHKERRQ(ierr);<br>
(gdb) up<br>
#14 0x00007fb264e46216 in kspsolve_ (ksp=0x832670
<__pfeapc_MOD_kspsol>, b=0x832698
<__pfeapc_MOD_rhs>, x=0x8326a0
<__pfeapc_MOD_sol>, __ierr=0x7ffc5010f358)<br>
at
/home/sg/petsc-3.13.2/src/ksp/ksp/interface/ftn-auto/itfuncf.c:266<br>
266 *__ierr = KSPSolve(<br>
(gdb) up<br>
#15 0x000000000043298d in usolve (flags=..., b=...) at
usolve.F:313<br>
313 call KSPSolve (kspsol, rhs, sol,
ierr)<br>
</blockquote>
<br>
<br>
</div>
</blockquote>
</div>
</blockquote>
<br>
</body>
</html>