<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On 24 Jun 2018, at 6:15 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" class="">mfadams@lbl.gov</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">First, <span style="text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline" class="">-pc_gamg_threshold 0.8<span class=""> is crazy. 0.08 is very high. probably best to start with 0.</span></span></div></div></blockquote><div><br class=""></div><div>I know it’s high, but that is what works for my application (otherwise the setup takes forever for just one or two iterations less).</div><div>The problem is related to my previous email from Friday ([petsc-dev] GAMG and custom MatMults in smoothers, thanks Barry for the thorough answer), I basically need only two or three levels (extremely aggressive coarsening).</div><div><br class=""></div><div>I don’t know if this can help, but depending on the value I use for the threshold, I also get this error:</div><div>ksp/pc/impls/gamg/agg.c: if (flid < 0) SETERRQ(PETSC_COMM_SELF,PETSC_ERR_PLIB,"Cannot find gid1 in table”);</div><div><br class=""></div><div>Of course, if I switch to an even crazier threshold, e.g., 0.999999, and end-up with a one-level method, I don’t see such an error.</div><div><br class=""></div><div>I know I am to blame here but my application does not use PetscInt as default integers (plain int instead). I use a lot of MatMPIAIJSetPreallocationCSR, so it is not as easy as simply recompiling with --with-64-bit-indices for me.</div><div>I’d gladly try anything that you suggest which would still use 32 bit indices as "primary PetscInt type".</div><div><br class=""></div><div>Thanks,</div><div>Pierre</div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class=""><span style="text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline" class=""><span class="">This threshold will delete all of your edges in the coarsening graph, probably, and GAMG will switch to a one level solve (silently although -info will tell you).</span></span></div><div class=""><span style="text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline" class=""><span class=""><br class=""></span></span></div><div class=""><span style="text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline" class=""><span class="">And yes, this looks like a 64 bit indices problem, we have:</span></span></div><div class=""><span style="text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline" class=""><span class=""><br class=""></span></span></div><div class=""><span style="text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline" class=""><span class="">#define GAMG_HASH(key) ((((PetscInt)7)*key)%a_tab->size)<br class=""></span></span></div><div class=""><br class=""></div><div class="">We should probably cast everything to (long long) or (unsigned PetscInt) I guess ... I'm not sure how this should be done exactly. Any ideas from the peanut gallery?</div><div class=""><br class=""></div><div class=""><span style="text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline" class=""><span class=""><br class=""></span></span></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="">On Sun, Jun 24, 2018 at 2:34 AM Pierre Jolivet <<a href="mailto:pierre.jolivet@enseeiht.fr" class="">pierre.jolivet@enseeiht.fr</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello,<br class="">
Any ideas on how to debug this?<br class="">
[406]PETSC ERROR: key 328214936 not found in table<br class="">
[406]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank" class="">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.<br class="">
[406]PETSC ERROR: Petsc Development GIT revision: v3.9.2-603-gceafe64 GIT Date: 2018-06-10 12:46:16 -0500<br class="">
[406]PETSC ERROR: /ccc/work/cont003/rndm/rndm/FreeFem-sources/src/mpi/FreeFem++-mpi on a arch-linux2-c-debug-bullxmpi named curie2698 by jolivetp Sun Jun 24 07:06:07 2018<br class="">
[406]PETSC ERROR: Configure options --with-scalar-type=real --with-hypre-dir=arch-linux2-c-opt-bullxmpi --with-metis-dir=arch-linux2-c-opt-bullxmpi --with-mumps-dir=arch-linux2-c-opt-bullxmpi --with-mkl_cpardiso=1 --with-mkl_pardiso=1 --with-parmetis-dir=arch-linux2-c-opt-bullxmpi --with-ptscotch-dir=arch-linux2-c-opt-bullxmpi --with-sowing-dir=arch-linux2-c-opt-bullxmpi --with-blacs-include=/ccc/products/mkl-18.0.1.163/default/<a href="http://18.0.1.163/mkl/include" rel="noreferrer" target="_blank" class="">18.0.1.163/mkl/include</a> --with-blacs-lib=/ccc/products/mkl-18.0.1.163/default/<a href="http://18.0.1.163/mkl/lib/intel64/libmkl_blacs_openmpi_lp64.so" rel="noreferrer" target="_blank" class="">18.0.1.163/mkl/lib/intel64/libmkl_blacs_openmpi_lp64.so</a> --with-blaslapack-dir=/ccc/products/mkl-18.0.1.163/default/<a href="http://18.0.1.163/mkl/lib/intel64" rel="noreferrer" target="_blank" class="">18.0.1.163/mkl/lib/intel64</a> --with-x=0 --with-debugging=1 --with-errorchecking=1 --with-mkl_pardiso-dir=/ccc/products/mkl-18.0.1.163/default/<a href="http://18.0.1.163/mkl" rel="noreferrer" target="_blank" class="">18.0.1.163/mkl</a> --with-mkl_cpardiso-dir=/ccc/products/mkl-18.0.1.163/default/<a href="http://18.0.1.163/mkl" rel="noreferrer" target="_blank" class="">18.0.1.163/mkl</a> --with-scalapack-include=/ccc/products/mkl-18.0.1.163/default/<a href="http://18.0.1.163/mkl/include" rel="noreferrer" target="_blank" class="">18.0.1.163/mkl/include</a> --with-scalapack-lib="[/ccc/products/mkl-18.0.1.163/default/<a href="http://18.0.1.163/mkl/lib/intel64/libmkl_scalapack_lp64.so,/ccc/products/mkl-18.0.1.163/default/18.0.1.163/mkl/lib/intel64/libmkl_blacs_openmpi_lp64.so" rel="noreferrer" target="_blank" class="">18.0.1.163/mkl/lib/intel64/libmkl_scalapack_lp64.so,/ccc/products/mkl-18.0.1.163/default/18.0.1.163/mkl/lib/intel64/libmkl_blacs_openmpi_lp64.so</a>]" --with-fortran-bindings=0 PETSC_ARCH=arch-linux2-c-debug-bullxmpi<br class="">
[406]PETSC ERROR: #1 PCGAMGHashTableFind() line 106 in /ccc/work/cont003/rndm/rndm/petsc/include/petsc/private/pcgamgimpl.h<br class="">
[406]PETSC ERROR: #2 smoothAggs() line 526 in /ccc/work/cont003/rndm/rndm/petsc/src/ksp/pc/impls/gamg/agg.c<br class="">
[406]PETSC ERROR: #3 PCGAMGCoarsen_AGG() line 970 in /ccc/work/cont003/rndm/rndm/petsc/src/ksp/pc/impls/gamg/agg.c<br class="">
[406]PETSC ERROR: #4 PCSetUp_GAMG() line 518 in /ccc/work/cont003/rndm/rndm/petsc/src/ksp/pc/impls/gamg/gamg.c<br class="">
[406]PETSC ERROR: #5 PCSetUp() line 932 in /ccc/work/cont003/rndm/rndm/petsc/src/ksp/pc/interface/precon.c<br class="">
[406]PETSC ERROR: #6 KSPSetUp() line 381 in /ccc/work/cont003/rndm/rndm/petsc/src/ksp/ksp/interface/itfunc.c<br class="">
<br class="">
For smaller problems, this works fine. Should I compile using 64 bit indices?<br class="">
I’m using -pc_type gamg -pc_gamg_threshold 0.8 -pc_gamg_agg_nsmooths 1 -pc_gamg_sym_graph true -pc_gamg_square_graph 10<br class="">
<br class="">
Thanks,<br class="">
Pierre</blockquote></div>
</div></blockquote></div><br class=""></body></html>