<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><br class=""></div> JT,<div class=""><br class=""></div><div class=""> These fixes will be merged into PETSc maint soon. <a href="https://gitlab.com/petsc/petsc/-/merge_requests/2986" class="">https://gitlab.com/petsc/petsc/-/merge_requests/2986</a></div><div class=""><br class=""></div><div class=""> Barry</div><div class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Jul 19, 2020, at 1:52 AM, Karl Yang <<a href="mailto:y.juntao@hotmail.com" class="">y.juntao@hotmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta http-equiv="Content-Type" content="text/html; charset=utf-8" class=""><div class="">Hi, Barry, </div><br class=""><div class="">Thanks for your reply. I see the issue and I think I can now continue my work with a quick fix as you suggested. </div><br class=""><div class="">Regards</div><div class="">JT</div><br class=""><div class="gmail_quote_attribution">On Jul 19 2020, at 1:26 pm, Barry Smith <<a href="mailto:bsmith@petsc.dev" class="">bsmith@petsc.dev</a>> wrote:</div><blockquote class=""><div class=""><br class=""><br class=""><div class="">> On Jul 18, 2020, at 11:27 PM, Karl Yang <<a href="mailto:y.juntao@hotmail.com" class="">y.juntao@hotmail.com</a>> wrote:</div><div class="">></div><div class="">></div><div class="">> Hello,</div><div class="">></div><div class="">> I was using FGMRES solver with fieldsplit preconditioner for solving a fluid equation.</div><div class="">> I have the code ran for coarser meshes and it works fine. But when I got it to run for very fine meshes I encountered the PetscLogFlops error.</div><div class="">></div><div class="">> The following is the output of error and the output from kspview.</div><div class="">></div><div class="">> [7]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</div><div class="">> [7]PETSC ERROR: Argument out of range</div><div class="">> [7]PETSC ERROR: Cannot log negative flops</div><div class="">> [7]PETSC ERROR: See <a href="https://www.mcs.anl.gov/petsc/documentation/faq.html" class="">https://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.</div><div class="">> [7]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020</div><div class="">> [7]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named hsw222 by yjuntao Sat Jul 18 20:30:05 2020</div><div class="">> [7]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda</div><div class="">> [7]PETSC ERROR: #1 PetscLogFlops() line 232 in /usr/local/petsc/petsc-3.12.5/include/petsclog.h</div><div class="">> [7]PETSC ERROR: #2 MatSolve_SeqAIJ() line 3200 in /usr/local/petsc/petsc-3.12.5/src/mat/impls/aij/seq/aijfact.c</div><div class="">> [7]PETSC ERROR: #3 MatSolve() line 3290 in /usr/local/petsc/petsc-3.12.5/src/mat/interface/matrix.c</div><div class="">> [7]PETSC ERROR: #4 PCApply_LU() line 177 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/factor/lu/lu.c</div><div class="">> [7]PETSC ERROR: #5 PCApply() line 444 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c</div><div class="">> [7]PETSC ERROR: #6 KSP_PCApply() line 281 in /usr/local/petsc/petsc-3.12.5/include/petsc/private/kspimpl.h</div><div class="">> [7]PETSC ERROR: #7 KSPInitialResidual() line 65 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itres.c</div><div class="">> [7]PETSC ERROR: #8 KSPSolve_GMRES() line 236 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/gmres.c</div><div class="">> [7]PETSC ERROR: #9 KSPSolve() line 760 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c</div><div class="">> [7]PETSC ERROR: #10 PCApply_FieldSplit_Schur() line 1180 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/fieldsplit/fieldsplit.c</div><div class="">> [7]PETSC ERROR: #11 PCApply() line 444 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c</div><div class="">> [7]PETSC ERROR: #12 KSP_PCApply() line 281 in /usr/local/petsc/petsc-3.12.5/include/petsc/private/kspimpl.h</div><div class="">> [7]PETSC ERROR: #13 KSPFGMRESCycle() line 166 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/fgmres/fgmres.c</div><div class="">> [7]PETSC ERROR: #14 KSPSolve_FGMRES() line 291 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/fgmres/fgmres.c</div><div class="">> [7]PETSC ERROR: #15 KSPSolve() line 760 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c</div><div class="">></div><br class=""><div class="">PetscErrorCode MatSolve_SeqAIJ(Mat A,Vec bb,Vec xx)</div><div class="">{</div><div class="">Mat_SeqAIJ *a = (Mat_SeqAIJ*)A->data;</div><div class="">IS iscol = a->col,isrow = a->row;</div><div class="">PetscErrorCode ierr;</div><div class="">PetscInt i,n=A->rmap->n,*vi,*ai=a->i,*aj=a->j,*adiag = a->diag,nz;</div><div class="">const PetscInt *rout,*cout,*r,*c;</div><div class="">PetscScalar *x,*tmp,sum;</div><div class="">const PetscScalar *b;</div><div class="">const MatScalar *aa = a->a,*v;</div><br class=""><div class="">PetscFunctionBegin;</div><div class="">if (!n) PetscFunctionReturn(0);</div><br class=""><div class="">......</div><br class=""><div class="">ierr = PetscLogFlops(2*a->nz - A->cmap->n);CHKERRQ(ierr);</div><div class="">PetscFunctionReturn(0);</div><br class=""><div class="">Could maybe be integer overflow where a->nz fits but 2*a->nz doesn't fit so it results in a negative integer that is then converted to a negative floating point number.</div><br class=""><div class="">It should have a 2.0*a->nz but we must have missed this place in the code. We can fix this.</div><br class=""><div class="">> There is some issues during kspview if PCLSC is used as shown below. But the code could be ran if I do not show ksp details.</div><div class="">> In order to show the whole ksp setup, I also attached another kspview output without PCLSC.</div><div class="">> ======================kspview output ==============================</div><div class="">> KSP Object: 1 MPI processes</div><div class="">> type: fgmres</div><div class="">> restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement</div><div class="">> happy breakdown tolerance 1e-30</div><div class="">> maximum iterations=10000, initial guess is zero</div><div class="">> tolerances: relative=1e-08, absolute=1e-50, divergence=10000.</div><div class="">> right preconditioning</div><div class="">> using UNPRECONDITIONED norm type for convergence test</div><div class="">> PC Object: 1 MPI processes</div><div class="">> type: fieldsplit</div><div class="">> FieldSplit with Schur preconditioner, blocksize = 1, factorization FULL</div><div class="">> Preconditioner for the Schur complement formed from S itself</div><div class="">> Split info:</div><div class="">> Split number 0 Defined by IS</div><div class="">> Split number 1 Defined by IS</div><div class="">> KSP solver for A00 block</div><div class="">> KSP Object: (fieldsplit_0_) 1 MPI processes</div><div class="">> type: gmres</div><div class="">> restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement</div><div class="">> happy breakdown tolerance 1e-30</div><div class="">> maximum iterations=10000, initial guess is zero</div><div class="">> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.</div><div class="">> left preconditioning</div><div class="">> using DEFAULT norm type for convergence test</div><div class="">> PC Object: (fieldsplit_0_) 1 MPI processes</div><div class="">> type: lu</div><div class="">> PC has not been set up so information may be incomplete</div><div class="">> out-of-place factorization</div><div class="">> tolerance for zero pivot 2.22045e-14</div><div class="">> matrix ordering: nd</div><div class="">> linear system matrix = precond matrix:</div><div class="">> Mat Object: (fieldsplit_0_) 1 MPI processes</div><div class="">> type: seqaijcusparse</div><div class="">> rows=8396802, cols=8396802</div><div class="">> total: nonzeros=75440146, allocated nonzeros=75440146</div><div class="">> total number of mallocs used during MatSetValues calls=0</div><div class="">> not using I-node routines</div><div class="">> KSP solver for S = A11 - A10 inv(A00) A01</div><div class="">> KSP Object: (fieldsplit_1_) 1 MPI processes</div><div class="">> type: gmres</div><div class="">> restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement</div><div class="">> happy breakdown tolerance 1e-30</div><div class="">> maximum iterations=10000, initial guess is zero</div><div class="">> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.</div><div class="">> left preconditioning</div><div class="">> using DEFAULT norm type for convergence test</div><div class="">> PC Object: (fieldsplit_1_) 1 MPI processes</div><div class="">> type: lsc</div><div class="">> PC has not been set up so information may be incomplete</div><div class="">> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</div><div class="">> [0]PETSC ERROR: Null argument, when expecting valid pointer</div><div class="">> [0]PETSC ERROR: Null Object: Parameter # 1</div><div class="">> [0]PETSC ERROR: See <a href="https://www.mcs.anl.gov/petsc/documentation/faq.html" class="">https://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.</div><div class="">> [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020</div><div class="">> [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named hsw221 by yjuntao Sat Jul 18 21:08:42 2020</div><div class="">> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda</div><div class="">> [0]PETSC ERROR: #1 KSPView() line 111 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itcreate.c</div><div class="">> [0]PETSC ERROR: #2 PCView_LSC() line 142 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/lsc/lsc.c</div><div class="">> [0]PETSC ERROR: #3 PCView() line 1588 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c</div><div class="">> [0]PETSC ERROR: #4 KSPView() line 217 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itcreate.c</div><div class="">> [0]PETSC ERROR: #5 PCView_FieldSplit_Schur() line 238 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/fieldsplit/fieldsplit.c</div><div class="">> [0]PETSC ERROR: #6 PCView() line 1588 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c</div><div class="">> [0]PETSC ERROR: #7 KSPView() line 217 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itcreate.c</div><br class=""><div class="">This is something we can fix. Since the LSC has not been setup yet it should skip the KSPView on the KSP that has not yet been created.</div><br class=""><div class="">static PetscErrorCode PCView_LSC(PC pc,PetscViewer viewer)</div><div class="">{</div><div class="">PC_LSC *jac = (PC_LSC*)pc->data;</div><div class="">PetscErrorCode ierr;</div><div class="">PetscBool iascii;</div><br class=""><div class="">PetscFunctionBegin;</div><div class="">ierr = PetscObjectTypeCompare((PetscObject)viewer,PETSCVIEWERASCII,&iascii);CHKERRQ(ierr);</div><div class="">if (iascii) {</div><div class="">ierr = PetscViewerASCIIPushTab(viewer);CHKERRQ(ierr);</div><div class="">ierr = KSPView(jac->kspL,viewer);CHKERRQ(ierr);</div><div class="">ierr = PetscViewerASCIIPopTab(viewer);CHKERRQ(ierr);</div><div class="">}</div><div class="">PetscFunctionReturn(0);</div><div class="">}</div><br class=""><div class="">></div><div class="">> =====================KSP without PCLSC======================================</div><div class="">> KSP Object: 1 MPI processes</div><div class="">> type: fgmres</div><div class="">> restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement</div><div class="">> happy breakdown tolerance 1e-30</div><div class="">> maximum iterations=10000, initial guess is zero</div><div class="">> tolerances: relative=1e-08, absolute=1e-50, divergence=10000.</div><div class="">> right preconditioning</div><div class="">> using UNPRECONDITIONED norm type for convergence test</div><div class="">> PC Object: 1 MPI processes</div><div class="">> type: fieldsplit</div><div class="">> FieldSplit with Schur preconditioner, blocksize = 1, factorization FULL</div><div class="">> Preconditioner for the Schur complement formed from S itself</div><div class="">> Split info:</div><div class="">> Split number 0 Defined by IS</div><div class="">> Split number 1 Defined by IS</div><div class="">> KSP solver for A00 block</div><div class="">> KSP Object: (fieldsplit_0_) 1 MPI processes</div><div class="">> type: gmres</div><div class="">> restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement</div><div class="">> happy breakdown tolerance 1e-30</div><div class="">> maximum iterations=10000, initial guess is zero</div><div class="">> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.</div><div class="">> left preconditioning</div><div class="">> using DEFAULT norm type for convergence test</div><div class="">> PC Object: (fieldsplit_0_) 1 MPI processes</div><div class="">> type: lu</div><div class="">> PC has not been set up so information may be incomplete</div><div class="">> out-of-place factorization</div><div class="">> tolerance for zero pivot 2.22045e-14</div><div class="">> matrix ordering: nd</div><div class="">> linear system matrix = precond matrix:</div><div class="">> Mat Object: (fieldsplit_0_) 1 MPI processes</div><div class="">> type: seqaijcusparse</div><div class="">> rows=8396802, cols=8396802</div><div class="">> total: nonzeros=75440146, allocated nonzeros=75440146</div><div class="">> total number of mallocs used during MatSetValues calls=0</div><div class="">> not using I-node routines</div><div class="">> KSP solver for S = A11 - A10 inv(A00) A01</div><div class="">> KSP Object: (fieldsplit_1_) 1 MPI processes</div><div class="">> type: gmres</div><div class="">> restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement</div><div class="">> happy breakdown tolerance 1e-30</div><div class="">> maximum iterations=10000, initial guess is zero</div><div class="">> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.</div><div class="">> left preconditioning</div><div class="">> using DEFAULT norm type for convergence test</div><div class="">> PC Object: (fieldsplit_1_) 1 MPI processes</div><div class="">> type: none</div><div class="">> PC has not been set up so information may be incomplete</div><div class="">> linear system matrix = precond matrix:</div><div class="">> Mat Object: (fieldsplit_1_) 1 MPI processes</div><div class="">> type: schurcomplement</div><div class="">> rows=1050625, cols=1050625</div><div class="">> Schur complement A11 - A10 inv(A00) A01</div><div class="">> A11</div><div class="">> Mat Object: (fieldsplit_1_) 1 MPI processes</div><div class="">> type: seqaijcusparse</div><div class="">> rows=1050625, cols=1050625</div><div class="">> total: nonzeros=0, allocated nonzeros=0</div><div class="">> total number of mallocs used during MatSetValues calls=0</div><div class="">> not using I-node routines</div><div class="">> A10</div><div class="">> Mat Object: 1 MPI processes</div><div class="">> type: seqaijcusparse</div><div class="">> rows=1050625, cols=8396802</div><div class="">> total: nonzeros=44060674, allocated nonzeros=44060674</div><div class="">> total number of mallocs used during MatSetValues calls=0</div><div class="">> not using I-node routines</div><div class="">> KSP of A00</div><div class="">> KSP Object: (fieldsplit_0_) 1 MPI processes</div><div class="">> type: gmres</div><div class="">> restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement</div><div class="">> happy breakdown tolerance 1e-30</div><div class="">> maximum iterations=10000, initial guess is zero</div><div class="">> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.</div><div class="">> left preconditioning</div><div class="">> using DEFAULT norm type for convergence test</div><div class="">> PC Object: (fieldsplit_0_) 1 MPI processes</div><div class="">> type: lu</div><div class="">> PC has not been set up so information may be incomplete</div><div class="">> out-of-place factorization</div><div class="">> tolerance for zero pivot 2.22045e-14</div><div class="">> matrix ordering: nd</div><div class="">> linear system matrix = precond matrix:</div><div class="">> Mat Object: (fieldsplit_0_) 1 MPI processes</div><div class="">> type: seqaijcusparse</div><div class="">> rows=8396802, cols=8396802</div><div class="">> total: nonzeros=75440146, allocated nonzeros=75440146</div><div class="">> total number of mallocs used during MatSetValues calls=0</div><div class="">> not using I-node routines</div><div class="">> A01</div><div class="">> Mat Object: 1 MPI processes</div><div class="">> type: seqaijcusparse</div><div class="">> rows=8396802, cols=1050625</div><div class="">> total: nonzeros=43995146, allocated nonzeros=43995146</div><div class="">> total number of mallocs used during MatSetValues calls=0</div><div class="">> not using I-node routines</div><div class="">> linear system matrix = precond matrix:</div><div class="">> Mat Object: 1 MPI processes</div><div class="">> type: seqaijcusparse</div><div class="">> rows=9447427, cols=9447427</div><div class="">> total: nonzeros=163495966, allocated nonzeros=163643398</div><div class="">> total number of mallocs used during MatSetValues calls=0</div><div class="">> not using I-node routines</div><div class="">></div><div class="">> Regards</div><div class="">> JT</div></div></blockquote></div></blockquote></div><br class=""></div></body></html>