<div dir="ltr">Placing PCReset(PC pc) before the second kspsolve might works.<br><div><br></div><div>Fande Kong,</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Sep 19, 2016 at 7:38 PM, murat keçeli <span dir="ltr"><<a href="mailto:keceli@gmail.com" target="_blank">keceli@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><p style="margin:1.2em 0px!important">Another guess: maybe you also need <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">KSPSetUp(ksp);</code> before the second <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">KSPSolve(ksp,b,x);</code>.</p>
<p style="margin:1.2em 0px!important">Murat Keceli</p>
<div title="MDH:QW5vdGhlciBndWVzczogbWF5YmUgeW91IGFsc28gbmVlZCBgPHNwYW4gc3R5bGU9ImZvbnQtc2l6
ZTogMTIuOHB4OyI+S1NQU2V0VXAoa3NwKTtgIGJlZm9yZSB0aGUgc2Vjb25kJm5ic3A7PC9zcGFu
PjxzcGFuIHN0eWxlPSJmb250LXNpemU6IDEyLjhweDsiPiZuYnNwOyBgS1NQU29sdmUoa3NwLGIs
eCk7YC48L3NwYW4+PGRpdj48c3BhbiBzdHlsZT0iZm9udC1zaXplOiAxMi44cHg7Ij48YnI+PC9z
cGFuPjwvZGl2PjxkaXY+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZTogMTIuOHB4OyI+TXVyYXQgS2Vj
ZWxpPGJyPjwvc3Bhbj48ZGl2PjxzcGFuIHN0eWxlPSJmb250LXNpemU6IDEyLjhweDsiPjxicj48
L3NwYW4+PGRpdj48c3BhbiBzdHlsZT0iZm9udC1zaXplOiAxMi44cHg7Ij48YnI+PC9zcGFuPjwv
ZGl2PjwvZGl2PjwvZGl2Pg==" style="min-height:0;width:0;max-height:0;max-width:0;overflow:hidden;font-size:0em;padding:0;margin:0"></div></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Sep 19, 2016 at 8:33 PM, David Knezevic <span dir="ltr"><<a href="mailto:david.knezevic@akselos.com" target="_blank">david.knezevic@akselos.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span>On Mon, Sep 19, 2016 at 7:26 PM, Dave May <span dir="ltr"><<a href="mailto:dave.mayhem23@gmail.com" target="_blank">dave.mayhem23@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span>On 19 September 2016 at 21:05, David Knezevic <span dir="ltr"><<a href="mailto:david.knezevic@akselos.com" target="_blank">david.knezevic@akselos.com</a>></span> wrote:<br><blockquote style="margin-top:0px;margin-bottom:0px" class="gmail_quote"><div dir="ltr">When I use MUMPS via PETSc, one issue is that it can sometimes fail with MUMPS error -9, which means that MUMPS didn't allocate a big enough workspace. This can typically be fixed by increasing MUMPS icntl 14, e.g. via the command line option -mat_mumps_icntl_14.<div><br></div><div>However, instead of having to run several times with different command line options, I'd like to be able to automatically increment icntl 14 value in a loop until the solve succeeds.</div><div><br></div><div>I have a saved matrix which fails when I use it for a solve with MUMPS with 4 MPI processes and the default ictnl values, so I'm using this to check that I can achieve the automatic icntl 14 update, as described above. (The matrix is 14MB so I haven't attached it here, but I'd be happy to send it to anyone else who wants to try this test case out.)</div><div><br></div><div>I've pasted some test code below which provides a simple test of this idea using two solves. The first solve uses the default value of icntl 14, which fails, and then we update icntl 14 to 30 and solve again. The second solve should succeed since icntl 14 of 30 is sufficient for MUMPS to succeed in this case, but for some reason the second solve still fails.<br></div><div><br></div><div>Below I've also pasted the output from -ksp_view, and you can see that ictnl 14 is being updated correctly (see the ICNTL(14) lines in the output), so it's not clear to me why the second solve fails. It seems like MUMPS is ignoring the update to the ictnl value?</div></div></blockquote><div><br></div></span><div>I believe this parameter is utilized during the numerical factorization phase.<br></div><div>In your code, the operator hasn't changed, however you haven't signalled to the KSP that you want to re-perform the numerical factorization.<br></div><div>You can do this by calling KSPSetOperators() before your second solve. <br></div><div>I think if you do this (please try it), the factorization will be performed again and the new value of icntl will have an effect.<br><br></div><div>Note this is a wild stab in the dark - I haven't dug through the petsc-mumps code in detail...<br></div></div></div></div></blockquote><div><br></div></span><div>That sounds like a plausible guess to me, but unfortunately it didn't work. I added KSPSetOperators(ksp,A,A)<wbr>; before the second solve and I got the same behavior as before.</div><div><div><div><br></div><div>Thanks,</div><div>David</div><div><br></div><div><br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><div><blockquote style="margin-top:0px;margin-bottom:0px" class="gmail_quote"><div dir="ltr"><div>------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>-----------</div><div>Test code:</div><div><br></div><div><div><div><div> Mat A;</div><div> MatCreate(PETSC_COMM_WORLD,&A)<wbr>;</div><div> MatSetType(A,MATMPIAIJ);</div><div><br></div><div> PetscViewer petsc_viewer;</div><div> PetscViewerBinaryOpen( PETSC_COMM_WORLD,</div><div> "matrix.dat",</div><div> FILE_MODE_READ,</div><div> &petsc_viewer);</div><div> MatLoad(A, petsc_viewer);</div><div> PetscViewerDestroy(&petsc_view<wbr>er);</div><div><br></div><div> PetscInt m, n;</div><div> MatGetSize(A, &m, &n);</div><div><br></div><div> Vec x;</div><div> VecCreate(PETSC_COMM_WORLD,&x)<wbr>;</div><div> VecSetSizes(x,PETSC_DECIDE,m);</div><div> VecSetFromOptions(x);</div><div> VecSet(x,1.0);</div><div><br></div><div> Vec b;</div><div> VecDuplicate(x,&b);</div><div><br></div><div> KSP ksp;</div><div> PC pc;</div><div><br></div><div> KSPCreate(PETSC_COMM_WORLD,&ks<wbr>p);</div><div> KSPSetOperators(ksp,A,A);</div><div><br></div><div> KSPSetType(ksp,KSPPREONLY);</div><div> KSPGetPC(ksp,&pc);</div><div><br></div><div> PCSetType(pc,PCCHOLESKY);</div><div><br></div><div> PCFactorSetMatSolverPackage(pc<wbr>,MATSOLVERMUMPS);</div><div> PCFactorSetUpMatSolverPackage(<wbr>pc);</div><div><br></div><div> KSPSetFromOptions(ksp);</div><div> KSPSetUp(ksp);</div><div><br></div><div> KSPSolve(ksp,b,x);</div><div><br></div><div> {</div><div> KSPConvergedReason reason;</div><div> KSPGetConvergedReason(ksp, &reason);</div><div> std::cout << "converged reason: " << reason << std::endl;</div><div> }</div><div><br></div><div> Mat F;</div><div> PCFactorGetMatrix(pc,&F);</div><div> MatMumpsSetIcntl(F,14,30);</div><div><br></div><div> KSPSolve(ksp,b,x);</div><div><br></div><div> {</div><div> KSPConvergedReason reason;</div><div> KSPGetConvergedReason(ksp, &reason);</div><div> std::cout << "converged reason: " << reason << std::endl;</div><div> }</div></div></div></div><div><br></div><div><div>------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>-----------</div></div><div>-ksp_view output (ICNTL(14) changes from 20 to 30, but we get "converged reason: -11" for both solves)</div><div><br></div><div><div>KSP Object: 4 MPI processes</div><div> type: preonly</div><div> maximum iterations=10000, initial guess is zero</div><div> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.</div><div> left preconditioning</div><div> using NONE norm type for convergence test</div><div>PC Object: 4 MPI processes</div><div> type: cholesky</div><div> Cholesky: out-of-place factorization</div><div> tolerance for zero pivot 2.22045e-14</div><div> matrix ordering: natural</div><div> factor fill ratio given 0., needed 0.</div><div> Factored matrix follows:</div><div> Mat Object: 4 MPI processes</div><div> type: mpiaij</div><div> rows=22878, cols=22878</div><div> package used to perform factorization: mumps</div><div> total: nonzeros=3361617, allocated nonzeros=3361617</div><div> total number of mallocs used during MatSetValues calls =0</div><div> MUMPS run parameters:</div><div> SYM (matrix type): 2 </div><div> PAR (host participation): 1 </div><div> ICNTL(1) (output for error): 6 </div><div> ICNTL(2) (output of diagnostic msg): 0 </div><div> ICNTL(3) (output for global info): 0 </div><div> ICNTL(4) (level of printing): 0 </div><div> ICNTL(5) (input mat struct): 0 </div><div> ICNTL(6) (matrix prescaling): 7 </div><div> ICNTL(7) (sequentia matrix ordering):7 </div><div> ICNTL(8) (scalling strategy): 77 </div><div> ICNTL(10) (max num of refinements): 0 </div><div> ICNTL(11) (error analysis): 0 </div><div> ICNTL(12) (efficiency control): 0 </div><div> ICNTL(13) (efficiency control): 0 </div><div> ICNTL(14) (percentage of estimated workspace increase): 20 </div><div> ICNTL(18) (input mat struct): 3 </div><div> ICNTL(19) (Shur complement info): 0 </div><div> ICNTL(20) (rhs sparse pattern): 0 </div><div> ICNTL(21) (solution struct): 1 </div><div> ICNTL(22) (in-core/out-of-core facility): 0 </div><div> ICNTL(23) (max size of memory can be allocated locally):0 </div><div> ICNTL(24) (detection of null pivot rows): 0 </div><div> ICNTL(25) (computation of a null space basis): 0 </div><div> ICNTL(26) (Schur options for rhs or solution): 0 </div><div> ICNTL(27) (experimental parameter): -24 </div><div> ICNTL(28) (use parallel or sequential ordering): 1 </div><div> ICNTL(29) (parallel ordering): 0 </div><div> ICNTL(30) (user-specified set of entries in inv(A)): 0 </div><div> ICNTL(31) (factors is discarded in the solve phase): 0 </div><div> ICNTL(33) (compute determinant): 0 </div><div> CNTL(1) (relative pivoting threshold): 0.01 </div><div> CNTL(2) (stopping criterion of refinement): 1.49012e-08 </div><div> CNTL(3) (absolute pivoting threshold): 0. </div><div> CNTL(4) (value of static pivoting): -1. </div><div> CNTL(5) (fixation for null pivots): 0. </div><div> RINFO(1) (local estimated flops for the elimination after analysis): </div><div> [0] 1.84947e+08 </div><div> [1] 2.42065e+08 </div><div> [2] 2.53044e+08 </div><div> [3] 2.18441e+08 </div><div> RINFO(2) (local estimated flops for the assembly after factorization): </div><div> [0] 945938. </div><div> [1] 906795. </div><div> [2] 897815. </div><div> [3] 998840. </div><div> RINFO(3) (local estimated flops for the elimination after factorization): </div><div> [0] 1.59835e+08 </div><div> [1] 1.50867e+08 </div><div> [2] 2.27932e+08 </div><div> [3] 1.52037e+08 </div><div> INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): </div><div> [0] 36 </div><div> [1] 37 </div><div> [2] 38 </div><div> [3] 39 </div><div> INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): </div><div> [0] 36 </div><div> [1] 37 </div><div> [2] 38 </div><div> [3] 39 </div><div> INFO(23) (num of pivots eliminated on this processor after factorization): </div><div> [0] 6450 </div><div> [1] 5442 </div><div> [2] 4386 </div><div> [3] 5526 </div><div> RINFOG(1) (global estimated flops for the elimination after analysis): 8.98497e+08 </div><div> RINFOG(2) (global estimated flops for the assembly after factorization): 3.74939e+06 </div><div> RINFOG(3) (global estimated flops for the elimination after factorization): 6.9067e+08 </div><div> (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0.,0.)*(2^0)</div><div> INFOG(3) (estimated real workspace for factors on all processors after analysis): 4082184 </div><div> INFOG(4) (estimated integer workspace for factors on all processors after analysis): 231846 </div><div> INFOG(5) (estimated maximum front size in the complete tree): 678 </div><div> INFOG(6) (number of nodes in the complete tree): 1380 </div><div> INFOG(7) (ordering option effectively use after analysis): 5 </div><div> INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 100 </div><div> INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 3521904 </div><div> INFOG(10) (total integer space store the matrix factors after factorization): 229416 </div><div> INFOG(11) (order of largest frontal matrix after factorization): 678 </div><div> INFOG(12) (number of off-diagonal pivots): 0 </div><div> INFOG(13) (number of delayed pivots after factorization): 0 </div><div> INFOG(14) (number of memory compress after factorization): 0 </div><div> INFOG(15) (number of steps of iterative refinement after solution): 0 </div><div> INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 39 </div><div> INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 150 </div><div> INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 39 </div><div> INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 150 </div><div> INFOG(20) (estimated number of entries in the factors): 3361617 </div><div> INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 35 </div><div> INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 136 </div><div> INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 </div><div> INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 </div><div> INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 </div><div> INFOG(28) (after factorization: number of null pivots encountered): 0</div><div> INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 2931438</div><div> INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 0, 0</div><div> INFOG(32) (after analysis: type of analysis done): 1</div><div> INFOG(33) (value used for ICNTL(8)): 7</div><div> INFOG(34) (exponent of the determinant if determinant is requested): 0</div><div> linear system matrix = precond matrix:</div><div> Mat Object: 4 MPI processes</div><div> type: mpiaij</div><div> rows=22878, cols=22878</div><div> total: nonzeros=1219140, allocated nonzeros=1219140</div><div> total number of mallocs used during MatSetValues calls =0</div><div> using I-node (on process 0) routines: found 1889 nodes, limit used is 5</div><div>converged reason: -11</div><div>KSP Object: 4 MPI processes</div><div> type: preonly</div><div> maximum iterations=10000, initial guess is zero</div><div> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.</div><div> left preconditioning</div><div> using NONE norm type for convergence test</div><div>PC Object: 4 MPI processes</div><div> type: cholesky</div><div> Cholesky: out-of-place factorization</div><div> tolerance for zero pivot 2.22045e-14</div><div> matrix ordering: natural</div><div> factor fill ratio given 0., needed 0.</div><div> Factored matrix follows:</div><div> Mat Object: 4 MPI processes</div><div> type: mpiaij</div><div> rows=22878, cols=22878</div><div> package used to perform factorization: mumps</div><div> total: nonzeros=3361617, allocated nonzeros=3361617</div><div> total number of mallocs used during MatSetValues calls =0</div><div> MUMPS run parameters:</div><div> SYM (matrix type): 2 </div><div> PAR (host participation): 1 </div><div> ICNTL(1) (output for error): 6 </div><div> ICNTL(2) (output of diagnostic msg): 0 </div><div> ICNTL(3) (output for global info): 0 </div><div> ICNTL(4) (level of printing): 0 </div><div> ICNTL(5) (input mat struct): 0 </div><div> ICNTL(6) (matrix prescaling): 7 </div><div> ICNTL(7) (sequentia matrix ordering):7 </div><div> ICNTL(8) (scalling strategy): 77 </div><div> ICNTL(10) (max num of refinements): 0 </div><div> ICNTL(11) (error analysis): 0 </div><div> ICNTL(12) (efficiency control): 0 </div><div> ICNTL(13) (efficiency control): 0 </div><div> ICNTL(14) (percentage of estimated workspace increase): 30 </div><div> ICNTL(18) (input mat struct): 3 </div><div> ICNTL(19) (Shur complement info): 0 </div><div> ICNTL(20) (rhs sparse pattern): 0 </div><div> ICNTL(21) (solution struct): 1 </div><div> ICNTL(22) (in-core/out-of-core facility): 0 </div><div> ICNTL(23) (max size of memory can be allocated locally):0 </div><div> ICNTL(24) (detection of null pivot rows): 0 </div><div> ICNTL(25) (computation of a null space basis): 0 </div><div> ICNTL(26) (Schur options for rhs or solution): 0 </div><div> ICNTL(27) (experimental parameter): -24 </div><div> ICNTL(28) (use parallel or sequential ordering): 1 </div><div> ICNTL(29) (parallel ordering): 0 </div><div> ICNTL(30) (user-specified set of entries in inv(A)): 0 </div><div> ICNTL(31) (factors is discarded in the solve phase): 0 </div><div> ICNTL(33) (compute determinant): 0 </div><div> CNTL(1) (relative pivoting threshold): 0.01 </div><div> CNTL(2) (stopping criterion of refinement): 1.49012e-08 </div><div> CNTL(3) (absolute pivoting threshold): 0. </div><div> CNTL(4) (value of static pivoting): -1. </div><div> CNTL(5) (fixation for null pivots): 0. </div><div> RINFO(1) (local estimated flops for the elimination after analysis): </div><div> [0] 1.84947e+08 </div><div> [1] 2.42065e+08 </div><div> [2] 2.53044e+08 </div><div> [3] 2.18441e+08 </div><div> RINFO(2) (local estimated flops for the assembly after factorization): </div><div> [0] 945938. </div><div> [1] 906795. </div><div> [2] 897815. </div><div> [3] 998840. </div><div> RINFO(3) (local estimated flops for the elimination after factorization): </div><div> [0] 1.59835e+08 </div><div> [1] 1.50867e+08 </div><div> [2] 2.27932e+08 </div><div> [3] 1.52037e+08 </div><div> INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): </div><div> [0] 36 </div><div> [1] 37 </div><div> [2] 38 </div><div> [3] 39 </div><div> INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): </div><div> [0] 36 </div><div> [1] 37 </div><div> [2] 38 </div><div> [3] 39 </div><div> INFO(23) (num of pivots eliminated on this processor after factorization): </div><div> [0] 6450 </div><div> [1] 5442 </div><div> [2] 4386 </div><div> [3] 5526 </div><div> RINFOG(1) (global estimated flops for the elimination after analysis): 8.98497e+08 </div><div> RINFOG(2) (global estimated flops for the assembly after factorization): 3.74939e+06 </div><div> RINFOG(3) (global estimated flops for the elimination after factorization): 6.9067e+08 </div><div> (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0.,0.)*(2^0)</div><div> INFOG(3) (estimated real workspace for factors on all processors after analysis): 4082184 </div><div> INFOG(4) (estimated integer workspace for factors on all processors after analysis): 231846 </div><div> INFOG(5) (estimated maximum front size in the complete tree): 678 </div><div> INFOG(6) (number of nodes in the complete tree): 1380 </div><div> INFOG(7) (ordering option effectively use after analysis): 5 </div><div> INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 100 </div><div> INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 3521904 </div><div> INFOG(10) (total integer space store the matrix factors after factorization): 229416 </div><div> INFOG(11) (order of largest frontal matrix after factorization): 678 </div><div> INFOG(12) (number of off-diagonal pivots): 0 </div><div> INFOG(13) (number of delayed pivots after factorization): 0 </div><div> INFOG(14) (number of memory compress after factorization): 0 </div><div> INFOG(15) (number of steps of iterative refinement after solution): 0 </div><div> INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 39 </div><div> INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 150 </div><div> INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 39 </div><div> INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 150 </div><div> INFOG(20) (estimated number of entries in the factors): 3361617 </div><div> INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 35 </div><div> INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 136 </div><div> INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 </div><div> INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 </div><div> INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 </div><div> INFOG(28) (after factorization: number of null pivots encountered): 0</div><div> INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 2931438</div><div> INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 0, 0</div><div> INFOG(32) (after analysis: type of analysis done): 1</div><div> INFOG(33) (value used for ICNTL(8)): 7</div><div> INFOG(34) (exponent of the determinant if determinant is requested): 0</div><div> linear system matrix = precond matrix:</div><div> Mat Object: 4 MPI processes</div><div> type: mpiaij</div><div> rows=22878, cols=22878</div><div> total: nonzeros=1219140, allocated nonzeros=1219140</div><div> total number of mallocs used during MatSetValues calls =0</div><div> using I-node (on process 0) routines: found 1889 nodes, limit used is 5</div><div>converged reason: -11</div></div><div><br></div><div>------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>-----------<br></div></div>
</blockquote></div></div></div><br></div></div>
</blockquote></div></div></div></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>