<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<font color="#0433ff" style="font-family: Helvetica; font-size: 12px">Dear Hong and Barry, <br>
</font></div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<font color="#0433ff" style="font-family: Helvetica; font-size: 12px"><br>
Thanks for the suggestions. So there could be some problems in my PETSc configuration? - but my PETSc lib was indeed compiled without the debug flags (--with-debugging=0). I use GCC/GFortran (Home-brew GCC 9.2.0) for the compiling and building of PETSc and
my own fortran code. My Fortran compiling flags are simply like: <br>
<br>
-O3 -ffree-line-length-none -fastsse <br>
<br>
Which is also used for -FOPTFLAGS in PETSc (I added -openmp for PETSc, but not my fortran code, as I dont have any OMP optimizations in my program). Note the performance test results I listed yesterday (e.g. 4.08s with 41 bcgs iterations.) are without any
CSR-array->PETSc translation overhead (only include the set and solve part). </font>
<div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px"><font color="#0433ff"><br>
</font></div>
<div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px"><font color="#0433ff">I have two questions about the performance difference - </font></div>
<div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px"><font color="#0433ff"><br>
</font></div>
<div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px"><font color="#0433ff">1. Is ilu only factorized once for each iteration, or ilu is performed at every outer ksp iteration steps? Sounds unlikely - but if so, this could cause some
extra overheads. </font></div>
<div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px"><font color="#0433ff"><br>
</font></div>
<div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px"><font color="#0433ff">2. Some KSP solvers like BCGS or TFQMR has two half-iterations for each iteration step. Not sure how it works in PETSc, but is that possible that both the
two half" relative residuals are counted in the output array, doubling the number of iterations (but that cannot explain the extra time consumed)?</font></div>
<div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px"><font color="#0433ff"><br>
Anyway, the output with -log_view from the same 278906 by 278906 matrix with 3-block D-ILU in PETSc is as follows: </font>
<div><br>
<br>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
---------------------------------------------- PETSc Performance Summary: ----------------------------------------------</div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(117, 117, 117); min-height: 13px">
<span style="font-variant-ligatures: no-common-ligatures"> </span><br>
</div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">MEMsolv.lu on a arch-darwin-c-opt named Haos-MBP with 1 processor, by donghao Thu Feb 6 09:07:35 2020</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">Using Petsc Release Version 3.12.3, unknown</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(117, 117, 117); min-height: 13px">
<span style="font-variant-ligatures: no-common-ligatures"> </span><br>
</div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> Max Max/Min Avg Total</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">Time (sec): 4.443e+00 1.000 4.443e+00</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">Objects: 1.155e+03 1.000 1.155e+03</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">Flop: 4.311e+09 1.000 4.311e+09 4.311e+09</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">Flop/sec: 9.703e+08 1.000 9.703e+08 9.703e+08</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">MPI Messages: 0.000e+00 0.000 0.000e+00 0.000e+00</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">MPI Message Lengths: 0.000e+00 0.000 0.000e+00 0.000e+00</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">MPI Reductions: 0.000e+00 0.000</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(117, 117, 117); min-height: 13px">
<span style="font-variant-ligatures: no-common-ligatures"> </span><br>
</div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> e.g., VecAXPY() for real vectors of length N --> 2N flop</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> and VecAXPY() for complex vectors of length N --> 8N flop</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(117, 117, 117); min-height: 13px">
<span style="font-variant-ligatures: no-common-ligatures"> </span><br>
</div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">Summary of Stages: ----- Time ------ ----- Flop ------ --- Messages --- -- Message Lengths -- -- Reductions
--</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> Avg %Total Avg %Total Count %Total Avg %Total Count %Total</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 0: Main Stage: 4.4435e+00 100.0% 4.3113e+09 100.0% 0.000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0%</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(117, 117, 117); min-height: 13px">
<span style="font-variant-ligatures: no-common-ligatures"> </span><br>
</div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"></span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures">
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures"> See the 'Profiling' chapter of the users' manual for details on interpreting output.</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">Phase summary info:</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> Count: number of times phase was executed</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> Time and Flop: Max - maximum over all processors</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> Ratio - ratio of maximum to minimum over all processors</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> Mess: number of messages sent</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> AvgLen: average message length (bytes)</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> Reduct: number of global reductions</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> Global: entire computation</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> %T - percent time in this phase %F - percent flop in this phase</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> %M - percent messages in this phase %L - percent message lengths in this
phase</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> %R - percent reductions in this phase</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over all processors)</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">------------------------------------------------------------------------------------------------------------------------</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">Event Count Time (sec) Flop ---
Global --- --- Stage ---- Total</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct %T
%F %M %L %R %T %F %M %L %R Mflop/s</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">------------------------------------------------------------------------------------------------------------------------</span></div>
<div><span style="font-variant-ligatures: no-common-ligatures">
<div style="margin: 0px; color: rgb(117, 117, 117); min-height: 13px"><span style="font-variant-ligatures: no-common-ligatures"> </span><br>
</div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">--- Event Stage 0: Main Stage</span></div>
<div style="margin: 0px; color: rgb(117, 117, 117); min-height: 13px"><span style="font-variant-ligatures: no-common-ligatures"> </span><br>
</div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">BuildTwoSidedF 1 1.0 2.3000e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0
0</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">MatMult 83 1.0 1.7815e+00 1.0 2.08e+09 1.0 0.0e+00 0.0e+00 0.0e+00
40 48 0 0 0 40 48 0 0 0 1168</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">MatSolve 252 1.0 1.2708e+00 1.0 1.19e+09 1.0 0.0e+00 0.0e+00 0.0e+00
29 28 0 0 0 29 28 0 0 0 939</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">MatLUFactorNum 3 1.0 7.9725e-02 1.0 2.37e+07 1.0 0.0e+00 0.0e+00 0.0e+00 2 1 0 0 0 2 1 0 0 0 298</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">MatILUFactorSym 3 1.0 2.6998e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0
0</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">MatAssemblyBegin 5 1.0 3.6000e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0
0</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">MatAssemblyEnd 5 1.0 3.1619e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0
0</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">MatGetRowIJ 3 1.0 2.0000e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0
0</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">MatCreateSubMats 1 1.0 3.9659e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0
0</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">MatGetOrdering 3 1.0 4.3070e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0
0</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">MatView 3 1.0 1.3600e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0
0</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">VecDot 82 1.0 1.8948e-01 1.0 1.83e+08 1.0 0.0e+00 0.0e+00 0.0e+00 4 4 0 0 0 4 4 0 0 0 966</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">VecDotNorm2 41 1.0 1.6812e-01 1.0 1.83e+08 1.0 0.0e+00 0.0e+00 0.0e+00 4 4 0 0 0 4 4 0 0 0 1088</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">VecNorm 43 1.0 9.5099e-02 1.0 9.59e+07 1.0 0.0e+00 0.0e+00 0.0e+00 2 2 0 0 0 2 2 0 0 0 1009</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">VecCopy 2 1.0 1.0120e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0
0</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">VecSet 271 1.0 3.8922e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0
0</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">VecAXPY 1 1.0 7.7200e-04 1.0 2.23e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 2890</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">VecAXPBYCZ 82 1.0 2.4370e-01 1.0 3.66e+08 1.0 0.0e+00 0.0e+00 0.0e+00 5 8 0 0 0 5 8 0 0 0 1502</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">VecWAXPY 82 1.0 1.4148e-01 1.0 1.83e+08 1.0 0.0e+00 0.0e+00 0.0e+00 3 4 0 0 0 3 4 0 0 0 1293</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">VecAssemblyBegin 2 1.0 0.0000e+00 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0
0</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">VecAssemblyEnd 2 1.0 0.0000e+00 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0
0</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">VecScatterBegin 84 1.0 5.9300e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0
0</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">KSPSetUp 4 1.0 1.4167e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0
0</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">KSPSolve 1 1.0 4.0250e+00 1.0 4.31e+09 1.0 0.0e+00 0.0e+00 0.0e+00
91100 0 0 0 91100 0 0 0 1071</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">PCSetUp 4 1.0 1.5207e-01 1.0 2.37e+07 1.0 0.0e+00 0.0e+00 0.0e+00 3 1 0 0 0 3 1 0 0 0 156</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">PCSetUpOnBlocks 1 1.0 1.1116e-01 1.0 2.37e+07 1.0 0.0e+00 0.0e+00 0.0e+00 3 1 0 0 0 3 1 0 0 0 214</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">PCApply 84 1.0 1.2912e+00 1.0 1.19e+09 1.0 0.0e+00 0.0e+00 0.0e+00
29 28 0 0 0 29 28 0 0 0 924</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">PCApplyOnBlocks 252 1.0 1.2909e+00 1.0 1.19e+09 1.0 0.0e+00 0.0e+00 0.0e+00
29 28 0 0 0 29 28 0 0 0 924</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">------------------------------------------------------------------------------------------------------------------------</span></div>
<div><span style="font-variant-ligatures: no-common-ligatures"><br>
</span></div>
<div><span style="font-variant-ligatures: no-common-ligatures"><font color="#0433ff"># I skipped the memory part - the options (and compiler options) are as follows: </font></span></div>
<div><span style="font-variant-ligatures: no-common-ligatures"><br>
</span></div>
<div><span style="font-variant-ligatures: no-common-ligatures">
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures">#PETSc Option Table entries: </span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">-ksp_type bcgs</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">-ksp_view</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">-log_view</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">-pc_bjacobi_local_blocks 3</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">-pc_factor_levels 0</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">-pc_sub_type ilu</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">-pc_type bjacobi</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">#End of PETSc Option Table entries</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">Compiled with FORTRAN kernels</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">Compiled with full precision matrices (default)</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar)
16 sizeof(PetscInt) 4</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">Configure options: --with-scalar-type=complex --download-mumps --download-scalapack
--with-fortran-kernels=1 -- </span><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">FOPTFLAGS=-O3 -fastsse -mp -openmp --COPTFLAGS=-O3 -fastsse
-mp -openmp --CXXOPTFLAGS="-O3 -fastsse -mp -openmp" -- </span><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">with-debugging=0</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">-----------------------------------------</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">Libraries compiled on 2020-02-03 10:44:31 on Haos-MBP</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">Machine characteristics: Darwin-19.2.0-x86_64-i386-64bit</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">Using PETSc directory: /Users/donghao/src/git/PETSc-current</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">Using PETSc arch: arch-darwin-c-opt</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">-----------------------------------------</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures"><br>
</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">Using C compiler: mpicc -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas
-fstack-protector -fno-stack- </span><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">check -Qunused-arguments -fvisibility=hidden</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">Using Fortran compiler: mpif90 -Wall -ffree-line-length-0 -Wno-unused-dummy-argument</span></div>
<div><span style="font-variant-ligatures: no-common-ligatures"><br>
</span></div>
</span></div>
<div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures">Using include paths: -I/Users/donghao/src/git/PETSc-current/include -I/Users/donghao/src/git/PETSc-current/arch-darwin-</span><span style="font-variant-ligatures: no-common-ligatures">c-opt/include</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">-----------------------------------------</span></div>
<div style="margin: 0px; color: rgb(117, 117, 117); min-height: 13px"><span style="font-variant-ligatures: no-common-ligatures"> </span><br>
</div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">Using C linker: mpicc</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">Using Fortran linker: mpif90</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">Using libraries: -Wl,-rpath,/Users/donghao/src/git/PETSc-current/arch-darwin-c-opt/lib
-L/Users/donghao/src/git/PETSc- </span><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">current/arch-darwin-c-opt/lib -lpetsc -Wl,-rpath,/Users/donghao/src/git/PETSc-current/arch-darwin-c-opt/lib
-L/Users/ </span><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">donghao/src/git/PETSc-current/arch-darwin-c-opt/lib -Wl,-rpath,/usr/local/opt/libevent/lib
-L/usr/local/opt/libevent/ </span><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">lib -Wl,-rpath,/usr/local/Cellar/open-mpi/4.0.2/lib -L/usr/local/Cellar/open-mpi/4.0.2/lib
-Wl,-rpath,/usr/local/Cellar/</span><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">gcc/9.2.0_3/lib/gcc/9/gcc/x86_64-apple-darwin19/9.2.0 -L/usr/local/Cellar/gcc/9.2.0_3/lib/gcc/9/gcc/x86_64-apple-
</span><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">darwin19/9.2.0 -Wl,-rpath,/usr/local/Cellar/gcc/9.2.0_3/lib/gcc/9 -L/usr/local/Cellar/gcc/9.2.0_3/lib/gcc/9
-lcmumps - </span><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -llapack -lblas -lc++ -ldl
-lmpi_usempif08 - </span><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lquadmath -lm
-lc++ -ldl</span></div>
</div>
<div><span style="font-variant-ligatures: no-common-ligatures"><br>
</span></div>
<div><span style="font-variant-ligatures: no-common-ligatures"><br>
</span></div>
<div><span style="font-variant-ligatures: no-common-ligatures"><font color="#0433ff">On the other hand, running PETSc with </font></span></div>
<div><span style="font-variant-ligatures: no-common-ligatures"><span style="font-variant-ligatures: no-common-ligatures"><font color="#0433ff"><br>
</font></span></span></div>
<div><font color="#0433ff"><span style="font-variant-ligatures: no-common-ligatures"><span style="font-variant-ligatures: no-common-ligatures">-pc_type bjacobi </span></span>-pc_bjacobi_local_blocks 3 <span style="font-variant-ligatures: no-common-ligatures">-pc_sub_type</span><span style="font-variant-ligatures: no-common-ligatures"> lu
-ksp_type gmres -ksp_monitor -ksp_view -log_view</span></font></div>
<div><span style="font-variant-ligatures: no-common-ligatures"><font color="#0433ff"><br>
</font></span></div>
</span></div>
</span></div>
<div><font color="#0433ff">For the same problem takes 5.37s and 72 GMRES iterations. Our previous testings show that BiCGstab (bcgs in PETSc) is almost always the most effective KSP solver for our non-symmetrical complex system. Strangely, the system is still
using ilu instead of lu for sub blocks. The output is like: </font></div>
<div><br>
</div>
<div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 0 KSP Residual norm 2.480412407430e+02</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 1 KSP Residual norm 8.848059967835e+01</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 2 KSP Residual norm 3.415272863261e+01</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 3 KSP Residual norm 1.563045190939e+01</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 4 KSP Residual norm 6.241296940043e+00</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 5 KSP Residual norm 2.739710899854e+00</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 6 KSP Residual norm 1.391304148888e+00</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 7 KSP Residual norm 7.959262020849e-01</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 8 KSP Residual norm 4.828323055231e-01</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 9 KSP Residual norm 2.918529739200e-01</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 10 KSP Residual norm 1.905508589557e-01</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 11 KSP Residual norm 1.291541892702e-01</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 12 KSP Residual norm 8.827145774707e-02</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 13 KSP Residual norm 6.521331095889e-02</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 14 KSP Residual norm 5.095787952595e-02</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 15 KSP Residual norm 4.043060387395e-02</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 16 KSP Residual norm 3.232590200012e-02</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 17 KSP Residual norm 2.593944982216e-02</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 18 KSP Residual norm 2.064639483533e-02</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 19 KSP Residual norm 1.653916663492e-02</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 20 KSP Residual norm 1.334946415452e-02</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 21 KSP Residual norm 1.092886880597e-02</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 22 KSP Residual norm 8.988004105542e-03</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 23 KSP Residual norm 7.466501315240e-03</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 24 KSP Residual norm 6.284389135436e-03</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 25 KSP Residual norm 5.425231669964e-03</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 26 KSP Residual norm 4.766338253084e-03</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 27 KSP Residual norm 4.241238878242e-03</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 28 KSP Residual norm 3.808113525685e-03</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 29 KSP Residual norm 3.449383788116e-03</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 30 KSP Residual norm 3.126025526388e-03</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 31 KSP Residual norm 2.958328054299e-03</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 32 KSP Residual norm 2.802344900403e-03</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 33 KSP Residual norm 2.621993580492e-03</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 34 KSP Residual norm 2.430066269304e-03</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 35 KSP Residual norm 2.259043079597e-03</span></div>
</div>
<div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 36 KSP Residual norm 2.104287972986e-03</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 37 KSP Residual norm 1.952916080045e-03</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 38 KSP Residual norm 1.804988937999e-03</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 39 KSP Residual norm 1.643302117377e-03</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 40 KSP Residual norm 1.471661332036e-03</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 41 KSP Residual norm 1.286445911163e-03</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 42 KSP Residual norm 1.127543025848e-03</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 43 KSP Residual norm 9.777148275484e-04</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 44 KSP Residual norm 8.293314450006e-04</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 45 KSP Residual norm 6.989331136622e-04</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 46 KSP Residual norm 5.852307780220e-04</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 47 KSP Residual norm 4.926715539762e-04</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 48 KSP Residual norm 4.215941372075e-04</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 49 KSP Residual norm 3.699489548162e-04</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 50 KSP Residual norm 3.293897163533e-04</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 51 KSP Residual norm 2.959954542998e-04</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 52 KSP Residual norm 2.700193032414e-04</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 53 KSP Residual norm 2.461789791204e-04</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 54 KSP Residual norm 2.218839085563e-04</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 55 KSP Residual norm 1.945154309976e-04</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 56 KSP Residual norm 1.661128781744e-04</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 57 KSP Residual norm 1.413198766258e-04</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 58 KSP Residual norm 1.213984003195e-04</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 59 KSP Residual norm 1.044317450754e-04</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 60 KSP Residual norm 8.919957502977e-05</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 61 KSP Residual norm 8.042584301275e-05</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 62 KSP Residual norm 7.292784493581e-05</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 63 KSP Residual norm 6.481935501872e-05</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 64 KSP Residual norm 5.718564652679e-05</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 65 KSP Residual norm 5.072589750116e-05</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 66 KSP Residual norm 4.487930741285e-05</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 67 KSP Residual norm 3.941040674119e-05</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 68 KSP Residual norm 3.492873281291e-05</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 69 KSP Residual norm 3.103798339845e-05</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 70 KSP Residual norm 2.822943237409e-05</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 71 KSP Residual norm 2.610615023776e-05</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 72 KSP Residual norm 2.441692671173e-05</span></div>
</div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo">
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">KSP Object: 1 MPI processes</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> type: gmres</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with
no iterative refinement</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> happy breakdown tolerance 1e-30</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> maximum iterations=150, nonzero initial guess</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> tolerances: relative=1e-07, absolute=1e-50, divergence=10000.</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> left preconditioning</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> using PRECONDITIONED norm type for convergence test</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">PC Object: 1 MPI processes</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> type: bjacobi</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> number of blocks = 3</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> Local solve is same for all blocks, in the following KSP and PC objects:</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> KSP Object: (sub_) 1 MPI processes</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> type: preonly</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> maximum iterations=10000, initial guess is zero</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> left preconditioning</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> using NONE norm type for convergence test</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> PC Object: (sub_) 1 MPI processes</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> type: ilu</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> out-of-place factorization</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> 0 levels of fill</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> tolerance for zero pivot 2.22045e-14</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> matrix ordering: natural</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> factor fill ratio given 1., needed 1.</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> Factored matrix follows:</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> Mat Object: 1 MPI processes</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> type: seqaij</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> rows=92969, cols=92969</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> package used to perform factorization: petsc</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> total: nonzeros=638417, allocated nonzeros=638417</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> total number of mallocs used during MatSetValues calls=0</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> not using I-node routines</span></div>
<div><span style="font-variant-ligatures: no-common-ligatures">
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> linear system matrix = precond matrix:</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> Mat Object: 1 MPI processes</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> type: seqaij</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> rows=92969, cols=92969</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> total: nonzeros=638417, allocated nonzeros=638417</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> total number of mallocs used during MatSetValues calls=0</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> not using I-node routines</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> linear system matrix = precond matrix:</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> Mat Object: 1 MPI processes</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> type: mpiaij</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> rows=278906, cols=278906</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> total: nonzeros=3274027, allocated nonzeros=3274027</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> total number of mallocs used during MatSetValues calls=0</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> not using I-node (on process 0) routines</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures">...</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures">
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures"> ------------------------------------------------------------------------------------------------------------------------</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">Event Count Time (sec) Flop ---
Global --- --- Stage ---- Total</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct %T
%F %M %L %R %T %F %M %L %R Mflop/s</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">------------------------------------------------------------------------------------------------------------------------</span></div>
<div style="margin: 0px; color: rgb(117, 117, 117); min-height: 13px"><span style="font-variant-ligatures: no-common-ligatures"> </span><br>
</div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">--- Event Stage 0: Main Stage</span></div>
<div style="margin: 0px; color: rgb(117, 117, 117); min-height: 13px"><span style="font-variant-ligatures: no-common-ligatures"> </span><br>
</div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">BuildTwoSidedF 1 1.0 2.3000e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0
0</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">MatMult 75 1.0 1.5812e+00 1.0 1.88e+09 1.0 0.0e+00 0.0e+00 0.0e+00
28 24 0 0 0 28 24 0 0 0 1189</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">MatSolve 228 1.0 1.1442e+00 1.0 1.08e+09 1.0 0.0e+00 0.0e+00 0.0e+00
20 14 0 0 0 20 14 0 0 0 944</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">MatLUFactorNum 3 1.0 8.1930e-02 1.0 2.37e+07 1.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 290</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">MatILUFactorSym 3 1.0 2.7102e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0
0</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">MatAssemblyBegin 5 1.0 3.7000e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0
0</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">MatAssemblyEnd 5 1.0 3.1895e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0
0</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">MatGetRowIJ 3 1.0 2.0000e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0
0</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">MatCreateSubMats 1 1.0 4.0904e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0
0</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">MatGetOrdering 3 1.0 4.2640e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0
0</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">MatView 3 1.0 1.4400e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0
0</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">VecMDot 72 1.0 1.1984e+00 1.0 2.25e+09 1.0 0.0e+00 0.0e+00 0.0e+00
21 28 0 0 0 21 28 0 0 0 1877</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">VecNorm 76 1.0 1.6841e-01 1.0 1.70e+08 1.0 0.0e+00 0.0e+00 0.0e+00 3 2 0 0 0 3 2 0 0 0 1007</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">VecScale 75 1.0 1.8241e-02 1.0 8.37e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 4587</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">VecCopy 3 1.0 1.4970e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0
0</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">VecSet 276 1.0 9.1970e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 2 0 0 0 0
0</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">VecAXPY 6 1.0 3.7450e-03 1.0 1.34e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 3575</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">VecMAXPY 75 1.0 1.0022e+00 1.0 2.41e+09 1.0 0.0e+00 0.0e+00 0.0e+00
18 30 0 0 0 18 30 0 0 0 2405</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">VecAssemblyBegin 2 1.0 1.0000e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0
0</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">VecAssemblyEnd 2 1.0 0.0000e+00 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0
0</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">VecScatterBegin 76 1.0 5.5100e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0
0</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">VecNormalize 75 1.0 1.8462e-01 1.0 2.51e+08 1.0 0.0e+00 0.0e+00 0.0e+00 3 3 0 0 0 3 3 0 0 0 1360</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">KSPSetUp 4 1.0 1.1341e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0
0</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">KSPSolve 1 1.0 5.3123e+00 1.0 7.91e+09 1.0 0.0e+00 0.0e+00 0.0e+00
93100 0 0 0 93100 0 0 0 1489</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">KSPGMRESOrthog 72 1.0 2.1316e+00 1.0 4.50e+09 1.0 0.0e+00 0.0e+00 0.0e+00
37 57 0 0 0 37 57 0 0 0 2110</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">PCSetUp 4 1.0 1.5531e-01 1.0 2.37e+07 1.0 0.0e+00 0.0e+00 0.0e+00 3 0 0 0 0 3 0 0 0 0 153</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">PCSetUpOnBlocks 1 1.0 1.1343e-01 1.0 2.37e+07 1.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 2 0 0 0 0 209</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">PCApply 76 1.0 1.1671e+00 1.0 1.08e+09 1.0 0.0e+00 0.0e+00 0.0e+00
20 14 0 0 0 20 14 0 0 0 925</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">PCApplyOnBlocks 228 1.0 1.1668e+00 1.0 1.08e+09 1.0 0.0e+00 0.0e+00 0.0e+00
20 14 0 0 0 20 14 0 0 0 925</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"></span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures">...</span></div>
</span></div>
<div><span style="font-variant-ligatures: no-common-ligatures">
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures">#PETSc </span><span style="font-variant-ligatures: no-common-ligatures; color: rgb(151, 151, 151); background-color: rgb(0, 0, 255)">Option</span><span style="font-variant-ligatures: no-common-ligatures"> Table
entries:</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">-ksp_monitor</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">-ksp_type gmres</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">-ksp_view</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">-log_view</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">-pc_bjacobi_local_blocks 3</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">-pc_sub_type lu</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">-pc_type bjacobi</span></div>
<div style="margin: 0px"><span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures">#End of PETSc </span><span style="font-variant-ligatures: no-common-ligatures; color: rgb(151, 151, 151); background-color: rgb(0, 0, 255)">Option</span><span style="font-variant-ligatures: no-common-ligatures"> Table
entries</span></div>
</span></div>
<div><span style="font-variant-ligatures: no-common-ligatures"><font color="#0433ff">...</font></span></div>
<div><span style="font-variant-ligatures: no-common-ligatures"><font color="#0433ff"><br>
</font></span></div>
<div><span style="font-variant-ligatures: no-common-ligatures"><font color="#0433ff">Does any of the setup/output ring a bell? </font></span></div>
<div><span style="font-variant-ligatures: no-common-ligatures"><font color="#0433ff"><br>
</font></span></div>
<div><span style="font-variant-ligatures: no-common-ligatures"><font color="#0433ff">BTW, o</font></span><span style="color: rgb(4, 51, 255)">ut of curiosity - what is a I-node routine?</span></div>
<div><br>
</div>
</span></div>
</div>
<div><span style="font-variant-ligatures: no-common-ligatures"><br>
</span></div>
<div><span style="font-variant-ligatures: no-common-ligatures">Cheers, </span></div>
<div><span style="font-variant-ligatures: no-common-ligatures">Hao</span></div>
</div>
</div>
<br>
</div>
<div>
<div id="appendonsend"></div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Smith, Barry F. <bsmith@mcs.anl.gov><br>
<b>Sent:</b> Wednesday, February 5, 2020 9:42 PM<br>
<b>To:</b> Hao DONG <dong-hao@outlook.com><br>
<b>Cc:</b> petsc-users@mcs.anl.gov <petsc-users@mcs.anl.gov><br>
<b>Subject:</b> Re: [petsc-users] What is the right way to implement a (block) Diagonal ILU as PC?</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt">
<div class="PlainText"><br>
<br>
> On Feb 5, 2020, at 4:36 AM, Hao DONG <dong-hao@outlook.com> wrote:<br>
> <br>
> Thanks a lot for your suggestions, Hong and Barry - <br>
> <br>
> As you suggested, I first tried the LU direct solvers (built-in and MUMPs) out this morning, which work perfectly, albeit slow. As my problem itself is a part of a PDE based optimization, the exact solution in the searching procedure is not necessary (I often
set a relative tolerance of 1E-7/1E-8, etc. for Krylov subspace methods). Also tried BJACOBI with exact LU, the KSP just converges in one or two iterations, as expected.
<br>
> <br>
> I added -kspview option for the D-ILU test (still with Block Jacobi as preconditioner and bcgs as KSP solver). The KSPview output from one of the examples in a toy model looks like:
<br>
> <br>
> KSP Object: 1 MPI processes<br>
> type: bcgs<br>
> maximum iterations=120, nonzero initial guess<br>
> tolerances: relative=1e-07, absolute=1e-50, divergence=10000.<br>
> left preconditioning<br>
> using PRECONDITIONED norm type for convergence test<br>
> PC Object: 1 MPI processes<br>
> type: bjacobi<br>
> number of blocks = 3<br>
> Local solve is same for all blocks, in the following KSP and PC objects:<br>
> KSP Object: (sub_) 1 MPI processes<br>
> type: preonly<br>
> maximum iterations=10000, initial guess is zero<br>
> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.<br>
> left preconditioning<br>
> using NONE norm type for convergence test<br>
> PC Object: (sub_) 1 MPI processes<br>
> type: ilu<br>
> out-of-place factorization<br>
> 0 levels of fill<br>
> tolerance for zero pivot 2.22045e-14<br>
> matrix ordering: natural<br>
> factor fill ratio given 1., needed 1.<br>
> Factored matrix follows:<br>
> Mat Object: 1 MPI processes<br>
> type: seqaij<br>
> rows=11294, cols=11294<br>
> package used to perform factorization: petsc<br>
> total: nonzeros=76008, allocated nonzeros=76008<br>
> total number of mallocs used during MatSetValues calls=0<br>
> not using I-node routines<br>
> linear system matrix = precond matrix:<br>
> Mat Object: 1 MPI processes<br>
> type: seqaij<br>
> rows=11294, cols=11294<br>
> total: nonzeros=76008, allocated nonzeros=76008<br>
> total number of mallocs used during MatSetValues calls=0<br>
> not using I-node routines<br>
> linear system matrix = precond matrix:<br>
> Mat Object: 1 MPI processes<br>
> type: mpiaij<br>
> rows=33880, cols=33880<br>
> total: nonzeros=436968, allocated nonzeros=436968<br>
> total number of mallocs used during MatSetValues calls=0<br>
> not using I-node (on process 0) routines<br>
> <br>
> do you see something wrong with my setup?<br>
> <br>
> I also tried a quick performance test with a small 278906 by 278906 matrix (3850990 nnz) with the following parameters:
<br>
> <br>
> -ksp_type bcgs -pc_type bjacobi -pc_bjacobi_local_blocks 3 -pc_sub_type ilu -ksp_view<br>
> <br>
> Reducing the relative residual to 1E-7 <br>
> <br>
> Took 4.08s with 41 bcgs iterations. <br>
> <br>
> Merely changing the -pc_bjacobi_local_blocks to 6 <br>
> <br>
> Took 7.02s with 73 bcgs iterations. 9 blocks would further take 9.45s with 101 bcgs iterations.<br>
<br>
This is normal. more blocks slower convergence<br>
> <br>
> As a reference, my home-brew Fortran code solves the same problem (3-block D-ILU0) in
<br>
> <br>
> 1.84s with 24 bcgs iterations (the bcgs code is also a home-brew one)
<br>
> <br>
Run the PETSc code with optimization ./configure --with-debugging=0 an run the code with -log_view this will show where the PETSc code is spending the time (send it to use)<br>
<br>
<br>
> <br>
> <br>
> Well, by saying use explicit L/U matrix as preconditioner, I wonder if a user is allowed to provide his own (separate) L and U Mat for preconditioning (like how it works in Matlab solvers), e.g.<br>
> <br>
> x = qmr(A,b,Tol,MaxIter,L,U,x)<br>
> <br>
> As I already explicitly constructed the L and U matrices in Fortran, it is not hard to convert them to Mat in Petsc to test Petsc and my Fortran code head-to-head. In that case, the A, b, x, and L/U are all identical, it would be easier to see where the problem
came from. <br>
> <br>
> <br>
No, we don't provide this kind of support<br>
<br>
<br>
> <br>
> BTW, there is another thing I wondered - is there a way to output residual in unpreconditioned norm? I tried to<br>
> <br>
> call KSPSetNormType(ksp_local, KSP_NORM_UNPRECONDITIONED, ierr)<br>
> <br>
> But always get an error that current ksp does not support unpreconditioned in LEFT/RIGHT (either way). Is it possible to do that (output unpreconditioned residual) in PETSc at all?<br>
<br>
-ksp_monitor_true_residual You can also run GMRES (and some other methods) with right preconditioning, -ksp_pc_side right then the residual computed is by the algorithm the unpreconditioned residual<br>
<br>
KSPSetNormType sets the norm used in the algorithm, it generally always has to left or right, only a couple algorithms support unpreconditioned directly.<br>
<br>
Barry<br>
<br>
<br>
> <br>
> Cheers, <br>
> Hao<br>
> <br>
> <br>
> From: Smith, Barry F. <bsmith@mcs.anl.gov><br>
> Sent: Tuesday, February 4, 2020 8:27 PM<br>
> To: Hao DONG <dong-hao@outlook.com><br>
> Cc: petsc-users@mcs.anl.gov <petsc-users@mcs.anl.gov><br>
> Subject: Re: [petsc-users] What is the right way to implement a (block) Diagonal ILU as PC?<br>
> <br>
> <br>
> <br>
> > On Feb 4, 2020, at 12:41 PM, Hao DONG <dong-hao@outlook.com> wrote:<br>
> > <br>
> > Dear all, <br>
> > <br>
> > <br>
> > I have a few questions about the implementation of diagonal ILU PC in PETSc. I want to solve a very simple system with KSP (in parallel), the nature of the system (finite difference time-harmonic Maxwell) is probably not important to the question itself.
Long story short, I just need to solve a set of equations of Ax = b with a block diagonal system matrix, like (not sure if the mono font works):
<br>
> > <br>
> > |X | <br>
> > A =| Y | <br>
> > | Z| <br>
> > <br>
> > Note that A is not really block-diagonal, its just a multi-diagonal matrix determined by the FD mesh, where most elements are close to diagonal. So instead of a full ILU decomposition, a D-ILU is a good approximation as a preconditioner, and the number
of blocks should not matter too much: <br>
> > <br>
> > |Lx | |Ux |<br>
> > L = | Ly | and U = | Uy |<br>
> > | Lz| | Uz|<br>
> > <br>
> > Where [Lx, Ux] = ILU0(X), etc. Indeed, the D-ILU preconditioner (with 3N blocks) is quite efficient with Krylov subspace methods like BiCGstab or QMR in my sequential Fortran/Matlab code.
<br>
> > <br>
> > So like most users, I am looking for a parallel implement with this problem in PETSc. After looking through the manual, it seems that the most straightforward way to do it is through PCBJACOBI. Not sure I understand it right, I just setup a 3-block PCJACOBI
and give each of the block a KSP with PCILU. Is this supposed to be equivalent to my D-ILU preconditioner? My little block of fortran code would look like:
<br>
> > ...<br>
> > call PCBJacobiSetTotalBlocks(pc_local,Ntotal, &<br>
> > & isubs,ierr)<br>
> > call PCBJacobiSetLocalBlocks(pc_local, Nsub, &<br>
> > & isubs(istart:iend),ierr)<br>
> > ! set up the block jacobi structure<br>
> > call KSPSetup(ksp_local,ierr)<br>
> > ! allocate sub ksps<br>
> > allocate(ksp_sub(Nsub))<br>
> > call PCBJacobiGetSubKSP(pc_local,Nsub,istart, &<br>
> > & ksp_sub,ierr)<br>
> > do i=1,Nsub<br>
> > call KSPGetPC(ksp_sub(i),pc_sub,ierr)<br>
> > !ILU preconditioner<br>
> > call PCSetType(pc_sub,ptype,ierr)<br>
> > call PCFactorSetLevels(pc_sub,1,ierr) ! use ILU(1) here<br>
> > call KSPSetType(ksp_sub(i),KSPPREONLY,ierr)]<br>
> > end do<br>
> > call KSPSetTolerances(ksp_local,KSSiter%tol,PETSC_DEFAULT_REAL, &<br>
> > & PETSC_DEFAULT_REAL,KSSiter%maxit,ierr)<br>
> >
<br>
> <br>
> This code looks essentially right. You should call with -ksp_view to see exactly what PETSc is using for a solver.
<br>
> <br>
> > <br>
> > I understand that the parallel performance may not be comparable, so I first set up a one-process test (with MPIAij, but all the rows are local since there is only one process). The system is solved without any problem (identical results within error).
But the performance is actually a lot worse (code built without debugging flags in performance tests) than my own home-brew implementation in Fortran (I wrote my own ILU0 in CSR sparse matrix format), which is hard to believe. I suspect the difference is from
the PC as the PETSc version took much more BiCGstab iterations (60-ish vs 100-ish) to converge to the same relative tol.
<br>
> <br>
> PETSc uses GMRES by default with a restart of 30 and left preconditioning. It could be different exact choices in the solver (which is why -ksp_view is so useful) can explain the differences in the runs between your code and PETSc's<br>
> > <br>
> > This is further confirmed when I change the setup of D-ILU (using 6 or 9 blocks instead of 3). While my Fortran/Matlab codes see minimal performance difference (<5%) when I play with the D-ILU setup, increasing the number of D-ILU blocks from 3 to 9 caused
the ksp setup with PCBJACOBI to suffer a performance decrease of more than 50% in sequential test.<br>
> <br>
> This is odd, the more blocks the smaller each block so the quicker the ILU set up should be. You can run various cases with -log_view and send us the output to see what is happening at each part of the computation in time.<br>
> <br>
> > So my implementation IS somewhat different in PETSc. Do I miss something in the PCBJACOBI setup? Or do I have some fundamental misunderstanding of how PCBJACOBI works in PETSc?
<br>
> <br>
> Probably not.<br>
> > <br>
> > If this is not the right way to implement a block diagonal ILU as (parallel) PC, please kindly point me to the right direction. I searched through the mail list to find some answers, only to find a couple of similar questions... An example would be nice.<br>
> <br>
> You approach seems fundamentally right but I cannot be sure of possible bugs.<br>
> > <br>
> > On the other hand, does PETSc support a simple way to use explicit L/U matrix as a preconditioner? I can import the D-ILU matrix (I already converted my A matrix into Mat) constructed in my Fortran code to make a better comparison. Or do I have to construct
my own PC using PCshell? If so, is there a good tutorial/example to learn about how to use PCSHELL (in a more advanced sense, like how to setup pc side and transpose)?
<br>
> <br>
> Not sure what you mean by explicit L/U matrix as a preconditioner. As Hong said, yes you can use a parallel LU from MUMPS or SuperLU_DIST or Pastix as the solver. You do not need any shell code. You simply need to set the PCType to lu
<br>
> <br>
> You can also set all this options from the command line and don't need to write the code specifically. So call KSPSetFromOptions() and then for example<br>
> <br>
> -pc_type bjacobi -pc_bjacobi_local_blocks 3 -pc_sub_type ilu (this last one is applied to each block so you could use -pc_type lu and it would use lu on each block.)
<br>
> <br>
> -ksp_type_none -pc_type lu -pc_factor_mat_solver_type mumps (do parallel LU with mumps)<br>
> <br>
> By not hardwiring in the code and just using options you can test out different cases much quicker<br>
> <br>
> Use -ksp_view to make sure that is using the solver the way you expect.<br>
> <br>
> Barry<br>
> <br>
> <br>
> <br>
> Barry<br>
> <br>
> > <br>
> > Thanks in advance, <br>
> > <br>
> > Hao<br>
<br>
</div>
</span></font></div>
</div>
</body>
</html>