<html><head></head><body><div style="color:#000; background-color:#fff; font-family:HelveticaNeue-Light, Helvetica Neue Light, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:16px"><div dir="ltr" id="yui_3_16_0_ym19_1_1465587369138_27318"><span id="yui_3_16_0_ym19_1_1465587369138_27384">Thanks for the suggestions. I checked but was not able to find out how to change the mumps </span>row pivot threshold of 1e-06. Maybe I will ask on the mumps user forum. Regarding:</div><div id="yui_3_16_0_ym19_1_1465587369138_27425" dir="ltr"><br id="yui_3_16_0_ym19_1_1465587369138_27407" clear="none"> > You need to look at the condition number just before
GMRES reaches the restart. It has to start all over again at the
restart. So what was the estimated condition number at 999 iterations?</div><div id="yui_3_16_0_ym19_1_1465587369138_27426" dir="ltr"><br></div><div id="yui_3_16_0_ym19_1_1465587369138_27427" dir="ltr">I ran again and the condition number at 999 iterations is: <br></div><div id="yui_3_16_0_ym19_1_1465587369138_27440" dir="ltr"><br></div><div id="yui_3_16_0_ym19_1_1465587369138_27460" dir="ltr">999 KSP preconditioned resid norm 5.921717188418e-02 true resid norm 5.921717188531e-02 ||r(i)||/||b|| 4.187286380279e-03 <br></div><div id="yui_3_16_0_ym19_1_1465587369138_27461" dir="ltr">999 KSP Residual norm 5.921717188418e-02 % max 1.070338898624e+05 min 1.002755075294e-01 max/min 1.067398136390e+06</div><div id="yui_3_16_0_ym19_1_1465587369138_27319" class="qtdSeparateBR"><br><br></div><div style="display: block;" id="yui_3_16_0_ym19_1_1465587369138_27325" class="yahoo_quoted"> <div id="yui_3_16_0_ym19_1_1465587369138_27324" style="font-family: HelveticaNeue-Light, Helvetica Neue Light, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 16px;"> <div id="yui_3_16_0_ym19_1_1465587369138_27323" style="font-family: HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 16px;"> <div id="yui_3_16_0_ym19_1_1465587369138_27322" dir="ltr"> <font id="yui_3_16_0_ym19_1_1465587369138_27321" face="Arial" size="2"> <hr size="1"> <b id="yui_3_16_0_ym19_1_1465587369138_27340"><span id="yui_3_16_0_ym19_1_1465587369138_27339" style="font-weight:bold;">From:</span></b> Barry Smith <bsmith@mcs.anl.gov><br> <b id="yui_3_16_0_ym19_1_1465587369138_27338"><span id="yui_3_16_0_ym19_1_1465587369138_27337" style="font-weight: bold;">To:</span></b> Faraz Hussain <faraz_hussain@yahoo.com> <br><b><span style="font-weight: bold;">Cc:</span></b> "petsc-users@mcs.anl.gov" <petsc-users@mcs.anl.gov><br> <b id="yui_3_16_0_ym19_1_1465587369138_27336"><span id="yui_3_16_0_ym19_1_1465587369138_27335" style="font-weight: bold;">Sent:</span></b> Thursday, June 9, 2016 5:56 PM<br> <b id="yui_3_16_0_ym19_1_1465587369138_27334"><span id="yui_3_16_0_ym19_1_1465587369138_27333" style="font-weight: bold;">Subject:</span></b> Re: [petsc-users] Receiving DIVERGED_PCSETUP_FAILED<br> </font> </div> <div id="yui_3_16_0_ym19_1_1465587369138_27328" class="y_msg_container"><br><br clear="none">> On Jun 9, 2016, at 3:32 PM, Faraz Hussain <<a id="yui_3_16_0_ym19_1_1465587369138_27327" shape="rect" ymailto="mailto:faraz_hussain@yahoo.com" href="mailto:faraz_hussain@yahoo.com">faraz_hussain@yahoo.com</a>> wrote:<br clear="none">> <br clear="none">> I have been following ex52.c ksp/ksp/examples/tutorials to use MUMPS to directly solve Ax=b. My matrix is symmetric and positive definite. I built a small cantilever beam model with matrix of 5000^2 size. It solves in 2 seconds and gives correct answer. But when I use a finer mesh of the cantilever beam with 3.3 million^2 matrix, I get the following error:<br clear="none">> <br clear="none">> Mumps row pivot threshhold = 1e-06<br clear="none"><br clear="none"> Maybe you can change this to get MUMPS to pivot less aggressively. Doing lots of pivoting will require a lot more memory. In theory since it is SPD it should not need to pivot at all.<br clear="none"><br clear="none">> Mumps determinant = (0., 0.) * 2^0<br clear="none">> Linear solve did not converge due to DIVERGED_PCSETUP_FAILED iterations 0<br clear="none">> PCSETUP_FAILED due to FACTOR_OUTMEMORY<br clear="none">> Norm of error inf. iterations 0<br clear="none">> <br clear="none">> It runs for more than an hour before aborting with this message. I am running it with this command:<br clear="none">> <br clear="none">> mpiexec -hostfile ./hostfile -np 48 ./ex12 -ksp_converged_reason<br clear="none">> <br clear="none">> My machines have 24 cpus and 125GB Ram. When I do "top" I see it correctly spans 48 processes on 2 nodes. The memory usage of each process is no more than 1-2GB. So I do not understand why it gives FACTOR_OUTMEMORY ?<br clear="none">> <br clear="none">> The same matrix solves in under 5 minutes in Intel Pardiso using 24 cpus on one host. <br clear="none"><br clear="none"> Mumps may be (likely?) is using a different matrix ordering then Intel Pardiso. Unfortunately each of these packages have a different way of asking for orderings and different orderings to chose from so you will need to look at the details for each package.<br clear="none"><br clear="none">> I thought maybe mumps thinks it is ill-conditioned? The model does converge in the iterative solver in 4000 iterations. I also tried running with these options per the FAQ on <br clear="none">> <br clear="none">> " How can I determine the condition number of a matrix? ".<br clear="none">> <br clear="none">> mpiexec -hostfile ./hostfile -np 48 ./ex12 -pc_type none -ksp_type gmres -ksp_monitor_singular_value -ksp_gmres_restart 1000 -ksp_converged_reason -ksp_monitor_true_residual<br clear="none">> <br clear="none">> After 1337 iterations I cancelled it, and the output was:<br clear="none"><br clear="none"> You need to look at the condition number just before GMRES reaches the restart. It has to start all over again at the restart. So what was the estimated condition number at 999 iterations?<br clear="none"><br clear="none"> It could be that Intel Pardiso produced a low quality solution if the matrix is ill conditioned. You can run with -ksp_type gmres -ksp_max_it 5 -ksp_monitor_true_residual with -pc_type lu to see how small the residuals are after the "direct" solver.<br clear="none"><br clear="none"> Barry<div class="yqt1805321203" id="yqtfd73515"><br clear="none"><br clear="none">> <br clear="none">> 1337 KSP preconditioned resid norm 5.647402411074e-02 true resid norm 5.647402411074e-02 ||r(i)||/||b|| 3.993316540960e-03<br clear="none">> 1337 KSP Residual norm 5.647402411074e-02 % max 1.070324243277e+05 min 1.220336631740e-01 max/min 8.770729448238e+05<br clear="none"></div><br><br></div> </div> </div> </div></div></body></html>