<html><head></head><body><div style="color:#000; background-color:#fff; font-family:HelveticaNeue-Light, Helvetica Neue Light, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:16px"><div id="yui_3_16_0_ym19_1_1466606883184_15744" class="qtdSeparateBR"><div id="yui_3_16_0_ym19_1_1466606883184_15917">My application already has Intel pardiso solver built-in, but it can only run on a single compute node ( no mpi ). Using a 3million^2 sparse symmetric matrix as a benchmark:</div><div id="yui_3_16_0_ym19_1_1466606883184_16114"><br></div><div id="yui_3_16_0_ym19_1_1466606883184_16115">Pardiso using all 24 cpus on 1 compute nodes factors it in 3 minutes</div><div id="yui_3_16_0_ym19_1_1466606883184_16116"><br></div><div id="yui_3_16_0_ym19_1_1466606883184_16234">Mumps takes:</div><div id="yui_3_16_0_ym19_1_1466606883184_16117"><br></div><div id="yui_3_16_0_ym19_1_1466606883184_16118">24 cpus ( 1 node ) : 15 minutes</div><div id="yui_3_16_0_ym19_1_1466606883184_16119">48 cpus ( 2 nodes ) : 9 minutes</div><div id="yui_3_16_0_ym19_1_1466606883184_16120">72 cpus ( 3 nodes ) : 6 minutes<br></div><div id="yui_3_16_0_ym19_1_1466606883184_16121"><br></div><div id="yui_3_16_0_ym19_1_1466606883184_16122">So mumps does scale well with more cpus, but still can not match pardiso solver on one compute node. I am not sure if this is expected behavior of Mumps or I need to spend more time playing with all the parameters in mumps...</div><div id="yui_3_16_0_ym19_1_1466606883184_16236"><br></div><div><br></div></div><div style="display: block;" id="yui_3_16_0_ym19_1_1466606883184_15750" class="yahoo_quoted"> <div id="yui_3_16_0_ym19_1_1466606883184_15749" style="font-family: HelveticaNeue-Light, Helvetica Neue Light, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 16px;"> <div id="yui_3_16_0_ym19_1_1466606883184_15748" style="font-family: HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 16px;"> <div id="yui_3_16_0_ym19_1_1466606883184_15747" dir="ltr"> <font id="yui_3_16_0_ym19_1_1466606883184_15746" face="Arial" size="2"> <hr size="1"> <b><span style="font-weight:bold;">From:</span></b> Hong <hzhang@mcs.anl.gov><br> <b><span style="font-weight: bold;">To:</span></b> Faraz Hussain <faraz_hussain@yahoo.com> <br><b><span style="font-weight: bold;">Cc:</span></b> Barry Smith <bsmith@mcs.anl.gov>; "petsc-users@mcs.anl.gov" <petsc-users@mcs.anl.gov><br> <b><span style="font-weight: bold;">Sent:</span></b> Wednesday, June 22, 2016 9:53 AM<br> <b><span style="font-weight: bold;">Subject:</span></b> Re: [petsc-users] Receiving DIVERGED_PCSETUP_FAILED<br> </font> </div> <div id="yui_3_16_0_ym19_1_1466606883184_15811" class="y_msg_container"><br><div id="yiv7610608590"><div id="yui_3_16_0_ym19_1_1466606883184_15810"><div id="yui_3_16_0_ym19_1_1466606883184_15809" dir="ltr"><div id="yui_3_16_0_ym19_1_1466606883184_15808" class="yiv7610608590gmail_extra"><div id="yui_3_16_0_ym19_1_1466606883184_15807" class="yiv7610608590gmail_quote">Faraz:<br clear="none"><blockquote id="yui_3_16_0_ym19_1_1466606883184_15806" class="yiv7610608590gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex;"><div id="yui_3_16_0_ym19_1_1466606883184_15805"><div id="yui_3_16_0_ym19_1_1466606883184_15804" style="color:rgb(0,0,0);font-family:HelveticaNeue-Light,;"><div id="yui_3_16_0_ym19_1_1466606883184_15803" dir="ltr">Just an update, I got this to work by rebuilding petsc with parmetis, metis and ptscotch.Then I used these settings for Mumps:</div><div id="yui_3_16_0_ym19_1_1466606883184_16238" dir="ltr"><br clear="none"></div><div id="yui_3_16_0_ym19_1_1466606883184_16239" dir="ltr"> icntl = 28; ival = 2;<br clear="none"> ierr = MatMumpsSetIcntl(F,icntl,ival);CHKERRQ(ierr);<br clear="none"><br clear="none"> icntl = 29; ival = 1;<br clear="none"> ierr = MatMumpsSetIcntl(F,icntl,ival);CHKERRQ(ierr);<br clear="none"></div></div></div></blockquote><div> </div><div>The options use MUMPS parallel symbolic factorization with ptscotch matrix ordering.</div><blockquote class="yiv7610608590gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex;"><div><div style="color:rgb(0,0,0);font-family:HelveticaNeue-Light,;"><div dir="ltr"></div><div dir="ltr"><br clear="none"></div><div dir="ltr">It still took 4X longer to solve than Intel Pardiso. But after re-configuring petsc with-debugging=0, it ran faster. Still slower than Pardiso, but only 2X slower. <br clear="none"></div></div></div></blockquote><div> </div><div>I've seen report that <span style="color:rgb(0,0,0);font-family:HelveticaNeue-Light,;">Intel Pardiso is much faster than mumps, e.g.,</span></div><div><span style="color:rgb(0,0,0);font-family:HelveticaNeue-Light,;">slepc developer Jose sent me following:</span></div><div><span style="font-size:12.8px;">With mumps:</span><br style="font-size:12.8px;" clear="none"><br style="font-size:12.8px;" clear="none"><span style="font-size:12.8px;">MatSolve 16 1.0 1.0962e+01</span><br style="font-size:12.8px;" clear="none"><span style="font-size:12.8px;">MatLUFactorSym 1 1.0 3.1131e+00</span><br style="font-size:12.8px;" clear="none"><span style="font-size:12.8px;">MatLUFactorNum 1 1.0 2.6120e+00</span><br style="font-size:12.8px;" clear="none"><br style="font-size:12.8px;" clear="none"><span style="font-size:12.8px;">With mkl_pardiso:</span><br style="font-size:12.8px;" clear="none"><br style="font-size:12.8px;" clear="none"><span style="font-size:12.8px;">MatSolve 16 1.0 6.4163e-01</span><br style="font-size:12.8px;" clear="none"><span style="font-size:12.8px;">MatLUFactorSym 1 1.0 2.4772e+00</span><br style="font-size:12.8px;" clear="none"><span style="font-size:12.8px;">MatLUFactorNum 1 1.0 8.6419e-01</span><span style="color:rgb(0,0,0);font-family:HelveticaNeue-Light,;"><br clear="none"></span></div><div><span style="font-size:12.8px;"><br clear="none"></span></div><div><span style="font-size:12.8px;">However, petsc only interfaces with sequential mkl_pardiso. Did you get results in parallel or sequential?</span></div><div><span style="font-size:12.8px;"><br clear="none"></span></div><div><span style="font-size:12.8px;">Hong</span></div><blockquote class="yiv7610608590gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex;"><div class="yiv7610608590yqt4836919510" id="yiv7610608590yqtfd70962"><div><div style="color:rgb(0,0,0);font-family:HelveticaNeue-Light,;"><div dir="ltr"></div><div class="yiv7610608590"><div dir="ltr"><br clear="none"></div><div><span><br clear="none"></span></div><div><br clear="none"><br clear="none"></div></div><div><div class="yiv7610608590"> </div><div style="font-family:HelveticaNeue-Light,;"><div class="yiv7610608590"> </div><div style="font-family:HelveticaNeue,;"><div class="yiv7610608590"> </div><div dir="ltr"><div class="yiv7610608590"> </div><font face="Arial" size="2"></font><div class="yiv7610608590"> <hr size="1"> <b><span style="font-weight:bold;">From:</span></b> Faraz Hussain <<a rel="nofollow" shape="rect" ymailto="mailto:faraz_hussain@yahoo.com" target="_blank" href="mailto:faraz_hussain@yahoo.com">faraz_hussain@yahoo.com</a>><br clear="none"> <b><span style="font-weight:bold;">To:</span></b> Barry Smith <<a rel="nofollow" shape="rect" ymailto="mailto:bsmith@mcs.anl.gov" target="_blank" href="mailto:bsmith@mcs.anl.gov">bsmith@mcs.anl.gov</a>> <br clear="none"><b><span style="font-weight:bold;">Cc:</span></b> "<a rel="nofollow" shape="rect" ymailto="mailto:petsc-users@mcs.anl.gov" target="_blank" href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>" <<a rel="nofollow" shape="rect" ymailto="mailto:petsc-users@mcs.anl.gov" target="_blank" href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>><br clear="none"> <b><span style="font-weight:bold;">Sent:</span></b> Friday, June 10, 2016 5:27 PM</div><div><div class="yiv7610608590h5"><br clear="none"> <b><span style="font-weight:bold;">Subject:</span></b> Re: [petsc-users] Receiving DIVERGED_PCSETUP_FAILED<br clear="none"> </div></div> </div><div><div class="yiv7610608590h5"> <div><br clear="none"><div><div><div style="color:rgb(0,0,0);font-family:HelveticaNeue-Light,;"><div dir="ltr">I think the issue is I need to play more with the "parrallel" settings here. <br clear="none"></div><div dir="ltr"><br clear="none"></div><div dir="ltr"><a rel="nofollow" shape="rect" target="_blank" href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MATSOLVERMUMPS.html">http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MATSOLVERMUMPS.html</a></div><div dir="ltr"><br clear="none"></div><div dir="ltr">The example 52.c was based on sequential running. So doing mpiexec -np 48, was basically just using one processor. I also never installed parmetis, metis or ptscotch. <br clear="none"></div><div><br clear="none"><span></span></div><div><span>Will install and adjust the MUMPS settings and hopefully will get it to converge this weekend!</span></div><div><br clear="none"><br clear="none"></div><div><div style="display:block;"> <div style="font-family:HelveticaNeue-Light,;"> <div style="font-family:HelveticaNeue,;"> <div dir="ltr"> <font face="Arial" size="2"> </font><hr size="1"> <b><span style="font-weight:bold;">From:</span></b> Barry Smith <<a rel="nofollow" shape="rect" ymailto="mailto:bsmith@mcs.anl.gov" target="_blank" href="mailto:bsmith@mcs.anl.gov">bsmith@mcs.anl.gov</a>><br clear="none"> <b><span style="font-weight:bold;">To:</span></b> Faraz Hussain <<a rel="nofollow" shape="rect" ymailto="mailto:faraz_hussain@yahoo.com" target="_blank" href="mailto:faraz_hussain@yahoo.com">faraz_hussain@yahoo.com</a>> <br clear="none"><b><span style="font-weight:bold;">Cc:</span></b> "<a rel="nofollow" shape="rect" ymailto="mailto:petsc-users@mcs.anl.gov" target="_blank" href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>" <<a rel="nofollow" shape="rect" ymailto="mailto:petsc-users@mcs.anl.gov" target="_blank" href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>><br clear="none"> <b><span style="font-weight:bold;">Sent:</span></b> Friday, June 10, 2016 4:02 PM<br clear="none"> <b><span style="font-weight:bold;">Subject:</span></b> Re: [petsc-users] Receiving DIVERGED_PCSETUP_FAILED<br clear="none"> </div> <div><br clear="none"><br clear="none">> On Jun 10, 2016, at 3:56 PM, Faraz Hussain <<a rel="nofollow" shape="rect" ymailto="mailto:faraz_hussain@yahoo.com" target="_blank" href="mailto:faraz_hussain@yahoo.com">faraz_hussain@yahoo.com</a>> wrote:<br clear="none">> <br clear="none">> Thanks for the suggestions. I checked but was not able to find out how to change the mumps row pivot threshold of 1e-06. Maybe I will ask on the mumps user forum.<br clear="none"><br clear="none">This might help. <br clear="none"><br clear="none"> ierr = PetscOptionsReal("-mat_mumps_cntl_1","CNTL(1): relative pivoting threshold","None",mumps->id.CNTL(1),&mumps->id.CNTL(1),NULL);CHKERRQ(ierr);<br clear="none"> ierr = PetscOptionsReal("-mat_mumps_cntl_2","CNTL(2): stopping criterion of refinement","None",mumps->id.CNTL(2),&mumps->id.CNTL(2),NULL);CHKERRQ(ierr);<br clear="none"> ierr = PetscOptionsReal("-mat_mumps_cntl_3","CNTL(3): absolute pivoting threshold","None",mumps->id.CNTL(3),&mumps->id.CNTL(3),NULL);CHKERRQ(ierr);<br clear="none"> ierr = PetscOptionsReal("-mat_mumps_cntl_4","CNTL(4): value for static pivoting","None",mumps->id.CNTL(4),&mumps->id.CNTL(4),NULL);CHKERRQ(ierr);<br clear="none"><br clear="none">Note I don't know what they mean so you need to read the mumps docs.<br clear="none"><br clear="none">> Regarding:<br clear="none">> <br clear="none">> > You need to look at the condition number just before GMRES reaches the restart. It has to start all over again at the restart. So what was the estimated condition number at 999 iterations?<br clear="none">> <br clear="none">> I ran again and the condition number at 999 iterations is: <br clear="none">> <br clear="none">> 999 KSP preconditioned resid norm 5.921717188418e-02 true resid norm 5.921717188531e-02 ||r(i)||/||b|| 4.187286380279e-03 <br clear="none">> 999 KSP Residual norm 5.921717188418e-02 % max 1.070338898624e+05 min 1.002755075294e-01 max/min 1.067398136390e+06<br clear="none"><br clear="none">Ok, so relatively ill-conditioned matrix. But seemingly not terrible.<br clear="none"><br clear="none"> Barry<div><br clear="none"><br clear="none">> <br clear="none">> <br clear="none">> From: Barry Smith <<a rel="nofollow" shape="rect" ymailto="mailto:bsmith@mcs.anl.gov" target="_blank" href="mailto:bsmith@mcs.anl.gov">bsmith@mcs.anl.gov</a>><br clear="none">> To: Faraz Hussain <<a rel="nofollow" shape="rect" ymailto="mailto:faraz_hussain@yahoo.com" target="_blank" href="mailto:faraz_hussain@yahoo.com">faraz_hussain@yahoo.com</a>> <br clear="none">> Cc: "<a rel="nofollow" shape="rect" ymailto="mailto:petsc-users@mcs.anl.gov" target="_blank" href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>" <<a rel="nofollow" shape="rect" ymailto="mailto:petsc-users@mcs.anl.gov" target="_blank" href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>><br clear="none">> Sent: Thursday, June 9, 2016 5:56 PM<br clear="none">> Subject: Re: [petsc-users] Receiving DIVERGED_PCSETUP_FAILED<br clear="none">> <br clear="none">> <br clear="none">> > On Jun 9, 2016, at 3:32 PM, Faraz Hussain <<a rel="nofollow" shape="rect" ymailto="mailto:faraz_hussain@yahoo.com" target="_blank" href="mailto:faraz_hussain@yahoo.com">faraz_hussain@yahoo.com</a>> wrote:<br clear="none">> > <br clear="none">> > I have been following ex52.c ksp/ksp/examples/tutorials to use MUMPS to directly solve Ax=b. My matrix is symmetric and positive definite. I built a small cantilever beam model with matrix of 5000^2 size. It solves in 2 seconds and gives correct answer. But when I use a finer mesh of the cantilever beam with 3.3 million^2 matrix, I get the following error:<br clear="none">> > <br clear="none">> > Mumps row pivot threshhold = 1e-06<br clear="none">> <br clear="none">> Maybe you can change this to get MUMPS to pivot less aggressively. Doing lots of pivoting will require a lot more memory. In theory since it is SPD it should not need to pivot at all.<br clear="none">> <br clear="none">> > Mumps determinant = (0., 0.) * 2^0<br clear="none">> > Linear solve did not converge due to DIVERGED_PCSETUP_FAILED iterations 0<br clear="none">> > PCSETUP_FAILED due to FACTOR_OUTMEMORY<br clear="none">> > Norm of error inf. iterations 0<br clear="none">> > <br clear="none">> > It runs for more than an hour before aborting with this message. I am running it with this command:<br clear="none">> > <br clear="none">> > mpiexec -hostfile ./hostfile -np 48 ./ex12 -ksp_converged_reason<br clear="none">> > <br clear="none">> > My machines have 24 cpus and 125GB Ram. When I do "top" I see it correctly spans 48 processes on 2 nodes. The memory usage of each process is no more than 1-2GB. So I do not understand why it gives FACTOR_OUTMEMORY ?<br clear="none">> > <br clear="none">> > The same matrix solves in under 5 minutes in Intel Pardiso using 24 cpus on one host. <br clear="none">> <br clear="none">> Mumps may be (likely?) is using a different matrix ordering then Intel Pardiso. Unfortunately each of these packages have a different way of asking for orderings and different orderings to chose from so you will need to look at the details for each package.<br clear="none">> <br clear="none">> > I thought maybe mumps thinks it is ill-conditioned? The model does converge in the iterative solver in 4000 iterations. I also tried running with these options per the FAQ on <br clear="none">> > <br clear="none">> > " How can I determine the condition number of a matrix? ".<br clear="none">> > <br clear="none">> > mpiexec -hostfile ./hostfile -np 48 ./ex12 -pc_type none -ksp_type gmres -ksp_monitor_singular_value -ksp_gmres_restart 1000 -ksp_converged_reason -ksp_monitor_true_residual<br clear="none">> > <br clear="none">> > After 1337 iterations I cancelled it, and the output was:<br clear="none">> <br clear="none">> You need to look at the condition number just before GMRES reaches the restart. It has to start all over again at the restart. So what was the estimated condition number at 999 iterations?<br clear="none">> <br clear="none">> It could be that Intel Pardiso produced a low quality solution if the matrix is ill conditioned. You can run with -ksp_type gmres -ksp_max_it 5 -ksp_monitor_true_residual with -pc_type lu to see how small the residuals are after the "direct" solver.<br clear="none">> <br clear="none">> Barry<br clear="none">> <br clear="none">> <br clear="none">> > <br clear="none">> > 1337 KSP preconditioned resid norm 5.647402411074e-02 true resid norm 5.647402411074e-02 ||r(i)||/||b|| 3.993316540960e-03<br clear="none">> > 1337 KSP Residual norm 5.647402411074e-02 % max 1.070324243277e+05 min 1.220336631740e-01 max/min 8.770729448238e+05<br clear="none">> <br clear="none">> <br clear="none"></div><br clear="none"><br clear="none"></div> </div> </div> </div></div></div></div></div><br clear="none"><br clear="none"></div> </div></div></div> </div> </div></div></div></div></blockquote></div><div class="yiv7610608590yqt4836919510" id="yiv7610608590yqtfd51052"><br clear="none"></div></div></div></div></div><br><br></div> </div> </div> </div></div></body></html>