<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Fri, Nov 21, 2014 at 4:49 PM, Barry Smith <span dir="ltr"><<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Here is my conjecture (more than a guess, but less than a full statement of fact). CHOLMOD uses a "supernodel" algorithm which means that it works on dense blocks of the factored matrix and can take advantage of BLAS 2 and 3 operations (unlike almost everything in PETSc).</blockquote></div><br>I though that PETSc would take more advantage from BLAS 2.</div><div class="gmail_extra"><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span style="font-family:arial,sans-serif;font-size:13px">1) the MKL BLAS/LAPACK are likely much faster than the Fortran versions</span></blockquote><div><br></div><div>Your conjecture was right! I compiled PETSc to use mkl.so from my Matlab folder and run the tests again. The computational time was pretty the same in comparison with MATLAB (See attached). </div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span style="font-family:arial,sans-serif;font-size:13px"> 2) it is possible the MKL BLAS/LAPACK used by MATLAB use multiple threads to take advantage of multiple cores on the hardware.</span></blockquote><div><br></div><div>At least in my computer, no. I opened the system monitor to see if MATLAB would use more than one thread and it didn't.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span style="font-family:arial,sans-serif;font-size:13px">I think CHOLMOD, used by MATLAB backslash operator, uses AMD (approximate min. degree) ordering scheme by default</span></blockquote><div><br></div><div>Yes, I searched about that and it really does. </div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span style="font-family:arial,sans-serif;font-size:13px">For PETSc + MATLAB, use the option -pc_factor_mat_ordering_type qmd </span></blockquote><div><br></div><div>The computation time got bigger! I was thinking that reordering would make it lower. Maybe it doesn't because this matrix comes from a 5-point discretization stencil... I tried also RCM and ND but the best time came from natural.</div><div><br></div><div>Thank you so much for helping, Barry and Abhyankar.</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Nov 21, 2014 at 6:34 PM, Barry Smith <span dir="ltr"><<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class=""><br>
> On Nov 21, 2014, at 1:46 PM, Abhyankar, Shrirang G. <<a href="mailto:abhyshr@mcs.anl.gov">abhyshr@mcs.anl.gov</a>> wrote:<br>
><br>
> I think this may have to do with the matrix reodering scheme. I think CHOLMOD, used by MATLAB backslash operator, uses AMD (approximate min. degree) ordering scheme by default. PETSc's default reordering scheme is nested dissection. For PETSc + MATLAB, use the option -pc_factor_mat_ordering_type qmd<br>
<br>
</span> This is possible and definitely worth trying with different orders, but I checked the nonzeros in the matrix and the L factor for both approaches and they are very similar<br>
<br>
PETSc call:<br>
Common.fl 1.81877e+10 (flop count from most recent analysis)<br>
Common.lnz 4.46748e+07 (fundamental nz in L)<br>
<br>
MatLab call:<br>
flop count: 1.8188e+10<br>
nnz(L): 4.4675e+07<br>
<br>
Unless I am misunderstanding something both factors likely look very similar.<br>
<span class="HOEnZb"><font color="#888888"><br>
Barry<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
<br>
><br>
> Shri<br>
><br>
> On Nov 21, 2014, at 11:23 AM, "Victor Magri" <<a href="mailto:victor.antonio.magri@gmail.com">victor.antonio.magri@gmail.com</a>> wrote:<br>
><br>
>> Hi,<br>
>><br>
>> I'm solving a linear system which comes from the discretization of an elliptic equation. The mesh has 1 million control volumes and there are two more constraints for the system.<br>
>><br>
>> First, I've solved it in MATLAB using backslash command, which in turn calls CHOLMOD once the matrix is SPD. Using tic and toc funtions, I could see that it takes about 2s for the system to be solved (See attached for more details).<br>
>><br>
>> I wrote both matrix and rhs in PETSc binary format and solved the linear system using the example problem defined in ksp/examples/tutorials/ex10.c. The command line options were:<br>
>><br>
>> mpiexec -n -1 ./ex10 -f A1mi.dat -rhs rhs1mi.dat -log_summary -ksp_monitos -ksp_view -ksp_type preonly -pc_type cholesky -pc_factor_mat_solver_package cholmod<br>
>><br>
>> The configuration parameters for CHOLMOD in PETSc and MATLAB seems the same, so I was expecting a similar computation time, but it took about 13s to solve in PETSc. (See attached for more details).<br>
>><br>
>> Why does CHOLMOD solution time differs between them?<br>
>><br>
>> If someone wants to try for yourself, here are the binaries for the matrix and the rhs, respectively.<br>
>> <a href="https://www.dropbox.com/s/q9s6mlrmv1qdxon/A1mi.dat?dl=0" target="_blank">https://www.dropbox.com/s/q9s6mlrmv1qdxon/A1mi.dat?dl=0</a><br>
>> <a href="https://www.dropbox.com/s/2zadcbjg5d9bycy/rhs1mi.dat?dl=0" target="_blank">https://www.dropbox.com/s/2zadcbjg5d9bycy/rhs1mi.dat?dl=0</a><br>
>><br>
>><br>
>><br>
>> <PetscOutput.dat><br>
>> <MatlabOutput.dat><br>
<br>
</div></div></blockquote></div><br></div>