[petsc-users] Why does CHOLMOD solution time differs between PETSc and MATLAB?

[Barry Smith]

  Victor,

    Here is my conjecture (more than a guess, but less than a full statement of fact). CHOLMOD uses a "supernodel" algorithm which means that it works on dense blocks of the factored matrix and can take advantage of BLAS 2 and 3 operations (unlike almost everything in PETSc).

    You have installed PETSc to use the --download-fblaslapack which is just the reference implementation of BLAS/LAPACK in Fortran. Meanwhile MATLAB uses the Intel MKL libraries for its BLAS/LAPACK

   1) the MKL BLAS/LAPACK are likely much faster than the Fortran versions
   2) it is possible the MKL BLAS/LAPACK used by MATLAB use multiple threads to take advantage of multiple cores on the hardware.

  So to improve your performance of the PETSc version you need to switch to MKL (and possibly turn on multithreaded version of MKL).

  No MKL installed on your machine? If you have MATLAB on the machine then you have MKL, for example on my Mac it is at /Applications/MATLAB_R2014b.app/bin/maci64/mkl.dylib  and /Applications/MATLAB_R2014b.app/bin/maci64/mklcompat.dylib  Also note that MATLAB's MKL uses 64 bit integers so you need to configure PETSc with the additional option --known-64-bit-blas-indices

  I think you are likely to get close to the MATLAB performance of CHOLMOD if you switch to using MKL blas/lapack with multithreading.


  Barry


   
> On Nov 21, 2014, at 11:23 AM, Victor Magri <victor.antonio.magri at gmail.com> wrote:
> 
> Hi,
> 
> I'm solving a linear system which comes from the discretization of an elliptic equation. The mesh has 1 million control volumes and there are two more constraints for the system.
> 
> First, I've solved it in MATLAB using backslash command, which in turn calls CHOLMOD once the matrix is SPD. Using tic and toc funtions, I could see that it takes about 2s for the system to be solved (See attached for more details).
> 
> I wrote both matrix and rhs in PETSc binary format and solved the linear system using the example problem defined in ksp/examples/tutorials/ex10.c. The command line options were:
> 
> mpiexec -n -1 ./ex10 -f A1mi.dat -rhs rhs1mi.dat -log_summary -ksp_monitos -ksp_view -ksp_type preonly -pc_type cholesky -pc_factor_mat_solver_package cholmod
> 
> The configuration parameters for CHOLMOD in PETSc and MATLAB seems the same, so I was expecting a similar computation time, but it took about 13s to solve in PETSc. (See attached for more details).
> 
> Why does CHOLMOD solution time differs between them?
> 
> If someone wants to try for yourself, here are the binaries for the matrix and the rhs, respectively.
> https://www.dropbox.com/s/q9s6mlrmv1qdxon/A1mi.dat?dl=0
> https://www.dropbox.com/s/2zadcbjg5d9bycy/rhs1mi.dat?dl=0
> 
> 
> 
> <PetscOutput.dat><MatlabOutput.dat>