[petsc-users] [SLEPc] Performance of Krylov-Schur with MUMPS-based shift-and-invert

[Thibaut Appel]

Good afternoon,

I am solving generalized eigenvalue problems {Ax = omegaBx} in complex 
arithmetic, where A is non-hermitian and B is singular. I think the only 
way to get round the singularity is to employ a shift-and-invert method, 
where I am using MUMPS to invert the shifted matrix.

I am using the Fortran interface of PETSc 3.8.3 and SLEPc 3.8.2 where my 
./configure line was
./configure --with-fortran-kernels=1 --with-scalar-type=complex 
--with-blaslapack-dir=/home/linuxbrew/.linuxbrew/opt/openblas 
--PETSC_ARCH=cplx_dble_optim 
--with-cmake-dir=/home/linuxbrew/.linuxbrew/opt/cmake 
--with-mpi-dir=/home/linuxbrew/.linuxbrew/opt/openmpi --with-debugging=0 
--download-scalapack --download-mumps --COPTFLAGS="-O3 -march=native" 
--CXXOPTFLAGS="-O3 -march=native" --FOPTFLAGS="-O3 -march=native"

My matrices A and B are assembled correctly in parallel and my 
preallocation is quasi-optimal in the sense that I don't have any called 
to mallocs but I may overestimate the required memory for some rows of 
the matrices. Here is how I setup the EPS problem and solve:

     CALL EPSSetProblemType(eps,EPS_GNHEP,ierr)
     CALL EPSSetOperators(eps,MatA,MatB,ierr)
     CALL EPSSetType(eps,EPSKRYLOVSCHUR,ierr)
     CALL EPSSetDimensions(eps,nev,ncv,PETSC_DECIDE,ierr)
     CALL EPSSetTolerances(eps,tol_ev,PETSC_DECIDE,ierr)

     CALL EPSSetFromOptions(eps,ierr)
     CALL EPSSetTarget(eps,shift,ierr)
     CALL EPSSetWhichEigenpairs(eps,EPS_TARGET_MAGNITUDE,ierr)

     CALL EPSGetST(eps,st,ierr)
     CALL STGetKSP(st,ksp,ierr)
     CALL KSPGetPC(ksp,pc,ierr)

     CALL STSetType(st,STSINVERT,ierr)
     CALL KSPSetType(ksp,KSPPREONLY,ierr)
     CALL PCSetType(pc,PCLU,ierr)

     CALL PCFactorSetMatSolverPackage(pc,MATSOLVERMUMPS,ierr)
     CALL PCSetFromOptions(pc,ierr)

     CALL EPSSolve(eps,ierr)
     CALL EPSGetIterationNumber(eps,iter,ierr)
     CALL EPSGetConverged(eps,nev_conv,ierr)

     - Using one MPI process, it takes 1 hour and 22 minutes to retrieve 
250 eigenvalues with a Krylov subspace of size 500, a tolerance of 
10^-12 when the leading dimension of the matrices is 405000. My matrix A 
has 98,415,000 non-zero elements and B has 1,215,000 non zero elements. 
Would you be shocked by that computation time? I would have expected 
something much lower given the values of nev and ncv I have but could be 
completely wrong in my understanding of the Krylov-Schur method.

     - My goal is speed and reliability. Is there anything you notice in 
my EPS solver that could be improved or corrected? I remember an 
exchange with Jose E. Roman where he said that the parameters of MUMPS 
are not worth being changed, however I notice some people play with the 
-mat_mumps_cntl_1 and  -mat_mumps_cntl_3 which control the 
relative/absolute pivoting threshold?

     - Would you advise the use of EPSSetTrueResidual and EPSSetBalance 
since I am using a spectral transformation?

     - Would you see anything that would prevent me from getting speedup 
in parallel executions?

Thank you very much in advance and I look forward to exchanging with you 
about these different points,

Thibaut