[petsc-users] SLEPC: partitioning for CISS

[Jose E. Roman]

If you run with debugging enabled you will get an error at EPSSetOperators() saying that "eps" and "matrix_petsc" have different communicators. You cannot pass a sequential matrix to a parallel solver.

Jose


> El 11 sept 2019, a las 5:46, Povolotskyi, Mykhailo via petsc-users <petsc-users at mcs.anl.gov> escribió:
> 
> Hello,
> I'm currently using CISS via SLEPc.
> I would like to use parallelization by partitioning over quadrature points, using the option -eps_ciss_partitions.
> 
> I have done the following:
> 
> 1. created a matrix on each MPI rank 
> 
>  MatCreateDense(MPI_COMM_SELF, matrix_size, matrix_size, PETSC_DECIDE,PETSC_DECIDE,NULL,&matrix_petsc);
> 
> 2. Created EPS object
> 
> EPSCreate(MPI_COMM_WORLD,&eps);
> EPSSetOperators( eps,matrix_petsc,NULL);
> EPSSetType(eps,EPSCISS);
> EPSSetProblemType(eps, EPS_NHEP);
>       
> EPSSetFromOptions(eps);
>       
>      
> EPSGetRG(eps,&rg);
> RGSetType(rg,RGRING);
>      
> RGRingSetParameters(rg,center,radius,vscale,start_ang,end_ang,width);
> 
> EPSSolve(eps);
> 
> 3. Then I run the code:
> 
> mpiexec -n 2 ./a.out -eps_ciss_partitions 2
> 
> The code gives an error:
> 
> [1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [1]PETSC ERROR: Arguments are incompatible
> [1]PETSC ERROR: MatMatMultSymbolic requires A, seqdense, to be compatible with B, mpidense
> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [1]PETSC ERROR: Petsc Release Version 3.8.4, Mar, 24, 2018 
> [1]PETSC ERROR: ../lib/a.out on a linux-complex named brown-a337.rcac.purdue.edu by mpovolot Tue Sep 10 23:43:44 2019
> [1]PETSC ERROR: Configure options --with-scalar-type=complex --with-x=0 --with-hdf5 --download-hdf5=1 --with-single-library=1 --with-pic=1 --with-shared-libraries=0 --with-log=0 --with-clanguage=C++ --CXXFLAGS="-fopenmp -fPIC" --CFLAGS="-fopenmp -fPIC" --with-fortran=0 --FFLAGS="-fopenmp -fPIC" --with-debugging=0 --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpicxx COPTFLAGS= CXXOPTFLAGS= FOPTFLAGS= --download-metis=1 --download-parmetis=1 --with-valgrind-dir=/apps/brown/valgrind/3.13.0_gcc-4.8.5 --download-mumps=1 --with-fortran-kernels=0 --download-superlu_dist=1 --with-blaslapack-lib="-L/apps/cent7/intel/compilers_and_libraries_2017.1.132/linux/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_gnu_thread -lmkl_core " --with-blacs-lib=/apps/cent7/intel/compilers_and_libraries_2017.1.132/linux/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.so --with-blacs-include=/apps/cent7/intel/compilers_and_libraries_2017.1.132/linux/mkl/include --with-scalapack-lib="-Wl,-rpath,/apps/cent7/intel/compilers_and_libraries_2017.1.132/linux/mkl/lib/intel64 -L/apps/cent7/intel/compilers_and_libraries_2017.1.132/linux/mkl/lib/intel64 -lmkl_gf_lp64 -lmkl_gnu_thread -lmkl_core  -lpthread  -L/apps/cent7/intel/compilers_and_libraries_2017.1.132/linux/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64" --with-scalapack-include=/apps/cent7/intel/compilers_and_libraries_2017.1.132/linux/mkl/include
> [1]PETSC ERROR: #463 MatMatMultSymbolic() line 9692 in /depot/kildisha/apps/brown/nemo5/libs/petsc/build-cplx/src/mat/interface/matrix.c
> [1]PETSC ERROR: #464 BVMatMult_Svec() line 229 in /depot/kildisha/apps/brown/nemo5/libs/slepc/build-cplx/src/sys/classes/bv/impls/svec/svec.c
> [1]PETSC ERROR: #465 BVMatMult() line 589 in /depot/kildisha/apps/brown/nemo5/libs/slepc/build-cplx/src/sys/classes/bv/interface/bvops.c
> [1]PETSC ERROR: #466 BVMatProject_MatMult() line 903 in /depot/kildisha/apps/brown/nemo5/libs/slepc/build-cplx/src/sys/classes/bv/interface/bvglobal.c
> [1]PETSC ERROR: #467 BVMatProject() line 1151 in /depot/kildisha/apps/brown/nemo5/libs/slepc/build-cplx/src/sys/classes/bv/interface/bvglobal.c
> [1]PETSC ERROR: #468 EPSSolve_CISS() line 1066 in /depot/kildisha/apps/brown/nemo5/libs/slepc/build-cplx/src/eps/impls/ciss/ciss.c
> [1]PETSC ERROR: #469 EPSSolve() line 147 in /depot/kildisha/apps/brown/nemo5/libs/slepc/build-cplx/src/eps/interface/epssolve.c
> 
> Could you, please, tell me what am I doing wrong?
> 
> 
>