[petsc-dev] CUDA GAMG coarse grid solver

[Smith, Barry F.]

> On Jul 21, 2019, at 8:55 AM, Mark Adams via petsc-dev <petsc-dev at mcs.anl.gov> wrote:
> 
> I am running ex56 with -ex56_dm_vec_type cuda -ex56_dm_mat_type aijcusparse and I see no GPU communication in MatSolve (the serial LU coarse grid solver).

   Do you mean to say, you DO see communication?

   What does -ksp_view should you? It should show the factor type in the information about the coarse grid solve?

   You might need something like -mg_coarse_mat_solver_type cusparse  (because it may default to the PETSc one, it may be possible to have it default to the cusparse if it exists and the matrix is of type MATSEQAIJCUSPARSE).

   The determination of the MatGetFactor() is a bit involved including pasting together strings and string compares and could be finding a CPU factorization.

   I could run on one MPI_Rank() in the debugger and put a break point in MatGetFactor() and track along to see what it picks and why. You could do this debugging without GAMG first, just -pc_type lu  

> GAMG does set the coarse grid solver to LU manually like this: ierr = PCSetType(pc2, PCLU);CHKERRQ(ierr);

  For parallel runs this won't work using the GPU code and only sequential direct solvers, so it must using BJACOBI in that case?

   Barry



   

> I am thinking the dispatch of the CUDA version of this got dropped somehow. 
> 
> I see that this is getting called:
> 
> PETSC_EXTERN PetscErrorCode MatSolverTypeRegister_CUSPARSE(void)
> {
>   PetscErrorCode ierr;
> 
>   PetscFunctionBegin;
>   ierr = MatSolverTypeRegister(MATSOLVERCUSPARSE,MATSEQAIJCUSPARSE,MAT_FACTOR_LU,MatGetFactor_seqaijcusparse_cusparse);CHKERRQ(ierr);
>   ierr = MatSolverTypeRegister(MATSOLVERCUSPARSE,MATSEQAIJCUSPARSE,MAT_FACTOR_CHOLESKY,MatGetFactor_seqaijcusparse_cusparse);CHKERRQ(ierr);
>   ierr = MatSolverTypeRegister(MATSOLVERCUSPARSE,MATSEQAIJCUSPARSE,MAT_FACTOR_ILU,MatGetFactor_seqaijcusparse_cusparse);CHKERRQ(ierr);
>   ierr = MatSolverTypeRegister(MATSOLVERCUSPARSE,MATSEQAIJCUSPARSE,MAT_FACTOR_ICC,MatGetFactor_seqaijcusparse_cusparse);CHKERRQ(ierr);
>   PetscFunctionReturn(0);
> }
> 
> but MatGetFactor_seqaijcusparse_cusparse is not getting  called.
> 
> GAMG does set the coarse grid solver to LU manually like this: ierr = PCSetType(pc2, PCLU);CHKERRQ(ierr);
> 
> Any ideas?
> 
> Thanks,
> Mark
> 
>