[petsc-dev] CUDA GAMG coarse grid solver

[Mark Adams]

Barry, I do NOT see communication. This is what made me think it was not
running on the GPU. I added print statements and found that
MatSolverTypeRegister_CUSPARSE IS called but (what it registers)
MatGetFactor_seqaijcusparse_cusparse does NOT get called.

I have a job waiting on the queue. I'll send ksp_view when it runs. I will
try -mg_coarse_mat_solver_type cusparse. That is probably the problem.
Maybe I should set the coarse grid solver in a more robust way in GAMG,
like use the matrix somehow? I currently use PCSetType(pc, PCLU).

I can't get an interactive shell now to run DDT, but I can try stepping
through from MatGetFactor to see what its doing.

Thanks,
Mark

On Sun, Jul 21, 2019 at 11:14 AM Smith, Barry F. <bsmith at mcs.anl.gov> wrote:

>
>
> > On Jul 21, 2019, at 8:55 AM, Mark Adams via petsc-dev <
> petsc-dev at mcs.anl.gov> wrote:
> >
> > I am running ex56 with -ex56_dm_vec_type cuda -ex56_dm_mat_type
> aijcusparse and I see no GPU communication in MatSolve (the serial LU
> coarse grid solver).
>
>    Do you mean to say, you DO see communication?
>
>    What does -ksp_view should you? It should show the factor type in the
> information about the coarse grid solve?
>
>    You might need something like -mg_coarse_mat_solver_type cusparse
> (because it may default to the PETSc one, it may be possible to have it
> default to the cusparse if it exists and the matrix is of type
> MATSEQAIJCUSPARSE).
>
>    The determination of the MatGetFactor() is a bit involved including
> pasting together strings and string compares and could be finding a CPU
> factorization.
>
>    I could run on one MPI_Rank() in the debugger and put a break point in
> MatGetFactor() and track along to see what it picks and why. You could do
> this debugging without GAMG first, just -pc_type lu
>
> > GAMG does set the coarse grid solver to LU manually like this: ierr =
> PCSetType(pc2, PCLU);CHKERRQ(ierr);
>
>   For parallel runs this won't work using the GPU code and only sequential
> direct solvers, so it must using BJACOBI in that case?
>
>    Barry
>
>
>
>
>
> > I am thinking the dispatch of the CUDA version of this got dropped
> somehow.
> >
> > I see that this is getting called:
> >
> > PETSC_EXTERN PetscErrorCode MatSolverTypeRegister_CUSPARSE(void)
> > {
> >   PetscErrorCode ierr;
> >
> >   PetscFunctionBegin;
> >   ierr =
> MatSolverTypeRegister(MATSOLVERCUSPARSE,MATSEQAIJCUSPARSE,MAT_FACTOR_LU,MatGetFactor_seqaijcusparse_cusparse);CHKERRQ(ierr);
> >   ierr =
> MatSolverTypeRegister(MATSOLVERCUSPARSE,MATSEQAIJCUSPARSE,MAT_FACTOR_CHOLESKY,MatGetFactor_seqaijcusparse_cusparse);CHKERRQ(ierr);
> >   ierr =
> MatSolverTypeRegister(MATSOLVERCUSPARSE,MATSEQAIJCUSPARSE,MAT_FACTOR_ILU,MatGetFactor_seqaijcusparse_cusparse);CHKERRQ(ierr);
> >   ierr =
> MatSolverTypeRegister(MATSOLVERCUSPARSE,MATSEQAIJCUSPARSE,MAT_FACTOR_ICC,MatGetFactor_seqaijcusparse_cusparse);CHKERRQ(ierr);
> >   PetscFunctionReturn(0);
> > }
> >
> > but MatGetFactor_seqaijcusparse_cusparse is not getting  called.
> >
> > GAMG does set the coarse grid solver to LU manually like this: ierr =
> PCSetType(pc2, PCLU);CHKERRQ(ierr);
> >
> > Any ideas?
> >
> > Thanks,
> > Mark
> >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20190721/4bbe4f49/attachment.html>