[petsc-dev] trying to make MKL work with GAMG

[Smith, Barry F.]

> On Jul 4, 2018, at 9:40 AM, Mark Adams <mfadams at lbl.gov> wrote:
> 
> "-mat_seqaij_type seqaijmkl" just worked.

    Mark,

      Please clarify. Does this mean you can use -mat_seqaij_type seqaijmkl to satisfy all your needs now without changing any code? 

   Barry


> On Wed, Jul 4, 2018 at 9:44 AM Mark Adams <mfadams at lbl.gov> wrote:
> 
> 
> On Wed, Jul 4, 2018 at 3:01 AM Richard Tran Mills <rtmills at anl.gov> wrote:
> Hi Mark,
> 
> I'd never looked at the code for MatCreateMPIAIJWithSeqAIJ(), but it looks like if you are using MPIAIJ matrices but you've set "-mat_seqaij_type seqaijmkl",
> 
> No, I am using -mat_type aijmkl
>  
> then the "diagonal" portion of the MPIAIJ matrix ends up as a SEQAIJMKL instance, but the off-diagonal portion ends up as a plain SEQAIJ when it is created with MatCreateSeqAIJWithArrays(). (I have not stepped through this with a debugger to verify that this is actually what happens.) So, if we want the off-diagonal matrix to also be a SEQAIJMKL, then we either need to add some logic after the MatCreateSeqAIJWithArrays() call to check to see if a different subtype of SEQAIJ is being used and then convert it to the correct type, or we need a variant of MatCreateSeqAIJWithArrays() that will honor the subtype of SEQAIJ that we are using. Barry, since you added the concept of subtyping for SEQAIJ, what do you prefer?
> 
> No preference.
>  
> I note that in some (most?) cases, I think we are OK or even better off with the off-diagonal matrix in the MPIAIJ not being a SEQAIJMKL one, because the PETSc routines can used the "compressed row" stuff to speed up handling the many zero rows that often end up in the off-diagonal, whereas the MKL routines cannot.
> 
> 
> OK , I'll keep that in mind. I pushed a branch with fixes to get my gamg tests working (ksp ex56 primarily). I'm going to make a new branch and try "-mat_seqaij_type seqaijmkl".
> 
> Thanks,
> Mark
>  
> --Richard
> 
> On Tue, Jul 3, 2018 at 4:31 PM, Mark Adams <mfadams at lbl.gov> wrote:
> I have having to fix AIJ methods that don't get the type from the args to set created matrix type, so as to keep the MKL typing.
> 
> I am not sure how to fix this because I don't know to say 'make MPI from SEQ' in this method
> 
> PetscErrorCode MatCreateMPIAIJWithSeqAIJ(MPI_Comm comm,Mat A,Mat B,const PetscInt garray[],Mat *mat)
> {
>  ....  
>   ierr = MatCreate(comm,mat);CHKERRQ(ierr);
>   ierr = MatGetSize(A,&m,&n);CHKERRQ(ierr);
> .... 
>   ierr = MatSetSizes(*mat,m,n,PETSC_DECIDE,N);CHKERRQ(ierr);
>   ierr = MatSetType(*mat,MATMPIAIJ);CHKERRQ(ierr);
> 
> How should I do this?  Other places in the code I get the type from input (MPI) args and set created matrix types accordingly. I could just put a switch with an error for any new types that might come along in the future ...
> 
> Mark
>