[petsc-dev] Adding Seq Dense MAGMA Mat

[Barry Smith]

   Both _p_Mat and _p_Vec have void pointers that can be used for the gpu memory as well as flags that could potentially be used. See below

struct _p_Mat {
  PETSCHEADER(struct _MatOps);
….
  PetscBool              nooffprocentries,nooffproczerorows;
#if defined(PETSC_HAVE_CUSP)
  PetscCUSPFlag          valid_GPU_matrix; /* flag pointing to the matrix on the gpu*/
#endif
#if defined(PETSC_HAVE_VIENNACL)
  PetscViennaCLFlag          valid_GPU_matrix; /* flag pointing to the matrix on the gpu*/
#endif
  void                   *spptr;          /* pointer for special library like SuperLU */

On Aug 27, 2013, at 12:20 AM, Harshad Sahasrabudhe <hsahasra at purdue.edu> wrote:

> Hi Barry,
> 
> Thanks, MatGetFactor() now much more clear.
> 
> However, I have run into trouble with the device memory allocation for Mat_SeqDense. How do I figure out whether the allocated memory is on host or device in MatSeqDenseSetPreallocation_SeqDense() ?
> 
> For some functions such as magma_xgetrf_gpu and magma_xgetrs_gpu, I need the memory allocated on the device. As the preallocation is done in MatSeqDenseSetPreallocation_SeqDense for dense Matrices, I want to write a similar function. But checking for if(!data) is not enough as I need to make sure that the allocated memory is on the device.
> 
> Is it a good idea to make another Mat class for magma which has 2 pointers, one for the host memory and one for device memory (only one of which should point to data at any given time) ?
> 
> Harshad
> 
> ----- Original Message -----
> From: "Barry Smith" <bsmith at mcs.anl.gov>
> To: "Harshad Sahasrabudhe" <hsahasra at purdue.edu>
> Cc: "For users of the development version of PETSc" <petsc-dev at mcs.anl.gov>
> Sent: Monday, August 26, 2013 6:19:21 PM
> Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat
> 
> 
> On Aug 26, 2013, at 4:24 PM, Harshad Sahasrabudhe <hsahasra at purdue.edu> wrote:
> 
>> Hi Barry,
>> 
>> Here is my analysis, please correct if I'm wrong:
>> 
>> For using the MatGetFactor() mechanism, I need to define MATSOLVERMAGMA in petscmat.h. I also need to define the functions MatFactorGetSolverPackage_SeqDense_magma(), MatGetFactor_SeqDense_magma() in dense.c
>> 
>> I'm not creating a new matrix class, so I think I don't need to add anything to MatRegisterAll()
>> 
>> I don't understand the following:
>> How does PETSc figure out that there is an additional solver present? Do I need to define any other variables or add anything for this?
> 
>   In MatCreate_SeqDense() note the line 
>  ierr = PetscObjectComposeFunction((PetscObject)B,"MatGetFactor_petsc_C",MatGetFactor_seqdense_petsc);CHKERRQ(ierr);
> 
> you will add an additional line (protected by ifdef for magma being available that registers your routine
> 
> ierr = PetscObjectComposeFunction((PetscObject)B,"MatGetFactor_magma_C",MatGetFactor_seqdense_magma);CHKERRQ(ierr);
> 
>   Barrt
> 
>> 
> 
>> What else do I need to change if I have to add the functions MatSolve_SeqDense_magma() and MatMatMult_magma() (for magma_xgemm dense matrix multiplication).
>> 
>> Harshad
>> 
>> 
>> ----- Original Message -----
>> From: "Barry Smith" <bsmith at mcs.anl.gov>
>> To: "Harshad Sahasrabudhe" <hsahasra at purdue.edu>
>> Cc: "For users of the development version of PETSc" <petsc-dev at mcs.anl.gov>
>> Sent: Tuesday, August 20, 2013 7:10:40 PM
>> Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat
>> 
>> 
>>  How general do you plan for this "magma" matrix class to be? If it all it is for is to do LU/Cholesky factorizations then you do NOT need to introduce src/mat/impls/dense/seq/magma at all. Simply use the MatGetFactor() mechanism to "How will the user be able to access this function?".   This is definitely the easy way to go.
>> 
>>  If you want to use the "magma" matrix class for all kinds of non-factorization matrix operations then you need to write a full magma matrix class. I don't recommend this.
>> 
>>  Barry
>> 
>> On Aug 20, 2013, at 1:35 PM, Harshad Sahasrabudhe <hsahasra at purdue.edu> wrote:
>> 
>>> Hi,
>>> 
>>> I am working on adding the Seq Dense MAGMA matrix type. For this, I have created the following files and directories:
>>> 
>>> src/mat/impls/dense/seq/magma
>>> src/mat/impls/dense/seq/magma/magma.c
>>> src/mat/impls/dense/seq/magma/makefile
>>> 
>>> Right now I am just trying the make LU factorization work through MAGMA. Barry had suggested looking at dense.c, as the MAGMA function for LU has the same sequence as the LAPACK function getrf. MAGMA does the memory allocation on the GPU inside it's function magma_?getrf. So can I directly use the matrix type declared in dense.c, and just define a function MatLUFactor_SeqDense_MAGMA magma.c which uses the same matrix type?
>>> 
>>> How will the user be able to access this function?
>>> 
>>> Thanks,
>>> Harshad
>> 
>