[petsc-dev] Adding Seq Dense MAGMA Mat

Tue Aug 27 22:49:01 CDT 2013

> Yes, magma has a function called magma_xgetrf which does that. But the problem with that is, it get the LU factorization done, and copies the matrix back to the host.

   > What they hey? That is so absurd.

Yes it is really absurd. They said in the forum that if you use magma_xgetrf, then it is better to just use LAPACK xgetrs after that.

> So there is no scope for using xgetrs on GPU after using magma_xgetrf. I learned this recently.

  > Does magma provide "utility" functions for moving the matrix to the gpu? Surely it must since magma_xgetrf  moves it down. 

Yes magma does provide these functions.

  > That is the function you want to hook up to in the numerical factorization and then call magma_xgetrf_gpu  and then in the solve download the vector to the gpu and use
    magma_xgetrs_gpu.  For the vector you can just turn on the cusp code and use the mechanism already in place to force it down to the GPU. Unfortunately this may be a little 
    hard to explain in email.

  > Have you built the version with cusp and studied how it moves the vectors up and down between the main memory and gpu as needed and how it triggers the move?

I haven't done this yet. I'll start working on it. But isn't it better to start writing a magma class right away?

Harshad

  > Barry

> 
> I can test if that works, but I don't know if it would be worthwhile to do that.
> 
> Which is why I thought we could take another approach and handle the GPU memory ourselves. Then we can use magma_xgetrf_gpu and magma_xgetrs_gpu to get everything done on the device.
> 
> 
> ----- Original Message -----
> From: "Barry Smith" <bsmith at mcs.anl.gov>
> To: "Harshad Sahasrabudhe" <hsahasra at purdue.edu>
> Cc: "For users of the development version of PETSc" <petsc-dev at mcs.anl.gov>
> Sent: Tuesday, August 27, 2013 11:20:44 PM
> Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat
> 
> 
> Does the LU factorization use magma on the gpu now work? You told us that for factorization magma allocated the memory on the GPU and copied the data over everything over automatically. If the factorization and solve stuff all works then we can talk about the MatMult() and other stuff but otherwise let's get the factorization all working before moving on to the "harder" stuff.
> 
>   barry
> 
> 
> 
> On Aug 27, 2013, at 10:12 PM, Harshad Sahasrabudhe <hsahasra at purdue.edu> wrote:
> 
>> Hi Barry,
>> 
>> The functions MatCreate_SeqDense and MatSeqDenseSetPreallocation_SeqDense do not handle memory allocation on GPU. I can see 2 ways to implement this:
>> 
>> 1) Modify these functions so that they allocate memory on host or device when the solver is magma. I don't know how to do this.
>> 
>> 2) Copy the relevant functions from dense.c to magma.c. Make a magma matrix class which is very similar to the dense matrix class PETSc uses, with some additional flags. Modify the MatMatMult, MatGetFactor, MatSolve functions so that they use magma functions in the proper way. I can do this.
>> 
>> Can you please advise on how I should progress?
>> 
>> Thanks,
>> Harshad
>> 
>> ----- Original Message -----
>> From: "Barry Smith" <bsmith at mcs.anl.gov>
>> To: "Harshad Sahasrabudhe" <hsahasra at purdue.edu>
>> Cc: "For users of the development version of PETSc" <petsc-dev at mcs.anl.gov>
>> Sent: Tuesday, August 27, 2013 8:43:57 AM
>> Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat
>> 
>> 
>>  Both _p_Mat and _p_Vec have void pointers that can be used for the gpu memory as well as flags that could potentially be used. See below
>> 
>> struct _p_Mat {
>> PETSCHEADER(struct _MatOps);
>> ….
>> PetscBool              nooffprocentries,nooffproczerorows;
>> #if defined(PETSC_HAVE_CUSP)
>> PetscCUSPFlag          valid_GPU_matrix; /* flag pointing to the matrix on the gpu*/
>> #endif
>> #if defined(PETSC_HAVE_VIENNACL)
>> PetscViennaCLFlag          valid_GPU_matrix; /* flag pointing to the matrix on the gpu*/
>> #endif
>> void                   *spptr;          /* pointer for special library like SuperLU */
>> 
>> On Aug 27, 2013, at 12:20 AM, Harshad Sahasrabudhe <hsahasra at purdue.edu> wrote:
>> 
>>> Hi Barry,
>>> 
>>> Thanks, MatGetFactor() now much more clear.
>>> 
>>> However, I have run into trouble with the device memory allocation for Mat_SeqDense. How do I figure out whether the allocated memory is on host or device in MatSeqDenseSetPreallocation_SeqDense() ?
>>> 
>>> For some functions such as magma_xgetrf_gpu and magma_xgetrs_gpu, I need the memory allocated on the device. As the preallocation is done in MatSeqDenseSetPreallocation_SeqDense for dense Matrices, I want to write a similar function. But checking for if(!data) is not enough as I need to make sure that the allocated memory is on the device.
>>> 
>>> Is it a good idea to make another Mat class for magma which has 2 pointers, one for the host memory and one for device memory (only one of which should point to data at any given time) ?
>>> 
>>> Harshad
>>> 
>>> ----- Original Message -----
>>> From: "Barry Smith" <bsmith at mcs.anl.gov>
>>> To: "Harshad Sahasrabudhe" <hsahasra at purdue.edu>
>>> Cc: "For users of the development version of PETSc" <petsc-dev at mcs.anl.gov>
>>> Sent: Monday, August 26, 2013 6:19:21 PM
>>> Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat
>>> 
>>> 
>>> On Aug 26, 2013, at 4:24 PM, Harshad Sahasrabudhe <hsahasra at purdue.edu> wrote:
>>> 
>>>> Hi Barry,
>>>> 
>>>> Here is my analysis, please correct if I'm wrong:
>>>> 
>>>> For using the MatGetFactor() mechanism, I need to define MATSOLVERMAGMA in petscmat.h. I also need to define the functions MatFactorGetSolverPackage_SeqDense_magma(), MatGetFactor_SeqDense_magma() in dense.c
>>>> 
>>>> I'm not creating a new matrix class, so I think I don't need to add anything to MatRegisterAll()
>>>> 
>>>> I don't understand the following:
>>>> How does PETSc figure out that there is an additional solver present? Do I need to define any other variables or add anything for this?
>>> 
>>> In MatCreate_SeqDense() note the line 
>>> ierr = PetscObjectComposeFunction((PetscObject)B,"MatGetFactor_petsc_C",MatGetFactor_seqdense_petsc);CHKERRQ(ierr);
>>> 
>>> you will add an additional line (protected by ifdef for magma being available that registers your routine
>>> 
>>> ierr = PetscObjectComposeFunction((PetscObject)B,"MatGetFactor_magma_C",MatGetFactor_seqdense_magma);CHKERRQ(ierr);
>>> 
>>> Barrt
>>> 
>>>> 
>>> 
>>>> What else do I need to change if I have to add the functions MatSolve_SeqDense_magma() and MatMatMult_magma() (for magma_xgemm dense matrix multiplication).
>>>> 
>>>> Harshad
>>>> 
>>>> 
>>>> ----- Original Message -----
>>>> From: "Barry Smith" <bsmith at mcs.anl.gov>
>>>> To: "Harshad Sahasrabudhe" <hsahasra at purdue.edu>
>>>> Cc: "For users of the development version of PETSc" <petsc-dev at mcs.anl.gov>
>>>> Sent: Tuesday, August 20, 2013 7:10:40 PM
>>>> Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat
>>>> 
>>>> 
>>>> How general do you plan for this "magma" matrix class to be? If it all it is for is to do LU/Cholesky factorizations then you do NOT need to introduce src/mat/impls/dense/seq/magma at all. Simply use the MatGetFactor() mechanism to "How will the user be able to access this function?".   This is definitely the easy way to go.
>>>> 
>>>> If you want to use the "magma" matrix class for all kinds of non-factorization matrix operations then you need to write a full magma matrix class. I don't recommend this.
>>>> 
>>>> Barry
>>>> 
>>>> On Aug 20, 2013, at 1:35 PM, Harshad Sahasrabudhe <hsahasra at purdue.edu> wrote:
>>>> 
>>>>> Hi,
>>>>> 
>>>>> I am working on adding the Seq Dense MAGMA matrix type. For this, I have created the following files and directories:
>>>>> 
>>>>> src/mat/impls/dense/seq/magma
>>>>> src/mat/impls/dense/seq/magma/magma.c
>>>>> src/mat/impls/dense/seq/magma/makefile
>>>>> 
>>>>> Right now I am just trying the make LU factorization work through MAGMA. Barry had suggested looking at dense.c, as the MAGMA function for LU has the same sequence as the LAPACK function getrf. MAGMA does the memory allocation on the GPU inside it's function magma_?getrf. So can I directly use the matrix type declared in dense.c, and just define a function MatLUFactor_SeqDense_MAGMA magma.c which uses the same matrix type?
>>>>> 
>>>>> How will the user be able to access this function?
>>>>> 
>>>>> Thanks,
>>>>> Harshad
>>>> 
>>> 
>> 
> 