[petsc-dev] Adding Seq Dense MAGMA Mat

Satish Balay balay at mcs.anl.gov
Wed Aug 28 15:45:49 CDT 2013 

Didn't we discuss this issue before?

Perhaps dense.c is supporsed to have:

#if !defined(HAVE_CUBLAS)
#define HAVE_CUBLAS
#endif
#include "magma.h"

Satish


>>>>>>>>
/usr/bin/gcc  -Dpetsc_EXPORTS -D__INSDIR__="" -malign-double -fPIC -g3 -fno-inline -O0   -fPIC -I/home/harshad/workspace/petsc-hg/include -I/home/harshad/workspace/petsc-hg/arch-linux2-c-debug/include -I/home/harshad/Downloads/magma-1.4.0/include -I/home/harshad/cuda-5.5/include -I/home/harshad/workspace/petsc-hg/include/mpiuni    -o CMakeFiles/petsc.dir/src/mat/impls/dense/seq/dense.c.o   -c /home/harshad/workspace/petsc-hg/src/mat/impls/dense/seq/dense.c
In file included from /home/harshad/Downloads/magma-1.4.0/include/magmablas_z.h:14:0,
                 from /home/harshad/Downloads/magma-1.4.0/include/magmablas.h:12,
                 from /home/harshad/Downloads/magma-1.4.0/include/magma.h:15,
                 from /home/harshad/workspace/petsc-hg/src/mat/impls/dense/seq/dense.c:10:
/home/harshad/Downloads/magma-1.4.0/include/magma_types.h:146:6: error: #error "One of HAVE_CUBLAS, HAVE_clAmdBlas, or HAVE_MIC must be defined. This typically happens in Makefile.internal."
<<<<<<<

On Wed, 28 Aug 2013, Harshad Sahasrabudhe wrote:

> Attached make.log
> 
> ----- Original Message -----
> From: "Harshad Sahasrabudhe" <hsahasra at purdue.edu>
> To: "petsc-dev" <petsc-dev at mcs.anl.gov>
> Cc: "Barry Smith" <bsmith at mcs.anl.gov>
> Sent: Wednesday, August 28, 2013 4:33:23 PM
> Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat
> 
> I'm getting errors when compiling with MAGMA and without the #requirespackage tag in the makefile. It builds correctly if I have #requirespackage in the makefile. There are lots of these errors with different type names:
> 
> magma-1.4.0/include/magma_s.h:607:38: error: unknown type name ‘magma_queue_t’
> 
> Harshad
> 
> ----- Original Message -----
> From: "Satish Balay" <balay at mcs.anl.gov>
> To: "Harshad Sahasrabudhe" <hsahasra at purdue.edu>
> Cc: "Barry Smith" <bsmith at mcs.anl.gov>, "For users of the development version of PETSc" <petsc-dev at mcs.anl.gov>
> Sent: Wednesday, August 28, 2013 4:26:20 PM
> Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat
> 
> If you have conditional code in dense.c - then you don't need
> "#requirespackage 'PETSC_HAVE_LIBMAGMA'" in the makefile
> 
> Are you getting errors when compiling with Magma or without magma?
> What errors?
> 
> Satish
> 
> 
> On Wed, 28 Aug 2013, Harshad Sahasrabudhe wrote:
> 
> > Hi Barry,
> > 
> > I'm trying to compile dense.c with the MAGMA code in it. Now the MAGMA code is protected by #if defined statements so it is only compiled when we have MAGMA.
> > 
> > When we have MAGMA, I get errors while compiling, if we don't have #requirespackage 'PETSC_HAVE_LIBMAGMA' in the Makefile. We obviously don't require MAGMA to compile dense.c. Is there any other way to tell make that MAGMA lib has to be used for compiling dense.c when it is present?
> > 
> > I have attached the makefile inside src/mat/impls/dense/seq folder.
> > 
> > Thanks,
> > Harshad
> > 
> > ----- Original Message -----
> > From: "Barry Smith" <bsmith at mcs.anl.gov>
> > To: "Harshad Sahasrabudhe" <hsahasra at purdue.edu>
> > Cc: "For users of the development version of PETSc" <petsc-dev at mcs.anl.gov>
> > Sent: Tuesday, August 27, 2013 11:20:44 PM
> > Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat
> > 
> > 
> > Does the LU factorization use magma on the gpu now work? You told us that for factorization magma allocated the memory on the GPU and copied the data over everything over automatically. If the factorization and solve stuff all works then we can talk about the MatMult() and other stuff but otherwise let's get the factorization all working before moving on to the "harder" stuff.
> > 
> >    barry
> > 
> > 
> > 
> > On Aug 27, 2013, at 10:12 PM, Harshad Sahasrabudhe <hsahasra at purdue.edu> wrote:
> > 
> > > Hi Barry,
> > > 
> > > The functions MatCreate_SeqDense and MatSeqDenseSetPreallocation_SeqDense do not handle memory allocation on GPU. I can see 2 ways to implement this:
> > > 
> > > 1) Modify these functions so that they allocate memory on host or device when the solver is magma. I don't know how to do this.
> > > 
> > > 2) Copy the relevant functions from dense.c to magma.c. Make a magma matrix class which is very similar to the dense matrix class PETSc uses, with some additional flags. Modify the MatMatMult, MatGetFactor, MatSolve functions so that they use magma functions in the proper way. I can do this.
> > > 
> > > Can you please advise on how I should progress?
> > > 
> > > Thanks,
> > > Harshad
> > > 
> > > ----- Original Message -----
> > > From: "Barry Smith" <bsmith at mcs.anl.gov>
> > > To: "Harshad Sahasrabudhe" <hsahasra at purdue.edu>
> > > Cc: "For users of the development version of PETSc" <petsc-dev at mcs.anl.gov>
> > > Sent: Tuesday, August 27, 2013 8:43:57 AM
> > > Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat
> > > 
> > > 
> > >   Both _p_Mat and _p_Vec have void pointers that can be used for the gpu memory as well as flags that could potentially be used. See below
> > > 
> > > struct _p_Mat {
> > >  PETSCHEADER(struct _MatOps);
> > > ….
> > >  PetscBool              nooffprocentries,nooffproczerorows;
> > > #if defined(PETSC_HAVE_CUSP)
> > >  PetscCUSPFlag          valid_GPU_matrix; /* flag pointing to the matrix on the gpu*/
> > > #endif
> > > #if defined(PETSC_HAVE_VIENNACL)
> > >  PetscViennaCLFlag          valid_GPU_matrix; /* flag pointing to the matrix on the gpu*/
> > > #endif
> > >  void                   *spptr;          /* pointer for special library like SuperLU */
> > > 
> > > On Aug 27, 2013, at 12:20 AM, Harshad Sahasrabudhe <hsahasra at purdue.edu> wrote:
> > > 
> > >> Hi Barry,
> > >> 
> > >> Thanks, MatGetFactor() now much more clear.
> > >> 
> > >> However, I have run into trouble with the device memory allocation for Mat_SeqDense. How do I figure out whether the allocated memory is on host or device in MatSeqDenseSetPreallocation_SeqDense() ?
> > >> 
> > >> For some functions such as magma_xgetrf_gpu and magma_xgetrs_gpu, I need the memory allocated on the device. As the preallocation is done in MatSeqDenseSetPreallocation_SeqDense for dense Matrices, I want to write a similar function. But checking for if(!data) is not enough as I need to make sure that the allocated memory is on the device.
> > >> 
> > >> Is it a good idea to make another Mat class for magma which has 2 pointers, one for the host memory and one for device memory (only one of which should point to data at any given time) ?
> > >> 
> > >> Harshad
> > >> 
> > >> ----- Original Message -----
> > >> From: "Barry Smith" <bsmith at mcs.anl.gov>
> > >> To: "Harshad Sahasrabudhe" <hsahasra at purdue.edu>
> > >> Cc: "For users of the development version of PETSc" <petsc-dev at mcs.anl.gov>
> > >> Sent: Monday, August 26, 2013 6:19:21 PM
> > >> Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat
> > >> 
> > >> 
> > >> On Aug 26, 2013, at 4:24 PM, Harshad Sahasrabudhe <hsahasra at purdue.edu> wrote:
> > >> 
> > >>> Hi Barry,
> > >>> 
> > >>> Here is my analysis, please correct if I'm wrong:
> > >>> 
> > >>> For using the MatGetFactor() mechanism, I need to define MATSOLVERMAGMA in petscmat.h. I also need to define the functions MatFactorGetSolverPackage_SeqDense_magma(), MatGetFactor_SeqDense_magma() in dense.c
> > >>> 
> > >>> I'm not creating a new matrix class, so I think I don't need to add anything to MatRegisterAll()
> > >>> 
> > >>> I don't understand the following:
> > >>> How does PETSc figure out that there is an additional solver present? Do I need to define any other variables or add anything for this?
> > >> 
> > >>  In MatCreate_SeqDense() note the line 
> > >> ierr = PetscObjectComposeFunction((PetscObject)B,"MatGetFactor_petsc_C",MatGetFactor_seqdense_petsc);CHKERRQ(ierr);
> > >> 
> > >> you will add an additional line (protected by ifdef for magma being available that registers your routine
> > >> 
> > >> ierr = PetscObjectComposeFunction((PetscObject)B,"MatGetFactor_magma_C",MatGetFactor_seqdense_magma);CHKERRQ(ierr);
> > >> 
> > >>  Barrt
> > >> 
> > >>> 
> > >> 
> > >>> What else do I need to change if I have to add the functions MatSolve_SeqDense_magma() and MatMatMult_magma() (for magma_xgemm dense matrix multiplication).
> > >>> 
> > >>> Harshad
> > >>> 
> > >>> 
> > >>> ----- Original Message -----
> > >>> From: "Barry Smith" <bsmith at mcs.anl.gov>
> > >>> To: "Harshad Sahasrabudhe" <hsahasra at purdue.edu>
> > >>> Cc: "For users of the development version of PETSc" <petsc-dev at mcs.anl.gov>
> > >>> Sent: Tuesday, August 20, 2013 7:10:40 PM
> > >>> Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat
> > >>> 
> > >>> 
> > >>> How general do you plan for this "magma" matrix class to be? If it all it is for is to do LU/Cholesky factorizations then you do NOT need to introduce src/mat/impls/dense/seq/magma at all. Simply use the MatGetFactor() mechanism to "How will the user be able to access this function?".   This is definitely the easy way to go.
> > >>> 
> > >>> If you want to use the "magma" matrix class for all kinds of non-factorization matrix operations then you need to write a full magma matrix class. I don't recommend this.
> > >>> 
> > >>> Barry
> > >>> 
> > >>> On Aug 20, 2013, at 1:35 PM, Harshad Sahasrabudhe <hsahasra at purdue.edu> wrote:
> > >>> 
> > >>>> Hi,
> > >>>> 
> > >>>> I am working on adding the Seq Dense MAGMA matrix type. For this, I have created the following files and directories:
> > >>>> 
> > >>>> src/mat/impls/dense/seq/magma
> > >>>> src/mat/impls/dense/seq/magma/magma.c
> > >>>> src/mat/impls/dense/seq/magma/makefile
> > >>>> 
> > >>>> Right now I am just trying the make LU factorization work through MAGMA. Barry had suggested looking at dense.c, as the MAGMA function for LU has the same sequence as the LAPACK function getrf. MAGMA does the memory allocation on the GPU inside it's function magma_?getrf. So can I directly use the matrix type declared in dense.c, and just define a function MatLUFactor_SeqDense_MAGMA magma.c which uses the same matrix type?
> > >>>> 
> > >>>> How will the user be able to access this function?
> > >>>> 
> > >>>> Thanks,
> > >>>> Harshad
> > >>> 
> > >> 
> > > 
> > 
> > 
>

More information about the petsc-dev mailing list