[petsc-dev] Adding Seq Dense MAGMA Mat
hsahasra@purdue.edu
hsahasra at purdue.edu
Thu Aug 29 11:25:25 CDT 2013
But PETSc defines PETSC_HAVE_LIBMAGMA in petscconf.h by default. I thought it would be PETSC_HAVE_MAGMA but it wasn't.
----- Reply message -----
From: "Satish Balay" <balay at mcs.anl.gov>
To: "Harshad Sahasrabudhe" <hsahasra at purdue.edu>
Cc: "petsc-dev" <petsc-dev at mcs.anl.gov>
Subject: [petsc-dev] Adding Seq Dense MAGMA Mat
Date: Thu, Aug 29, 2013 12:14 PM
For one change it to PETSC_HAVE_MAGMA
And make sure you add it after a petsc include.
i.e:
>>>>>
#include <../src/mat/impls/dense/seq/dense.h> /*I "petscmat.h" I*/
#include <petscblaslapack.h>
#include <../src/mat/impls/aij/seq/aij.h>
#if defined(PETSC_HAVE_MAGMA)
#if !defined(HAVE_CUBLAS)
#define HAVE_CUBLAS
#endif
#include "magma.h"
#endif
<<<<<<
Satish
On Thu, 29 Aug 2013, Harshad Sahasrabudhe wrote:
> Hi Satish,
>
> It worked out. The HAVE_CUBLAS problem was still there. I'm not really sure why it wasn't defining HAVE_CUBLAS before. I tested the code by putting it outside PETSC_HAVE_LIBAMAGMA #ifdef and it compiled. I was also getting some BLAS LAPACK mangling errors, as ADD_ etc. was supposed to be defined. But they went away after I put in HAVE_CUBLAS.
>
> I don't know why they have designed MAGMA like this. It's a very wrong way to let the user set these flags.
>
> I got the code compiled and now I'm testing it.
>
> Thanks,
> Harshad
>
> ----- Original Message -----
> From: "Satish Balay" <balay at mcs.anl.gov>
> To: "Harshad Sahasrabudhe" <hsahasra at purdue.edu>
> Cc: "petsc-dev" <petsc-dev at mcs.anl.gov>
> Sent: Thursday, August 29, 2013 12:01:41 PM
> Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat
>
> Lets step back.
>
> Since you are adding conditional #ifdefs to dense.c - its not
> appropriate to add #requirespackage to the corresponding makefile.
> [as dense.c should be compiled with or without magma]
>
> Secondly - you should use PETSC_HAVE_MAGMA - not PETSC_HAVE_LIBMAGMA
>
> If you still have errors - send the logs again.
>
> Also send the following:
>
> dense.c
>
> preprocessed dense.c. i.e:
>
> cd src/mat/impl/dense/seq
> make dense.o CFLAGS=-E
>
> [the preprocessed output might be in dense.o]
>
> Satish
>
> On Wed, 28 Aug 2013, Harshad Sahasrabudhe wrote:
>
> > I even tried this with a fresh git pull from Hg repository. I added magma.py for configuring with magma. Then I added only the following in dense.c
> >
> > #if defined(PETSC_HAVE_LIBMAGMA)
> > #if !defined(HAVE_CUBLAS)
> > #define HAVE_CUBLAS
> > #endif
> > #include "magma.h"
> > #endif
> >
> > Then I followed the procedure I mentioned below, and I can reproduce the problem. Now I'm certain that removing #requirespackage is causing it.
> >
> > 1) remove #requirespackage from src/mat/impl/dense/seq/makefile
> > 2) remove build directory
> > 3) remove externalpackages directory
> > 4) configure && make
> > I get these errors.
> > 5) add #requirespackage to src/mat/impl/dense/seq/makefile
> > 6) remove build directory
> > 7) remove externalpackages directory
> > 8) configure && make
> > No errors.
> >
> > Harshad
> >
> >
> >
> > >>>>>>>>
> > /usr/bin/gcc -Dpetsc_EXPORTS -D__INSDIR__="" -malign-double -fPIC -g3 -fno-inline -O0 -fPIC -I/home/harshad/workspace/petsc-hg/include -I/home/harshad/workspace/petsc-hg/arch-linux2-c-debug/include -I/home/harshad/Downloads/magma-1.4.0/include -I/home/harshad/cuda-5.5/include -I/home/harshad/workspace/petsc-hg/include/mpiuni -o CMakeFiles/petsc.dir/src/mat/impls/dense/seq/dense.c.o -c /home/harshad/workspace/petsc-hg/src/mat/impls/dense/seq/dense.c
> > In file included from /home/harshad/Downloads/magma-1.4.0/include/magmablas_z.h:14:0,
> > from /home/harshad/Downloads/magma-1.4.0/include/magmablas.h:12,
> > from /home/harshad/Downloads/magma-1.4.0/include/magma.h:15,
> > from /home/harshad/workspace/petsc-hg/src/mat/impls/dense/seq/dense.c:10:
> > /home/harshad/Downloads/magma-1.4.0/include/magma_types.h:146:6: error: #error "One of HAVE_CUBLAS, HAVE_clAmdBlas, or HAVE_MIC must be defined. This typically happens in Makefile.internal."
> > <<<<<<<
> >
> > On Wed, 28 Aug 2013, Harshad Sahasrabudhe wrote:
> >
> > > Attached make.log
> > >
> > > ----- Original Message -----
> > > From: "Harshad Sahasrabudhe" <hsahasra at purdue.edu>
> > > To: "petsc-dev" <petsc-dev at mcs.anl.gov>
> > > Cc: "Barry Smith" <bsmith at mcs.anl.gov>
> > > Sent: Wednesday, August 28, 2013 4:33:23 PM
> > > Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat
> > >
> > > I'm getting errors when compiling with MAGMA and without the #requirespackage tag in the makefile. It builds correctly if I have #requirespackage in the makefile. There are lots of these errors with different type names:
> > >
> > > magma-1.4.0/include/magma_s.h:607:38: error: unknown type name ‘magma_queue_t’
> > >
> > > Harshad
> > >
> > > ----- Original Message -----
> > > From: "Satish Balay" <balay at mcs.anl.gov>
> > > To: "Harshad Sahasrabudhe" <hsahasra at purdue.edu>
> > > Cc: "Barry Smith" <bsmith at mcs.anl.gov>, "For users of the development version of PETSc" <petsc-dev at mcs.anl.gov>
> > > Sent: Wednesday, August 28, 2013 4:26:20 PM
> > > Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat
> > >
> > > If you have conditional code in dense.c - then you don't need
> > > "#requirespackage 'PETSC_HAVE_LIBMAGMA'" in the makefile
> > >
> > > Are you getting errors when compiling with Magma or without magma?
> > > What errors?
> > >
> > > Satish
> > >
> > >
> > > On Wed, 28 Aug 2013, Harshad Sahasrabudhe wrote:
> > >
> > > > Hi Barry,
> > > >
> > > > I'm trying to compile dense.c with the MAGMA code in it. Now the MAGMA code is protected by #if defined statements so it is only compiled when we have MAGMA.
> > > >
> > > > When we have MAGMA, I get errors while compiling, if we don't have #requirespackage 'PETSC_HAVE_LIBMAGMA' in the Makefile. We obviously don't require MAGMA to compile dense.c. Is there any other way to tell make that MAGMA lib has to be used for compiling dense.c when it is present?
> > > >
> > > > I have attached the makefile inside src/mat/impls/dense/seq folder.
> > > >
> > > > Thanks,
> > > > Harshad
> > > >
> > > > ----- Original Message -----
> > > > From: "Barry Smith" <bsmith at mcs.anl.gov>
> > > > To: "Harshad Sahasrabudhe" <hsahasra at purdue.edu>
> > > > Cc: "For users of the development version of PETSc" <petsc-dev at mcs.anl.gov>
> > > > Sent: Tuesday, August 27, 2013 11:20:44 PM
> > > > Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat
> > > >
> > > >
> > > > Does the LU factorization use magma on the gpu now work? You told us that for factorization magma allocated the memory on the GPU and copied the data over everything over automatically. If the factorization and solve stuff all works then we can talk about the MatMult() and other stuff but otherwise let's get the factorization all working before moving on to the "harder" stuff.
> > > >
> > > > barry
> > > >
> > > >
> > > >
> > > > On Aug 27, 2013, at 10:12 PM, Harshad Sahasrabudhe <hsahasra at purdue.edu> wrote:
> > > >
> > > > > Hi Barry,
> > > > >
> > > > > The functions MatCreate_SeqDense and MatSeqDenseSetPreallocation_SeqDense do not handle memory allocation on GPU. I can see 2 ways to implement this:
> > > > >
> > > > > 1) Modify these functions so that they allocate memory on host or device when the solver is magma. I don't know how to do this.
> > > > >
> > > > > 2) Copy the relevant functions from dense.c to magma.c. Make a magma matrix class which is very similar to the dense matrix class PETSc uses, with some additional flags. Modify the MatMatMult, MatGetFactor, MatSolve functions so that they use magma functions in the proper way. I can do this.
> > > > >
> > > > > Can you please advise on how I should progress?
> > > > >
> > > > > Thanks,
> > > > > Harshad
> > > > >
> > > > > ----- Original Message -----
> > > > > From: "Barry Smith" <bsmith at mcs.anl.gov>
> > > > > To: "Harshad Sahasrabudhe" <hsahasra at purdue.edu>
> > > > > Cc: "For users of the development version of PETSc" <petsc-dev at mcs.anl.gov>
> > > > > Sent: Tuesday, August 27, 2013 8:43:57 AM
> > > > > Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat
> > > > >
> > > > >
> > > > > Both _p_Mat and _p_Vec have void pointers that can be used for the gpu memory as well as flags that could potentially be used. See below
> > > > >
> > > > > struct _p_Mat {
> > > > > PETSCHEADER(struct _MatOps);
> > > > > ….
> > > > > PetscBool nooffprocentries,nooffproczerorows;
> > > > > #if defined(PETSC_HAVE_CUSP)
> > > > > PetscCUSPFlag valid_GPU_matrix; /* flag pointing to the matrix on the gpu*/
> > > > > #endif
> > > > > #if defined(PETSC_HAVE_VIENNACL)
> > > > > PetscViennaCLFlag valid_GPU_matrix; /* flag pointing to the matrix on the gpu*/
> > > > > #endif
> > > > > void *spptr; /* pointer for special library like SuperLU */
> > > > >
> > > > > On Aug 27, 2013, at 12:20 AM, Harshad Sahasrabudhe <hsahasra at purdue.edu> wrote:
> > > > >
> > > > >> Hi Barry,
> > > > >>
> > > > >> Thanks, MatGetFactor() now much more clear.
> > > > >>
> > > > >> However, I have run into trouble with the device memory allocation for Mat_SeqDense. How do I figure out whether the allocated memory is on host or device in MatSeqDenseSetPreallocation_SeqDense() ?
> > > > >>
> > > > >> For some functions such as magma_xgetrf_gpu and magma_xgetrs_gpu, I need the memory allocated on the device. As the preallocation is done in MatSeqDenseSetPreallocation_SeqDense for dense Matrices, I want to write a similar function. But checking for if(!data) is not enough as I need to make sure that the allocated memory is on the device.
> > > > >>
> > > > >> Is it a good idea to make another Mat class for magma which has 2 pointers, one for the host memory and one for device memory (only one of which should point to data at any given time) ?
> > > > >>
> > > > >> Harshad
> > > > >>
> > > > >> ----- Original Message -----
> > > > >> From: "Barry Smith" <bsmith at mcs.anl.gov>
> > > > >> To: "Harshad Sahasrabudhe" <hsahasra at purdue.edu>
> > > > >> Cc: "For users of the development version of PETSc" <petsc-dev at mcs.anl.gov>
> > > > >> Sent: Monday, August 26, 2013 6:19:21 PM
> > > > >> Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat
> > > > >>
> > > > >>
> > > > >> On Aug 26, 2013, at 4:24 PM, Harshad Sahasrabudhe <hsahasra at purdue.edu> wrote:
> > > > >>
> > > > >>> Hi Barry,
> > > > >>>
> > > > >>> Here is my analysis, please correct if I'm wrong:
> > > > >>>
> > > > >>> For using the MatGetFactor() mechanism, I need to define MATSOLVERMAGMA in petscmat.h. I also need to define the functions MatFactorGetSolverPackage_SeqDense_magma(), MatGetFactor_SeqDense_magma() in dense.c
> > > > >>>
> > > > >>> I'm not creating a new matrix class, so I think I don't need to add anything to MatRegisterAll()
> > > > >>>
> > > > >>> I don't understand the following:
> > > > >>> How does PETSc figure out that there is an additional solver present? Do I need to define any other variables or add anything for this?
> > > > >>
> > > > >> In MatCreate_SeqDense() note the line
> > > > >> ierr = PetscObjectComposeFunction((PetscObject)B,"MatGetFactor_petsc_C",MatGetFactor_seqdense_petsc);CHKERRQ(ierr);
> > > > >>
> > > > >> you will add an additional line (protected by ifdef for magma being available that registers your routine
> > > > >>
> > > > >> ierr = PetscObjectComposeFunction((PetscObject)B,"MatGetFactor_magma_C",MatGetFactor_seqdense_magma);CHKERRQ(ierr);
> > > > >>
> > > > >> Barrt
> > > > >>
> > > > >>>
> > > > >>
> > > > >>> What else do I need to change if I have to add the functions MatSolve_SeqDense_magma() and MatMatMult_magma() (for magma_xgemm dense matrix multiplication).
> > > > >>>
> > > > >>> Harshad
> > > > >>>
> > > > >>>
> > > > >>> ----- Original Message -----
> > > > >>> From: "Barry Smith" <bsmith at mcs.anl.gov>
> > > > >>> To: "Harshad Sahasrabudhe" <hsahasra at purdue.edu>
> > > > >>> Cc: "For users of the development version of PETSc" <petsc-dev at mcs.anl.gov>
> > > > >>> Sent: Tuesday, August 20, 2013 7:10:40 PM
> > > > >>> Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat
> > > > >>>
> > > > >>>
> > > > >>> How general do you plan for this "magma" matrix class to be? If it all it is for is to do LU/Cholesky factorizations then you do NOT need to introduce src/mat/impls/dense/seq/magma at all. Simply use the MatGetFactor() mechanism to "How will the user be able to access this function?". This is definitely the easy way to go.
> > > > >>>
> > > > >>> If you want to use the "magma" matrix class for all kinds of non-factorization matrix operations then you need to write a full magma matrix class. I don't recommend this.
> > > > >>>
> > > > >>> Barry
> > > > >>>
> > > > >>> On Aug 20, 2013, at 1:35 PM, Harshad Sahasrabudhe <hsahasra at purdue.edu> wrote:
> > > > >>>
> > > > >>>> Hi,
> > > > >>>>
> > > > >>>> I am working on adding the Seq Dense MAGMA matrix type. For this, I have created the following files and directories:
> > > > >>>>
> > > > >>>> src/mat/impls/dense/seq/magma
> > > > >>>> src/mat/impls/dense/seq/magma/magma.c
> > > > >>>> src/mat/impls/dense/seq/magma/makefile
> > > > >>>>
> > > > >>>> Right now I am just trying the make LU factorization work through MAGMA. Barry had suggested looking at dense.c, as the MAGMA function for LU has the same sequence as the LAPACK function getrf. MAGMA does the memory allocation on the GPU inside it's function magma_?getrf. So can I directly use the matrix type declared in dense.c, and just define a function MatLUFactor_SeqDense_MAGMA magma.c which uses the same matrix type?
> > > > >>>>
> > > > >>>> How will the user be able to access this function?
> > > > >>>>
> > > > >>>> Thanks,
> > > > >>>> Harshad
> > > > >>>
> > > > >>
> > > > >
> > > >
> > > >
> > >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20130829/921cdd6d/attachment.html>
More information about the petsc-dev
mailing list