[petsc-dev] Using PETSc MatIS, how to matmult a global IS matrix and a global vector ?

Wed May 24 04:45:58 CDT 2017

Coming from FEM, I believe the very confusing thing is that the local size of the user problem (math, physics point of view - DDM domain size) is not (can not be ?) the local size expected in MatCreateIS. 

My understanding is that the local size in MatIS is "just" related to backend implementation problems (it's logical that this local size is necessary, but, for another purpose: MPI machinery). Taking a few steps back, I can not see a case (I may be wrong) when a user does know how to compute or set "by hand" the local size that MatIS will expect: my understanding (once again, not sure) is that in most cases, the user will need local size to be PETSC_DECIDE in MatIS (because he doesn't want to "bother" with that or can not guess / compute it => unfortunatelly, as is, this jam the whole thing). 

I guess this kind of signature for MatIS would avoid/limit confusion in most cases and for most users : 
PetscErrorCode MatCreateIS(MPI_Comm comm,PetscInt bs,PetscInt M,PetscInt N,ISLocalToGlobalMapping rmap,ISLocalToGlobalMapping cmap,Mat *A,PetscInt m = PETSC_DECIDE ,PetscInt n = PETSC_DECIDE ) 
Or even 
PetscErrorCode MatCreateIS(MPI_Comm comm,PetscInt bs,PetscInt M,PetscInt N,ISLocalToGlobalMapping rmap,ISLocalToGlobalMapping cmap,Mat *A ) // Always use PETSC_DECIDE backstage ? 

Franck 

----- Mail original -----

> De: "Matthew Knepley" <knepley at gmail.com>
> À: "Franck Houssen" <franck.houssen at inria.fr>
> Cc: "Stefano Zampini" <stefano.zampini at gmail.com>, "PETSc"
> <petsc-users at mcs.anl.gov>, "PETSc" <petsc-dev at mcs.anl.gov>
> Envoyé: Mardi 23 Mai 2017 19:02:28
> Objet: Re: [petsc-dev] Using PETSc MatIS, how to matmult a global IS matrix
> and a global vector ?

> On Tue, May 23, 2017 at 11:51 AM, Franck Houssen < franck.houssen at inria.fr >
> wrote:

> > Not sure to know what question you're talking about ?!...
> 
> > I use MatIS to test some kind of domain decomposition methods. I define my
> > own preconditioner for that: in the apply callback, I need to matmult my
> > (matIS) matrix with the incoming vector.
> 

> Okay. I will create an example using your suggestion.

> Thanks,

> Matt

> > Franck
> 

> > > De: "Matthew Knepley" < knepley at gmail.com >
> > 
> 
> > > À: "Franck Houssen" < franck.houssen at inria.fr >
> > 
> 
> > > Cc: "Stefano Zampini" < stefano.zampini at gmail.com >, "PETSc" <
> > > petsc-users at mcs.anl.gov >, "PETSc" < petsc-dev at mcs.anl.gov >
> > 
> 
> > > Envoyé: Mardi 23 Mai 2017 18:46:34
> > 
> 
> > > Objet: Re: [petsc-dev] Using PETSc MatIS, how to matmult a global IS
> > > matrix
> > > and a global vector ?
> > 
> 

> > > On Tue, May 23, 2017 at 11:28 AM, Franck Houssen <
> > > franck.houssen at inria.fr
> > > >
> > > wrote:
> > 
> 

> > > > OK, thanks. This is helpfull... But I really think the doc should be
> > > > more
> > > > verbose about that: this is really confusing and I didn't find any
> > > > simple
> > > > example to begin with which make all this even more confusing (personal
> > > > opinion).
> > > 
> > 
> 

> > > Did you respond to my other question (how are you using them)? That would
> > > help me understand how to phrase it.
> > 
> 

> > > Thanks,
> > 
> 

> > > Matt
> > 
> 

> > > > Franck
> > > 
> > 
> 

> > > > > De: "Matthew Knepley" < knepley at gmail.com >
> > > > 
> > > 
> > 
> 
> > > > > À: "Franck Houssen" < franck.houssen at inria.fr >
> > > > 
> > > 
> > 
> 
> > > > > Cc: "Stefano Zampini" < stefano.zampini at gmail.com >, "PETSc" <
> > > > > petsc-users at mcs.anl.gov >, "PETSc" < petsc-dev at mcs.anl.gov >
> > > > 
> > > 
> > 
> 
> > > > > Envoyé: Mardi 23 Mai 2017 13:21:21
> > > > 
> > > 
> > 
> 
> > > > > Objet: Re: [petsc-dev] Using PETSc MatIS, how to matmult a global IS
> > > > > matrix
> > > > > and a global vector ?
> > > > 
> > > 
> > 
> 

> > > > > On Tue, May 23, 2017 at 4:53 AM, Franck Houssen <
> > > > > franck.houssen at inria.fr
> > > > > >
> > > > > wrote:
> > > > 
> > > 
> > 
> 

> > > > > > The first thing I did was to put 3, not 4 : I got an error thrown
> > > > > > in
> > > > > > MatCreateIS (see the git diff + stack below). As the error said I
> > > > > > used
> > > > > > globalSize = numberOfMPIProcessus * localSize : my understanding is
> > > > > > that,
> > > > > > when using MatIS, the global size needs to be the sum of all local
> > > > > > sizes.
> > > > > > Correct ?
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > No. MatIS means that the matrix is not assembled. The easiest way
> > > > > (for
> > > > > me)
> > > > > to
> > > > > think of this is that processes do not have
> > > > 
> > > 
> > 
> 
> > > > > to hold full rows. One process can hold part of row i, and another
> > > > > processes
> > > > > can hold another part. However, there are still
> > > > 
> > > 
> > 
> 
> > > > > the same number of global rows.
> > > > 
> > > 
> > 
> 

> > > > > > I have a 3x3 global matrix made of two overlapping 2x2 local matrix
> > > > > > (=
> > > > > > diagonal with 1.). Each local matrix correspond to one domain (each
> > > > > > domain
> > > > > > is delegated to one MPI proc, so, I have 2 MPI procs because I have
> > > > > > 2
> > > > > > domains).
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > So the global size is 3. The local size here is not the size of the
> > > > > local
> > > > > IS
> > > > > block, since that is a property only of MatIS. It is the
> > > > 
> > > 
> > 
> 
> > > > > size of the local piece of the vector you multiply. This allows PETSc
> > > > > to
> > > > > understand the parallel layout of the Vec, and how it
> > > > 
> > > 
> > 
> 
> > > > > matched the Mat.
> > > > 
> > > 
> > 
> 

> > > > > This is somewhat confusing because FEM people mean something
> > > > > different
> > > > > by
> > > > > "local" than we do here, and in fact we use this
> > > > 
> > > 
> > 
> 
> > > > > other definition of local when assembling operators.
> > > > 
> > > 
> > 
> 

> > > > > Matt
> > > > 
> > > 
> > 
> 

> > > > > > This is the simplest possible example: I have two 2x2 (local) diag
> > > > > > matrix
> > > > > > that overlap so that the global matrix built from them is 1, 2, 1
> > > > > > on
> > > > > > the
> > > > > > diagonal (local contributions add up in the middle).
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > I need to MatMult this global matrix with a global vector filled
> > > > > > with
> > > > > > 1.
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > Franck
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > Git diff :
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > --- a/matISLocalMat.cpp
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > +++ b/matISLocalMat.cpp
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > @@ -16,7 +16,7 @@ int main(int argc,char **argv) {
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > int size = 0; MPI_Comm_size(MPI_COMM_WORLD, &size); if (size != 2)
> > > > > > return
> > > > > > 1;
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > int rank = 0; MPI_Comm_rank(MPI_COMM_WORLD, &rank);
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > - PetscInt localSize = 2, globalSize = localSize*2 /*2 MPI*/;
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > + PetscInt localSize = 2, globalSize = 3;
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > PetscInt localIdx[2] = {0, 0};
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > if (rank == 0) {localIdx[0] = 0; localIdx[1] = 1;}
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > else {localIdx[0] = 1; localIdx[1] = 2;}
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > Stack error:
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > [0]PETSC ERROR: Nonconforming object sizes
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > [0]PETSC ERROR: Sum of local lengths 4 does not equal global length
> > > > > > 3,
> > > > > > my
> > > > > > local length 2
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > [0]PETSC ERROR: [0] ISG2LMapApply line 17
> > > > > > /home/fghoussen/Documents/INRIA/petsc-3.7.6/src/vec/is/utils/isltog.c
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > [0]PETSC ERROR: [0] MatSetValues_IS line 692
> > > > > > /home/fghoussen/Documents/INRIA/petsc-3.7.6/src/mat/impls/is/matis.c
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > [0]PETSC ERROR: [0] MatSetValues line 1157
> > > > > > /home/fghoussen/Documents/INRIA/petsc-3.7.6/src/mat/interface/matrix.c
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > [0]PETSC ERROR: [0] MatISSetPreallocation_IS line 95
> > > > > > /home/fghoussen/Documents/INRIA/petsc-3.7.6/src/mat/impls/is/matis.c
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > [0]PETSC ERROR: [0] MatISSetPreallocation line 80
> > > > > > /home/fghoussen/Documents/INRIA/petsc-3.7.6/src/mat/impls/is/matis.c
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > [0]PETSC ERROR: [0] PetscSplitOwnership line 80
> > > > > > /home/fghoussen/Documents/INRIA/petsc-3.7.6/src/sys/utils/psplit.c
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > [0]PETSC ERROR: [0] PetscLayoutSetUp line 129
> > > > > > /home/fghoussen/Documents/INRIA/petsc-3.7.6/src/vec/is/utils/pmap.c
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > [0]PETSC ERROR: [0] MatSetLocalToGlobalMapping_IS line 628
> > > > > > /home/fghoussen/Documents/INRIA/petsc-3.7.6/src/mat/impls/is/matis.c
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > [0]PETSC ERROR: [0] MatSetLocalToGlobalMapping line 1899
> > > > > > /home/fghoussen/Documents/INRIA/petsc-3.7.6/src/mat/interface/matrix.c
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > [0]PETSC ERROR: [0] MatCreateIS line 986
> > > > > > /home/fghoussen/Documents/INRIA/petsc-3.7.6/src/mat/impls/is/matis.c
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > > De: "Stefano Zampini" < stefano.zampini at gmail.com >
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > À: "Matthew Knepley" < knepley at gmail.com >
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > Cc: "Franck Houssen" < franck.houssen at inria.fr >, "PETSc" <
> > > > > > > petsc-users at mcs.anl.gov >, "PETSc" < petsc-dev at mcs.anl.gov >
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > Envoyé: Dimanche 21 Mai 2017 23:02:37
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > Objet: Re: [petsc-dev] Using PETSc MatIS, how to matmult a global
> > > > > > > IS
> > > > > > > matrix
> > > > > > > and a global vector ?
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > > Franck,
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > > PETSc takes care of doing the matrix-vector multiplication
> > > > > > > properly
> > > > > > > using
> > > > > > > MatIS. As Matt said, the layout of the vectors is the usual
> > > > > > > parallel
> > > > > > > layout.
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > The local sizes of the MatIS matrix (i.e. the local size of the
> > > > > > > left
> > > > > > > and
> > > > > > > right vectors used in MatMult) are not the sizes of the local
> > > > > > > subdomain
> > > > > > > matrices in MatIS.
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > > > On May 21, 2017, at 6:47 PM, Matthew Knepley <
> > > > > > > > knepley at gmail.com
> > > > > > > > >
> > > > > > > > wrote:
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > > > On Sun, May 21, 2017 at 11:26 AM, Franck Houssen <
> > > > > > > > franck.houssen at inria.fr
> > > > > > > > >
> > > > > > > > wrote:
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > > > > Using PETSc MatIS, how to matmult a global IS matrix and a
> > > > > > > > > global
> > > > > > > > > vector
> > > > > > > > > ?
> > > > > > > > > Example is attached : I don't get what I expect that is a
> > > > > > > > > vector
> > > > > > > > > such
> > > > > > > > > that
> > > > > > > > > proc0 = [1, 2] and proc1 = [2, 1]
> > > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > > > 1) I think the global size of your matrix is wrong. You seem to
> > > > > > > > want
> > > > > > > > 3,
> > > > > > > > not
> > > > > > > > 4
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > > > 2) Global vectors have a non-overlapping row partition. You
> > > > > > > > might
> > > > > > > > be
> > > > > > > > thinking
> > > > > > > > of local vectors
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > > > Thanks,
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > > > Matt
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > > > --
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > What most experimenters take for granted before they begin
> > > > > > > > their
> > > > > > > > experiments
> > > > > > > > is infinitely more interesting than any results to which their
> > > > > > > > experiments
> > > > > > > > lead.
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > -- Norbert Wiener
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > > > http://www.caam.rice.edu/~mk51/
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > > De: "Stefano Zampini" < stefano.zampini at gmail.com >
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > À: "Matthew Knepley" < knepley at gmail.com >
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > Cc: "Franck Houssen" < franck.houssen at inria.fr >, "PETSc" <
> > > > > > > petsc-users at mcs.anl.gov >, "PETSc" < petsc-dev at mcs.anl.gov >
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > Envoyé: Dimanche 21 Mai 2017 23:02:37
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > Objet: Re: [petsc-dev] Using PETSc MatIS, how to matmult a global
> > > > > > > IS
> > > > > > > matrix
> > > > > > > and a global vector ?
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > > Franck,
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > > PETSc takes care of doing the matrix-vector multiplication
> > > > > > > properly
> > > > > > > using
> > > > > > > MatIS. As Matt said, the layout of the vectors is the usual
> > > > > > > parallel
> > > > > > > layout.
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > The local sizes of the MatIS matrix (i.e. the local size of the
> > > > > > > left
> > > > > > > and
> > > > > > > right vectors used in MatMult) are not the sizes of the local
> > > > > > > subdomain
> > > > > > > matrices in MatIS.
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > > > On May 21, 2017, at 6:47 PM, Matthew Knepley <
> > > > > > > > knepley at gmail.com
> > > > > > > > >
> > > > > > > > wrote:
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > > > On Sun, May 21, 2017 at 11:26 AM, Franck Houssen <
> > > > > > > > franck.houssen at inria.fr
> > > > > > > > >
> > > > > > > > wrote:
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > > > > Using PETSc MatIS, how to matmult a global IS matrix and a
> > > > > > > > > global
> > > > > > > > > vector
> > > > > > > > > ?
> > > > > > > > > Example is attached : I don't get what I expect that is a
> > > > > > > > > vector
> > > > > > > > > such
> > > > > > > > > that
> > > > > > > > > proc0 = [1, 2] and proc1 = [2, 1]
> > > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > > > 1) I think the global size of your matrix is wrong. You seem to
> > > > > > > > want
> > > > > > > > 3,
> > > > > > > > not
> > > > > > > > 4
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > > > 2) Global vectors have a non-overlapping row partition. You
> > > > > > > > might
> > > > > > > > be
> > > > > > > > thinking
> > > > > > > > of local vectors
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > > > Thanks,
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > > > Matt
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > > > > Franck
> > > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > > > --
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > What most experimenters take for granted before they begin
> > > > > > > > their
> > > > > > > > experiments
> > > > > > > > is infinitely more interesting than any results to which their
> > > > > > > > experiments
> > > > > > > > lead.
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > -- Norbert Wiener
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > > > http://www.caam.rice.edu/~mk51/
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > --
> > > > 
> > > 
> > 
> 
> > > > > What most experimenters take for granted before they begin their
> > > > > experiments
> > > > > is infinitely more interesting than any results to which their
> > > > > experiments
> > > > > lead.
> > > > 
> > > 
> > 
> 
> > > > > -- Norbert Wiener
> > > > 
> > > 
> > 
> 

> > > > > http://www.caam.rice.edu/~mk51/
> > > > 
> > > 
> > 
> 

> > > --
> > 
> 
> > > What most experimenters take for granted before they begin their
> > > experiments
> > > is infinitely more interesting than any results to which their
> > > experiments
> > > lead.
> > 
> 
> > > -- Norbert Wiener
> > 
> 

> > > http://www.caam.rice.edu/~mk51/
> > 
> 

> --
> What most experimenters take for granted before they begin their experiments
> is infinitely more interesting than any results to which their experiments
> lead.
> -- Norbert Wiener

> http://www.caam.rice.edu/~mk51/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20170524/7682d04b/attachment.html>