[petsc-dev] Bug introduced in MatGetSubmatrices()

Fri Jan 20 20:06:12 CST 2017

On Fri, Jan 20, 2017 at 7:39 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> > On Jan 20, 2017, at 7:01 PM, Matthew Knepley <knepley at gmail.com> wrote:
> >
> > On Fri, Jan 20, 2017 at 3:08 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> > > On Jan 20, 2017, at 3:01 PM, Zhang, Hong <hzhang at mcs.anl.gov> wrote:
> > >
> > > We set C->submat_singleis = false as default.
> > > User turns it on when he knows a single IS will be used for
> MatGetSubMatrices(), e.g., when asm is used.
> > > User should turn the flag off after it is being used.
> >
> >    Right, this is why I am confused why this is messing up Mat since
> presumably he is not setting it?
> >
> > ASM is turning this flag on, and it is corrupting my UNRELATED call to
> MatGetSubMatrices().
>
>   Yes, but you need to tell us exactly how. The flag is set into
> C->submat_singleis  which is associated with the C from ASM, so why is it
> affecting your UNRELATED call to MatGetSubMatrices(). It is not a global
> variable so shouldn't.
>

It sets the flag on the primary matrix. I will go step by step

  1) I have a linear system with matrix A

  2) To form a preconditioner for FieldSplit, I extract some diagonal
blocks from A using MatGetSubMatrices()

  3) I also use ASM on the another block of FS

  4) The second Newton iterate, I try to call MatGetSubMatrices(), but ASM
has set a flag on the matrix that causes this to fail.

>    Explain or send a code that reproduces the problem.
>

You can easily reproduce this by installing PyLith (a widely used package)
and running an automatic test. However, since this
is so simple, I am not sure this is necessary.

  Matt

>    Barry
>
> >
> >   Matt
> >
> > >
> > > This flag is for optimization -- avoid MPI_Allreduce, which is
> expensive for np>10k.
> > > MatGetSubMatrices() works without this flag turns on.
> > > I just merged the improved MatGetSubMatrices_MPIAIJ() which only calls
> MPI_Allreduce once
> > > for repeated use. Previously, it calls MPI_Allreduce twice for each
> call of MatGetSubMatrices_MPIAIJ().
> > >
> > > Hong
> > >
> > > ________________________________________
> > > From: Barry Smith [bsmith at mcs.anl.gov]
> > > Sent: Friday, January 20, 2017 1:11 PM
> > > To: Matthew Knepley
> > > Cc: Zhang, Hong; PETSc
> > > Subject: Re: [petsc-dev] Bug introduced in MatGetSubmatrices()
> > >
> > >> On Jan 20, 2017, at 12:45 PM, Matthew Knepley <knepley at gmail.com>
> wrote:
> > >>
> > >> On Fri, Jan 20, 2017 at 11:55 AM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> > >>
> > >>> On Jan 20, 2017, at 11:49 AM, Matthew Knepley <knepley at gmail.com>
> wrote:
> > >>>
> > >>> On Fri, Jan 20, 2017 at 10:49 AM, Hong <hzhang at mcs.anl.gov> wrote:
> > >>> Matt,
> > >>> By default, the flag C->submat_singleis = false.
> > >>> In PCSetUp_ASM(), we set it as 'true' to use
> MatGetSubMatrices_MPIAIJ_SingleIS().
> > >>>
> > >>> Can you check the value of this flag in your case?
> > >>>
> > >>> The problem is the following:
> > >>>
> > >>>  1) We use MatGetSubMatrices() to extract small matrices in order to
> form a preconditioner
> > >>>
> > >>>  2) We do this at each Newton iteration
> > >>>
> > >>>  3) We use ASM as a preconditioner for the eventual Newton solve
> > >>>
> > >>>  4) The second time we call MatGetSubMatrices(), it has this flag
> set, even though we are using multiple ISes
> > >>>
> > >>> Solution: ALSO check that the user is in fact passing a single IS.
> > >>>
> > >>    This requires a communication. I am confused, is the number of IS
> changing each time? If not why is the flag set?
> > >>
> > >> ASM sets this flag because it knows that IT is going to call
> MatGetSubMatrices() later, but it unsafe if any user calls
> > >> MatGetSubMatrices() as well. I think overall its a fragile design and
> should be scrapped.
> > >
> > >   You mean the user calls the SAME MatGetSubMatrices with the same
> matrices, right? Not a completely different unrelated MatGetSubMatrices()
> which should not be affect by the previous unrelated call.
> > >
> > >
> > >   Barry
> > >
> > >>
> > >>  Matt
> > >>
> > >>
> > >>   Barry
> > >>
> > >>>  Matt
> > >>>
> > >>> Hong
> > >>>
> > >>> It comes from here:
> > >>>
> > >>>  https://bitbucket.org/petsc/petsc/commits/
> c10200c1442b553b7ad65c70101560db4fa22e78
> > >>>
> > >>> If we ask for more than 1 matrix, it dispatches to
> > >>>
> > >>>  MatGetSubMatrices_MPIAIJ_SingleIS()
> > >>>
> > >>> but then fails here
> > >>>
> > >>>  https://bitbucket.org/petsc/petsc/annotate/
> 2e559809f9aee9c95ee79eb0939630cfe5502c8d/src/mat/impls/aij/
> mpi/mpiov.c?at=master&fileviewer=file-view-default#mpiov.c-1306
> > >>>
> > >>> because ismax > 1. I think the ismax check needs to move up to here
> > >>>
> > >>>  https://bitbucket.org/petsc/petsc/annotate/
> 2e559809f9aee9c95ee79eb0939630cfe5502c8d/src/mat/impls/aij/
> mpi/mpiov.c?at=master&fileviewer=file-view-default#mpiov.c-2012
> > >>>
> > >>> but I don't know for sure. Please fix this since it is breaking
> PyLith.
> > >>>
> > >>>  Matt
> > >>>
> > >>> --
> > >>> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > >>> -- Norbert Wiener
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > >>> -- Norbert Wiener
> > >>
> > >>
> > >>
> > >>
> > >> --
> > >> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > >> -- Norbert Wiener
> > >
> > >
> >
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20170120/adb4b3e4/attachment.html>