[petsc-dev] Bug introduced in MatGetSubmatrices()

[Barry Smith]

> On Jan 20, 2017, at 3:01 PM, Zhang, Hong <hzhang at mcs.anl.gov> wrote:
> 
> We set C->submat_singleis = false as default. 
> User turns it on when he knows a single IS will be used for MatGetSubMatrices(), e.g., when asm is used.
> User should turn the flag off after it is being used.

   Right, this is why I am confused why this is messing up Mat since presumably he is not setting it?

> 
> This flag is for optimization -- avoid MPI_Allreduce, which is expensive for np>10k.
> MatGetSubMatrices() works without this flag turns on. 
> I just merged the improved MatGetSubMatrices_MPIAIJ() which only calls MPI_Allreduce once
> for repeated use. Previously, it calls MPI_Allreduce twice for each call of MatGetSubMatrices_MPIAIJ().
> 
> Hong
> 
> ________________________________________
> From: Barry Smith [bsmith at mcs.anl.gov]
> Sent: Friday, January 20, 2017 1:11 PM
> To: Matthew Knepley
> Cc: Zhang, Hong; PETSc
> Subject: Re: [petsc-dev] Bug introduced in MatGetSubmatrices()
> 
>> On Jan 20, 2017, at 12:45 PM, Matthew Knepley <knepley at gmail.com> wrote:
>> 
>> On Fri, Jan 20, 2017 at 11:55 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>> 
>>> On Jan 20, 2017, at 11:49 AM, Matthew Knepley <knepley at gmail.com> wrote:
>>> 
>>> On Fri, Jan 20, 2017 at 10:49 AM, Hong <hzhang at mcs.anl.gov> wrote:
>>> Matt,
>>> By default, the flag C->submat_singleis = false.
>>> In PCSetUp_ASM(), we set it as 'true' to use MatGetSubMatrices_MPIAIJ_SingleIS().
>>> 
>>> Can you check the value of this flag in your case?
>>> 
>>> The problem is the following:
>>> 
>>>  1) We use MatGetSubMatrices() to extract small matrices in order to form a preconditioner
>>> 
>>>  2) We do this at each Newton iteration
>>> 
>>>  3) We use ASM as a preconditioner for the eventual Newton solve
>>> 
>>>  4) The second time we call MatGetSubMatrices(), it has this flag set, even though we are using multiple ISes
>>> 
>>> Solution: ALSO check that the user is in fact passing a single IS.
>>> 
>>    This requires a communication. I am confused, is the number of IS changing each time? If not why is the flag set?
>> 
>> ASM sets this flag because it knows that IT is going to call MatGetSubMatrices() later, but it unsafe if any user calls
>> MatGetSubMatrices() as well. I think overall its a fragile design and should be scrapped.
> 
>   You mean the user calls the SAME MatGetSubMatrices with the same matrices, right? Not a completely different unrelated MatGetSubMatrices() which should not be affect by the previous unrelated call.
> 
> 
>   Barry
> 
>> 
>>  Matt
>> 
>> 
>>   Barry
>> 
>>>  Matt
>>> 
>>> Hong
>>> 
>>> It comes from here:
>>> 
>>>  https://bitbucket.org/petsc/petsc/commits/c10200c1442b553b7ad65c70101560db4fa22e78
>>> 
>>> If we ask for more than 1 matrix, it dispatches to
>>> 
>>>  MatGetSubMatrices_MPIAIJ_SingleIS()
>>> 
>>> but then fails here
>>> 
>>>  https://bitbucket.org/petsc/petsc/annotate/2e559809f9aee9c95ee79eb0939630cfe5502c8d/src/mat/impls/aij/mpi/mpiov.c?at=master&fileviewer=file-view-default#mpiov.c-1306
>>> 
>>> because ismax > 1. I think the ismax check needs to move up to here
>>> 
>>>  https://bitbucket.org/petsc/petsc/annotate/2e559809f9aee9c95ee79eb0939630cfe5502c8d/src/mat/impls/aij/mpi/mpiov.c?at=master&fileviewer=file-view-default#mpiov.c-2012
>>> 
>>> but I don't know for sure. Please fix this since it is breaking PyLith.
>>> 
>>>  Matt
>>> 
>>> --
>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>> -- Norbert Wiener
>>> 
>>> 
>>> 
>>> 
>>> 
>>> --
>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>> -- Norbert Wiener
>> 
>> 
>> 
>> 
>> --
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> -- Norbert Wiener
> 
>