[petsc-dev] Dealing with off-processor aij-matrices not having preallocation information

[Jed Brown]

On Thu, Mar 21, 2013 at 7:50 AM, Karl Rupp <rupp at mcs.anl.gov> wrote:

> Hi Jed,
>
>
> >     As the comment noted, this is quite a hack and actually breaks the
>
>>     current implementation for GPU matrices. I think that the problem
>>     can be completely resolved by not checking for PETSC_DEFAULT and
>>     PETSC_DECIDE in the MPI preallocation above, but just forward d_nz
>>     and o_nz to the sequential preallocation routine. This way, the
>>     logic remains in MatSeqAIJSetPreallocation(), where
>>     MAT_NEW_NONZERO_ALLOCATION_ERR is only set if the user provided an
>>     explicit preallocation scheme.
>>
>>
>> Hmm, the 'nonew' member is accessed in the MATMPIAIJ so we have to
>> either change the code to avoid ever using it from the top level or we
>> need to propagate the setting back to the parent. See also this
>> discussion:
>>
>> http://lists.mcs.anl.gov/**pipermail/petsc-dev/2012-**January/007121.html<http://lists.mcs.anl.gov/pipermail/petsc-dev/2012-January/007121.html>
>>
>> https://bitbucket.org/petsc/**petsc/commits/7827cd58<https://bitbucket.org/petsc/petsc/commits/7827cd58>
>>
>
> Yes, I want to preserve the behavior Barry described in 2), i.e. only
> error if the user specified an allocation pattern explicitly. I'm only
> questioning the way it is currently implemented:
>
> If the number of nonzeros is specified as PETSC_DECIDE to
> MatSeqAIJSetPreallocation_**SeqAIJ(),
> it does not set MAT_NEW_NONZERO_ALLOCATION_**ERR. If the number of
> nonzeros is larger than zero, it does. So far, so good.
>

If nnz is nonnegative, yes.


> MatMPIAIJSetPreallocation_**MPIAIJ(), however, first checks for the
> number of nonzeros. If it is PETSC_DECIDE, it sets defaults and then calls
> MatSeqAIJSetPreallocation_**SeqAIJ() with the defaults. Because
> MatSeqAIJSetPreallocation_**SeqAIJ() no longer sees PETSC_DECIDE, it sets
> MAT_NEW_NONZERO_ALLOCATION_**ERR, which is then reset right after in
> MatMPIAIJSetPreallocation_**MPIAIJ(). The CUSP-implementation does not
> reset the flag and thus leads to errors, even though the implementation of
> the CUSP-Preallocation routine looks perfectly fine. I rather want to fix
> the hack in MatMPIAIJSetPreallocation_**MPIAIJ() than to propagate it to
> the GPU implementations as well.
>

I thought perhaps you were going to leave it to the _SeqAIJ parts to store
the nonew state. If the _MPIAIJ part resolves PETSC_DECIDE independently
from the _SeqAIJ part, then they have to *always* coincide exactly,
otherwise we'll have very confusing bugs. You can likely forward
PETSC_DECIDE down to the _SeqAIJ parts, then query a->nonew and b->nonew to
set the mpiaij->nonew flag. (I'm not sure it's consistent now, but we want
it to be consistent.)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20130321/41533fc7/attachment.html>