[petsc-dev] Fwd: [petsc-users] MatAssemblyEnd hangs during a parallel calculation with PETSc>3.3

[Barry Smith]

  Jed,

    I think it is better to make this safe. If user calls MatXXXSetPreallocation() then set nonew flag, if they do not call it then do not set nonew flag. Having it depend on the exact arguments passed in is just too dangerous.

    See my other mail to petsc-dev

when MatMPIXXXXSetPreallocation_MPIXXX() is called, ALWAYS set the nonew flag, but when it is called because the user never called it like in

PetscErrorCode MatSetUp_MPIAIJ(Mat A)
{
 PetscErrorCode ierr;

 PetscFunctionBegin;
 ierr =  MatMPIAIJSetPreallocation(A,PETSC_DEFAULT,0,PETSC_DEFAULT,0);CHKERRQ(ierr);
 PetscFunctionReturn(0);
}

we immediately change the flag after the call by added a MatSetOption() after the call to MatMPIXXXSetPreallocation()

Request-assigned:  Matt volunteered to fix. MatAssemblyEnd hangs during a parallel calculation do to preallocation specifics

Begin forwarded message:

> From: Jed Brown <jed at jedbrown.org>
> Subject: Re: [petsc-users] MatAssemblyEnd hangs during a parallel calculation with PETSc>3.3
> Date: January 16, 2014 at 8:12:59 AM CST
> To: Projet_TRIOU <triou at cea.fr>, <petsc-users at mcs.anl.gov>
> 
> Projet_TRIOU <triou at cea.fr> writes:
> 
>> Sorry a typo, it is really MatMPISBAIJSetPreallocation()
>> 
>> Yes, I call MatMPISBAIJSetPreallocation() on ALL processes and
>> sometimes a local part of the matrix has zero rows. It worked well
>> with Petsc 3.3 and before in this particular case.
> 
> I believe this is the relevant conditional:
> 
>  if (nz >= 0 || nnz) realalloc = PETSC_TRUE;
> 
> Instead of this:
> 
>       if (nb_rows==0) // Local matrix is empty
>          ierr = MatMPISBAIJSetPreallocation(MatricePetsc, block_size_, 0, NULL, 0, NULL);
>       else
>          ierr = MatMPISBAIJSetPreallocation(MatricePetsc, block_size_, PETSC_DEFAULT, d_nnz.addr(), PETSC_DEFAULT, o_nnz.addr());
> 
> can you just call the following?
> 
>          ierr = MatMPISBAIJSetPreallocation(MatricePetsc, block_size_, 0, d_nnz.addr(), 0, o_nnz.addr());
> 
> If you are passing an nnz array (can be empty if there are zero rows),
> we will trust that.  PETSC_DEFAULT is only intended to be used if you
> don't pass an array, though we're currently being tolerant by ignoring
> it if the array is non-NULL.  The problem arises because we can't tell
> the difference between a zero-length semantically-meaningful array and
> an "I'm not passing this argument" NULL.
> 
> Is this something we can fix in the documentation or should we guard in
> the code?