[petsc-dev] Fwd: [petsc-users] MatAssemblyEnd hangs during a parallel calculation with PETSc>3.3

[Barry Smith]

On Jan 15, 2014, at 11:20 AM, Matthew Knepley <knepley at gmail.com> wrote:

> This is a bug in the setup phase for Mat. I did not put in the logic, so I might be missing the point. Here is
> my outline:
> 
>   1) When MatSetPreallocation_*() is called, we set the NEW_NONZERO_ALLOCATION_ERR option, but ONLY
>        if the user passed a nonzero nz or nnz.

   Matt,

   Ahhh. 

   The background is we wanted PETSc to automatically YELL if someone preallocated but did not preallocate enough. 

   But if the user did not preallocate we automatically call MatMPIXXSetPreallocation_MPIXXX() with default values that do not setup the yelling.

   If might be that some process has some rows, but they are all empty so our current model is totally incorrect

1)   We could do a global reduction in MatMPIXXXXSetPreallocation_MPIXXX() and if ANY process allocates space then mark the nonew.  We definitely don’t want to do a global reduction on every mat assembly.   For efficiency we could not automatically set the nonew for optimized builds but that is a dangerous game.

2)  Alternative, when MatMPIXXXXSetPreallocation_MPIXXX() is called, ALWAYS set the nonew flag, but when it is called because the user never called it like in

PetscErrorCode MatSetUp_MPIAIJ(Mat A)
{
  PetscErrorCode ierr;

  PetscFunctionBegin;
  ierr =  MatMPIAIJSetPreallocation(A,PETSC_DEFAULT,0,PETSC_DEFAULT,0);CHKERRQ(ierr);
  PetscFunctionReturn(0);
}

we immediately change the flag after the call by added a MatSetOption() after the call to MatMPIXXXSetPreallocation()

I think 2 is ok and suggest doing that

   Barry


   

> 
>   2) During assembly, we check A->B->nonew, and do a collective call if it is 0, meaning NEW_NONZERO_LOCATIONS
> 
> I understand that we would like to avoid collective calls in MatAssembly(), but to guarantee correctness then we would
> need to make the option collective, which is not true right now. I see
> 
>   Slightly Dangerous: Change SetPreallocation to make the option value collective
> 
>   Slightly Slow: Always check for B disassembly
> 
> What do you think?
> 
>    Matt
> 
> ---------- Forwarded message ----------
> From: Projet_TRIOU <triou at cea.fr>
> Date: Wed, Jan 15, 2014 at 5:57 AM
> Subject: [petsc-users] MatAssemblyEnd hangs during a parallel calculation with PETSc>3.3
> To: petsc-users at mcs.anl.gov
> 
> 
> Hi all,
> 
> I switched from PETSc 3.3 to PETSc 3.4.3 and all was fine except for
> one of my test case on 3 processors where one processor was
> dealing an empty local part of the global matrix.
> 
> My code hangs just during the call at MatAssemblyEnd:
> 
> ierr = MatMPIAIJSetPreallocation(MatricePetsc, PETSC_DEFAULT, d_nnz.addr(), PETSC_DEFAULT, o_nnz.addr());
> ...
> ierr = MatAssemblyEnd(MatricePetsc, MAT_FINAL_ASSEMBLY);
> 
> When I debugged, I notice on the empty processor, that in
> src/mat/impls/aij/mpi/mpiaij.c:
> 
>   if (!((Mat_SeqAIJ*)aij->B->data)->nonew) {
>     ierr = MPI_Allreduce(&mat->was_assembled,&other_disassembled,1,MPIU_BOOL,MPI_PROD,PetscObjectComm((PetscObject)mat));CHKERRQ(ierr);
>     if (mat->was_assembled && !other_disassembled) {
>       ierr = MatDisAssemble_MPIAIJ(mat);CHKERRQ(ierr);
>     }
> 
> ((Mat_SeqAIJ*)aij->B->data)->nonew was 0 on the "empty" processor
> and -2 on the 2 others...
> 
> I bypassed my problem with a different call to MatMPIAIJSetPreallocation():
> 
>       if (nb_rows==0) // Local matrix is empty
>          ierr = MatMPISBAIJSetPreallocation(MatricePetsc, block_size_, 0, NULL, 0, NULL);
>       else
>          ierr = MatMPISBAIJSetPreallocation(MatricePetsc, block_size_, PETSC_DEFAULT, d_nnz.addr(), PETSC_DEFAULT, o_nnz.addr());
> 
> Now, it runs well. So I don't know if it is a PETSc regression or if I was abusively
> calling MatMPISBAIJSetPreallocation with d_nnz/o_nnz empty arrays....
> 
> Thanks,
> 
> Pierre
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener