[petsc-users] Problem with MPI, MatAXPY and SAME_NONZERO_PATTERN

[Klaus Kaiser]

Hallo,

I have a strange behavior in my code concerning the function MatAXPY. I 
create 3 different Matrices

     ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size, local_size, 
local_size, system_size, system_size, 0, d_nnz, 0, o_nnz,&A);
     ierr = MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
     ierr = MatSetOption(A,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
     ierr = MatSetOption(A,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);

     ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size, local_size, 
local_size, system_size, system_size, 0, d_nnz, 0, o_nnz,&At);
     ierr = MatSetOption(At,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
     ierr = MatSetOption(At,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
     ierr = MatSetOption(At,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);

     ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size, local_size, 
local_size, system_size, system_size, 0, d_nnz, 0, o_nnz,&Ah);
     ierr = MatSetOption(Ah,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
     ierr = MatSetOption(Ah,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
     ierr = MatSetOption(Ah,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);

and want to sum these three matrixes with different factors. First I 
fill the Matrix A with some values, and duplicate the structure of A to 
the other two matrices:

     MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);
     MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);
     MatDuplicate(A,MAT_DO_NOT_COPY_VALUES,&Ah);
     MatDuplicate(A,MAT_DO_NOT_COPY_VALUES,&At);
     MatAssemblyBegin(Ah,MAT_FINAL_ASSEMBLY);
     MatAssemblyEnd(Ah,MAT_FINAL_ASSEMBLY);
     MatAssemblyBegin(At,MAT_FINAL_ASSEMBLY);
     MatAssemblyEnd(At,MAT_FINAL_ASSEMBLY);

After this I fill the matrices At and Ah with some other values, which 
are not beside the non zero structure (I also tried with just copying 
the Matrix A). Now after another MatAssembly I want to add these 
Matrices in the form A+c*(Ah+d*At):

     MatAXPY(Ah,c,At,SAME_NONZERO_PATTERN);
     MatAXPY(A,d,Ah,SAME_NONZERO_PATTERN);

When I run the method with mpi and one core everything works fine. 
Starting the same method with more cores, the sum of the matrices fails. 
It seems like some values are added correctly and many values are 
missed. Using DIFFERENT_NONZERO_STRUCTURE leads to the right behavior in 
the multi-core case, but is very slow. I checked with a viewer if all 
matrices have the same nonzero structure and this is the case.

Does anyone know why this fails, or do I have made any wrong thoughts?

I'm corrently working with a petsc version (Petsc Release Version 3.3.0, 
Patch 5, Sat Dec  1 15:10:41 CST 2012), I looked into the changelogs up 
to the current version and did not find any note about MatAXPY or MatAYPX.


Best and Thanks a lot for your help

Klaus