[petsc-users] Problem with MPI, MatAXPY and SAME_NONZERO_PATTERN
Matthew Knepley
knepley at gmail.com
Mon Mar 30 06:43:30 CDT 2015
On Mon, Mar 30, 2015 at 5:18 AM, Klaus Kaiser <kaiser at igpm.rwth-aachen.de>
wrote:
> Hallo Metthew,
>
> at the weekend I updated petsc to the newest release version 3.5.2 and I
> still got this strange behavior.
>
Okay, I will check it out. The first thing I am going to do is try with AIJ
matrices instead of BAIJ, which
you can probably do before me. If you get the right result, then its a bug
in the BAIJ implementation,
which I am confident has not been tested as thoroughly.
Thanks,
Matt
> Best
>
> Klaus
>
>
> On 03/27/2015 05:07 PM, Klaus Kaiser wrote:
>
> Hallo Matthew,
>
> here is a short example I used for 8 cores:
>
> int d_nnz[3];
> int o_nnz[3];
> d_nnz[0] = 3; d_nnz[1] = 3; d_nnz[2] = 3;
> o_nnz[0] = 6; o_nnz[1] = 6; o_nnz[2] = 6;
> ierr = MatCreateBAIJ(PETSC_COMM_WORLD, 3, 9, 9, 72, 72, 0, d_nnz, 0,
> o_nnz,&A);
> ierr = MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
> ierr = MatSetOption(A,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
> ierr = MatSetOption(A,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);
> ierr = MatCreateBAIJ(PETSC_COMM_WORLD, 3, 9, 9, 72, 72, 0, d_nnz, 0,
> o_nnz,&Ah);
> ierr = MatCreateBAIJ(PETSC_COMM_WORLD, 3, 9, 9, 72, 72, 0, d_nnz, 0,
> o_nnz,&At);
>
> std::vector<double> insert(3*3*3*3, 1.0);
> for(int i=0;i<8;++i)
> {
> int rows[3] = {i,i+1,i+3};
> int cols[3] = {i,i+1,i+3};
>
> MatSetValuesBlocked(A, 3, rows, 3, cols, &insert[0], ADD_VALUES);
> }
>
> MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);
> MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);
> MatDuplicate(A,MAT_COPY_VALUES,&Ah);
> MatDuplicate(A,MAT_COPY_VALUES,&At);
> MatAssemblyBegin(Ah,MAT_FINAL_ASSEMBLY);
> MatAssemblyEnd(Ah,MAT_FINAL_ASSEMBLY);
> MatAssemblyBegin(At,MAT_FINAL_ASSEMBLY);
> MatAssemblyEnd(At,MAT_FINAL_ASSEMBLY);
>
> MatAXPY(Ah,1.,At,SAME_NONZERO_PATTERN);
> MatAXPY(A,1.,Ah,SAME_NONZERO_PATTERN);
>
> MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);
> MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);
>
> The result is, that only some values are computed correctly.
>
> Best and Thank you a lot
>
> Klaus
>
>
>
> On 03/27/2015 04:26 PM, Matthew Knepley wrote:
>
> On Fri, Mar 27, 2015 at 10:16 AM, Klaus Kaiser <
> kaiser at igpm.rwth-aachen.de> wrote:
>
>> Hallo Matthew,
>>
>> thanks for your fast response. With "no beside the nonzero structure" I
>> meant, that I do not create a different non-zero structure while I'm adding
>> values to my matrices.
>>
>> I also tried MAT_COPY_VALUES and the result was the same.
>>
>
> Please send the small example and we will figure out what is going wrong.
>
> Thanks,
>
> Matt
>
>
>> I do not get a error message but the resulting matrix is wrong. Here I
>> have a short example. The first 9 rows and first 3 colomn of the matrices
>> Ah, At and Ah+At on the first processor (of 8):
>>
>> Ah:
>> 0: 60.3553 -0.249975
>> 2.77556e-17
>> 1: 0
>> 60.3553 0
>> 2: 0 0
>> 60.3553
>> 3: 17.6777 0.374962
>> 0.124987
>> 4: 0
>> 17.6777 0
>> 5: 0 0
>> 17.6777
>> 6: -7.32233 -0.124987
>> 0.374962
>> 7: 0
>> -7.32233 0
>> 8: 0 0
>> -7.32233
>>
>>
>> At:
>> 0: 0
>> 0 0
>> 1: 2500
>> 0 0
>> 2: -4.54747e-13
>> 0 0
>> 3: 0
>> 0 0
>> 4: 1250
>> 0 0
>> 5: 1250
>> 0 0
>> 6: 0
>> 0 0
>> 7: 1250
>> 0 0
>> 8: 3750
>> 0 0
>>
>> Ah+At
>> 0: 60.3553 -0.249975
>> 2.77556e-17
>> 1: 2500
>> 60.3553 0
>> 2: -4.54747e-13 0
>> 60.3553
>> 3: 17.6777 0.374962
>> 0.124987
>> 4: 0
>> 17.6777 0
>> 5: 0 0
>> 17.6777
>> 6: -7.32233 -0.124987
>> 0.374962
>> 7: 0
>> -7.32233 0
>> 8: 0 0
>> -7.32233
>>
>> you can see the first 3 rows of the resulting matrix looks exactly like
>> what I would expect, but the last 6 rows only consists of the values of Ah.
>> When you would also look on the matrix A and A+Ah+At you would also see,
>> that values where both matrices have an nonzero entry are not sum correctly.
>>
>> Best
>>
>> Klaus
>>
>> On 03/27/2015 03:59 PM, Matthew Knepley wrote:
>>
>> On Fri, Mar 27, 2015 at 9:48 AM, Klaus Kaiser <
>> kaiser at igpm.rwth-aachen.de> wrote:
>>
>>> Hallo,
>>>
>>> I have a strange behavior in my code concerning the function MatAXPY. I
>>> create 3 different Matrices
>>>
>>> ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size, local_size,
>>> local_size, system_size, system_size, 0, d_nnz, 0, o_nnz,&A);
>>> ierr = MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
>>> ierr = MatSetOption(A,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
>>> ierr = MatSetOption(A,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);
>>>
>>> ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size, local_size,
>>> local_size, system_size, system_size, 0, d_nnz, 0, o_nnz,&At);
>>> ierr = MatSetOption(At,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
>>> ierr = MatSetOption(At,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
>>> ierr = MatSetOption(At,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);
>>>
>>> ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size, local_size,
>>> local_size, system_size, system_size, 0, d_nnz, 0, o_nnz,&Ah);
>>> ierr = MatSetOption(Ah,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
>>> ierr = MatSetOption(Ah,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
>>> ierr = MatSetOption(Ah,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);
>>>
>>
>> These creations are superfluous since you use MatDuplicate() below.
>>
>>
>>> and want to sum these three matrixes with different factors. First I
>>> fill the Matrix A with some values, and duplicate the structure of A to the
>>> other two matrices:
>>>
>>> MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);
>>> MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);
>>> MatDuplicate(A,MAT_DO_NOT_COPY_VALUES,&Ah);
>>> MatDuplicate(A,MAT_DO_NOT_COPY_VALUES,&At);
>>> MatAssemblyBegin(Ah,MAT_FINAL_ASSEMBLY);
>>> MatAssemblyEnd(Ah,MAT_FINAL_ASSEMBLY);
>>> MatAssemblyBegin(At,MAT_FINAL_ASSEMBLY);
>>> MatAssemblyEnd(At,MAT_FINAL_ASSEMBLY);
>>>
>>
>>
>>> After this I fill the matrices At and Ah with some other values, which
>>> are not beside the non zero structure (I also tried with just copying the
>>> Matrix A). Now after another MatAssembly
>>
>>
>> I do not understand "no beside the nonzero structure". Do you mean that
>> the nonzero structure is the same?
>>
>> Can you first test with MAT_COPY_VALUES?
>>
>>
>>> I want to add these Matrices in the form A+c*(Ah+d*At):
>>
>>
>>> MatAXPY(Ah,c,At,SAME_NONZERO_PATTERN);
>>> MatAXPY(A,d,Ah,SAME_NONZERO_PATTERN);
>>>
>>> When I run the method with mpi and one core everything works fine.
>>> Starting the same method with more cores, the sum of the matrices fails. It
>>> seems like some values are added
>>
>>
>> Please send the full output of a failure when you use MAT_COPY_VALUES
>> on 2 procs.
>>
>> Thanks,
>>
>> Matt
>>
>>
>>> correctly and many values are missed. Using DIFFERENT_NONZERO_STRUCTURE
>>> leads to the right behavior in the multi-core case, but is very slow. I
>>> checked with a viewer if all matrices have the same nonzero structure and
>>> this is the case.
>>>
>>> Does anyone know why this fails, or do I have made any wrong thoughts?
>>>
>>> I'm corrently working with a petsc version (Petsc Release Version 3.3.0,
>>> Patch 5, Sat Dec 1 15:10:41 CST 2012), I looked into the changelogs up to
>>> the current version and did not find any note about MatAXPY or MatAYPX.
>>>
>>>
>>> Best and Thanks a lot for your help
>>>
>>> Klaus
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>
>
--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150330/873f72da/attachment-0001.html>
More information about the petsc-users
mailing list