[petsc-users] Problem with MPI, MatAXPY and SAME_NONZERO_PATTERN
Klaus Kaiser
kaiser at igpm.rwth-aachen.de
Fri Apr 10 02:02:08 CDT 2015
Of course this is not my real world example, and not the best way to
fill a matrix. I just wanted to quickly fill the matrices and give an
example for my problem. Therefore it does not matter how the matrices
look like, the problem is that the resulting sum of these three matrices
is wrong.
(e.g. all matricies have values beside the diagonal block which have the
value a, therefore the resulting sum should have the value 3*a, but in
my computation the result is a. Some values are computed correctly, the
entry (1,1) for example, but most are wrong.)
Overall the interessting part is:
MatAXPY(Ah,1.,At,SAME_NONZERO_PATTERN);
MatAXPY(A,1.,Ah,SAME_NONZERO_PATTERN);
I made an matrix output for the Matrix A before, after this calculation
and at the end of the code. With the result, that the matrices after
computing the sum are wrong.
Best
Klaus
On 04/09/2015 09:11 PM, Matthew Knepley wrote:
> On Fri, Mar 27, 2015 at 11:07 AM, Klaus Kaiser
> <kaiser at igpm.rwth-aachen.de <mailto:kaiser at igpm.rwth-aachen.de>> wrote:
>
> Hallo Matthew,
>
> here is a short example I used for 8 cores:
>
> int d_nnz[3];
> int o_nnz[3];
> d_nnz[0] = 3; d_nnz[1] = 3; d_nnz[2] = 3;
> o_nnz[0] = 6; o_nnz[1] = 6; o_nnz[2] = 6;
> ierr = MatCreateBAIJ(PETSC_COMM_WORLD, 3, 9, 9, 72, 72, 0,
> d_nnz, 0, o_nnz,&A);
> ierr = MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
> ierr = MatSetOption(A,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
> ierr = MatSetOption(A,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);
> ierr = MatCreateBAIJ(PETSC_COMM_WORLD, 3, 9, 9, 72, 72, 0,
> d_nnz, 0, o_nnz,&Ah);
> ierr = MatCreateBAIJ(PETSC_COMM_WORLD, 3, 9, 9, 72, 72, 0,
> d_nnz, 0, o_nnz,&At);
>
> std::vector<double> insert(3*3*3*3, 1.0);
> for(int i=0;i<8;++i)
> {
> int rows[3] = {i,i+1,i+3};
> int cols[3] = {i,i+1,i+3};
>
> MatSetValuesBlocked(A, 3, rows, 3, cols, &insert[0],
> ADD_VALUES);
> }
>
>
> I do not understand this example. All the procs (8) are filling up
> only the first 32 rows. Is that
> what you intended? Can you tell me what output you expect?
>
> Thanks,
>
> Matt
>
> MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);
> MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);
> MatDuplicate(A,MAT_COPY_VALUES,&Ah);
> MatDuplicate(A,MAT_COPY_VALUES,&At);
> MatAssemblyBegin(Ah,MAT_FINAL_ASSEMBLY);
> MatAssemblyEnd(Ah,MAT_FINAL_ASSEMBLY);
> MatAssemblyBegin(At,MAT_FINAL_ASSEMBLY);
> MatAssemblyEnd(At,MAT_FINAL_ASSEMBLY);
>
> MatAXPY(Ah,1.,At,SAME_NONZERO_PATTERN);
> MatAXPY(A,1.,Ah,SAME_NONZERO_PATTERN);
>
> MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);
> MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);
>
> The result is, that only some values are computed correctly.
>
> Best and Thank you a lot
>
> Klaus
>
>
>
>
> On 03/27/2015 04:26 PM, Matthew Knepley wrote:
>> On Fri, Mar 27, 2015 at 10:16 AM, Klaus Kaiser
>> <kaiser at igpm.rwth-aachen.de <mailto:kaiser at igpm.rwth-aachen.de>>
>> wrote:
>>
>> Hallo Matthew,
>>
>> thanks for your fast response. With "no beside the nonzero
>> structure" I meant, that I do not create a different non-zero
>> structure while I'm adding values to my matrices.
>>
>> I also tried MAT_COPY_VALUES and the result was the same.
>>
>>
>> Please send the small example and we will figure out what is
>> going wrong.
>>
>> Thanks,
>>
>> Matt
>>
>> I do not get a error message but the resulting matrix is
>> wrong. Here I have a short example. The first 9 rows and
>> first 3 colomn of the matrices Ah, At and Ah+At on the first
>> processor (of 8):
>>
>> Ah:
>> 0: 60.3553 -0.249975 2.77556e-17
>> 1: 0 60.3553 0
>> 2: 0 0 60.3553
>> 3: 17.6777 0.374962 0.124987
>> 4: 0 17.6777 0
>> 5: 0 0 17.6777
>> 6: -7.32233 -0.124987 0.374962
>> 7: 0 -7.32233 0
>> 8: 0 0 -7.32233
>>
>>
>> At:
>> 0: 0 0 0
>> 1: 2500 0 0
>> 2: -4.54747e-13 0 0
>> 3: 0 0 0
>> 4: 1250 0 0
>> 5: 1250 0 0
>> 6: 0 0 0
>> 7: 1250 0 0
>> 8: 3750 0 0
>>
>> Ah+At
>> 0: 60.3553 -0.249975 2.77556e-17
>> 1: 2500 60.3553 0
>> 2: -4.54747e-13 0 60.3553
>> 3: 17.6777 0.374962 0.124987
>> 4: 0 17.6777 0
>> 5: 0 0 17.6777
>> 6: -7.32233 -0.124987 0.374962
>> 7: 0 -7.32233 0
>> 8: 0 0 -7.32233
>>
>> you can see the first 3 rows of the resulting matrix looks
>> exactly like what I would expect, but the last 6 rows only
>> consists of the values of Ah. When you would also look on the
>> matrix A and A+Ah+At you would also see, that values where
>> both matrices have an nonzero entry are not sum correctly.
>>
>> Best
>>
>> Klaus
>>
>> On 03/27/2015 03:59 PM, Matthew Knepley wrote:
>>> On Fri, Mar 27, 2015 at 9:48 AM, Klaus Kaiser
>>> <kaiser at igpm.rwth-aachen.de
>>> <mailto:kaiser at igpm.rwth-aachen.de>> wrote:
>>>
>>> Hallo,
>>>
>>> I have a strange behavior in my code concerning the
>>> function MatAXPY. I create 3 different Matrices
>>>
>>> ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size,
>>> local_size, local_size, system_size, system_size, 0,
>>> d_nnz, 0, o_nnz,&A);
>>> ierr =
>>> MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
>>> ierr =
>>> MatSetOption(A,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
>>> ierr =
>>> MatSetOption(A,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);
>>>
>>> ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size,
>>> local_size, local_size, system_size, system_size, 0,
>>> d_nnz, 0, o_nnz,&At);
>>> ierr =
>>> MatSetOption(At,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
>>> ierr =
>>> MatSetOption(At,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
>>> ierr =
>>> MatSetOption(At,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);
>>>
>>> ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size,
>>> local_size, local_size, system_size, system_size, 0,
>>> d_nnz, 0, o_nnz,&Ah);
>>> ierr =
>>> MatSetOption(Ah,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
>>> ierr =
>>> MatSetOption(Ah,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
>>> ierr =
>>> MatSetOption(Ah,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);
>>>
>>>
>>> These creations are superfluous since you use MatDuplicate()
>>> below.
>>>
>>> and want to sum these three matrixes with different
>>> factors. First I fill the Matrix A with some values, and
>>> duplicate the structure of A to the other two matrices:
>>>
>>> MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);
>>> MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);
>>> MatDuplicate(A,MAT_DO_NOT_COPY_VALUES,&Ah);
>>> MatDuplicate(A,MAT_DO_NOT_COPY_VALUES,&At);
>>> MatAssemblyBegin(Ah,MAT_FINAL_ASSEMBLY);
>>> MatAssemblyEnd(Ah,MAT_FINAL_ASSEMBLY);
>>> MatAssemblyBegin(At,MAT_FINAL_ASSEMBLY);
>>> MatAssemblyEnd(At,MAT_FINAL_ASSEMBLY);
>>>
>>>
>>> After this I fill the matrices At and Ah with some other
>>> values, which are not beside the non zero structure (I
>>> also tried with just copying the Matrix A). Now after
>>> another MatAssembly
>>>
>>>
>>> I do not understand "no beside the nonzero structure". Do
>>> you mean that the nonzero structure is the same?
>>>
>>> Can you first test with MAT_COPY_VALUES?
>>>
>>> I want to add these Matrices in the form A+c*(Ah+d*At):
>>>
>>>
>>> MatAXPY(Ah,c,At,SAME_NONZERO_PATTERN);
>>> MatAXPY(A,d,Ah,SAME_NONZERO_PATTERN);
>>>
>>> When I run the method with mpi and one core everything
>>> works fine. Starting the same method with more cores,
>>> the sum of the matrices fails. It seems like some values
>>> are added
>>>
>>>
>>> Please send the full output of a failure when you use
>>> MAT_COPY_VALUES on 2 procs.
>>>
>>> Thanks,
>>>
>>> Matt
>>>
>>> correctly and many values are missed. Using
>>> DIFFERENT_NONZERO_STRUCTURE leads to the right behavior
>>> in the multi-core case, but is very slow. I checked with
>>> a viewer if all matrices have the same nonzero structure
>>> and this is the case.
>>>
>>> Does anyone know why this fails, or do I have made any
>>> wrong thoughts?
>>>
>>> I'm corrently working with a petsc version (Petsc
>>> Release Version 3.3.0, Patch 5, Sat Dec 1 15:10:41 CST
>>> 2012), I looked into the changelogs up to the current
>>> version and did not find any note about MatAXPY or MatAYPX.
>>>
>>>
>>> Best and Thanks a lot for your help
>>>
>>> Klaus
>>>
>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin
>>> their experiments is infinitely more interesting than any
>>> results to which their experiments lead.
>>> -- Norbert Wiener
>>
>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to
>> which their experiments lead.
>> -- Norbert Wiener
>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which
> their experiments lead.
> -- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150410/6b12ae01/attachment-0001.html>
More information about the petsc-users
mailing list