[petsc-users] Problem with MPI, MatAXPY and SAME_NONZERO_PATTERN
Matthew Knepley
knepley at gmail.com
Thu Apr 9 14:11:53 CDT 2015
On Fri, Mar 27, 2015 at 11:07 AM, Klaus Kaiser <kaiser at igpm.rwth-aachen.de>
wrote:
> Hallo Matthew,
>
> here is a short example I used for 8 cores:
>
> int d_nnz[3];
> int o_nnz[3];
> d_nnz[0] = 3; d_nnz[1] = 3; d_nnz[2] = 3;
> o_nnz[0] = 6; o_nnz[1] = 6; o_nnz[2] = 6;
> ierr = MatCreateBAIJ(PETSC_COMM_WORLD, 3, 9, 9, 72, 72, 0, d_nnz, 0,
> o_nnz,&A);
> ierr = MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
> ierr = MatSetOption(A,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
> ierr = MatSetOption(A,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);
> ierr = MatCreateBAIJ(PETSC_COMM_WORLD, 3, 9, 9, 72, 72, 0, d_nnz, 0,
> o_nnz,&Ah);
> ierr = MatCreateBAIJ(PETSC_COMM_WORLD, 3, 9, 9, 72, 72, 0, d_nnz, 0,
> o_nnz,&At);
>
> std::vector<double> insert(3*3*3*3, 1.0);
> for(int i=0;i<8;++i)
> {
> int rows[3] = {i,i+1,i+3};
> int cols[3] = {i,i+1,i+3};
>
> MatSetValuesBlocked(A, 3, rows, 3, cols, &insert[0], ADD_VALUES);
> }
>
I do not understand this example. All the procs (8) are filling up only the
first 32 rows. Is that
what you intended? Can you tell me what output you expect?
Thanks,
Matt
> MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);
> MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);
> MatDuplicate(A,MAT_COPY_VALUES,&Ah);
> MatDuplicate(A,MAT_COPY_VALUES,&At);
> MatAssemblyBegin(Ah,MAT_FINAL_ASSEMBLY);
> MatAssemblyEnd(Ah,MAT_FINAL_ASSEMBLY);
> MatAssemblyBegin(At,MAT_FINAL_ASSEMBLY);
> MatAssemblyEnd(At,MAT_FINAL_ASSEMBLY);
>
> MatAXPY(Ah,1.,At,SAME_NONZERO_PATTERN);
> MatAXPY(A,1.,Ah,SAME_NONZERO_PATTERN);
>
> MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);
> MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);
>
> The result is, that only some values are computed correctly.
>
> Best and Thank you a lot
>
> Klaus
>
>
>
>
> On 03/27/2015 04:26 PM, Matthew Knepley wrote:
>
> On Fri, Mar 27, 2015 at 10:16 AM, Klaus Kaiser <
> kaiser at igpm.rwth-aachen.de> wrote:
>
>> Hallo Matthew,
>>
>> thanks for your fast response. With "no beside the nonzero structure" I
>> meant, that I do not create a different non-zero structure while I'm adding
>> values to my matrices.
>>
>> I also tried MAT_COPY_VALUES and the result was the same.
>>
>
> Please send the small example and we will figure out what is going wrong.
>
> Thanks,
>
> Matt
>
>
>> I do not get a error message but the resulting matrix is wrong. Here I
>> have a short example. The first 9 rows and first 3 colomn of the matrices
>> Ah, At and Ah+At on the first processor (of 8):
>>
>> Ah:
>> 0: 60.3553 -0.249975
>> 2.77556e-17
>> 1: 0
>> 60.3553 0
>> 2: 0 0
>> 60.3553
>> 3: 17.6777 0.374962
>> 0.124987
>> 4: 0
>> 17.6777 0
>> 5: 0 0
>> 17.6777
>> 6: -7.32233 -0.124987
>> 0.374962
>> 7: 0
>> -7.32233 0
>> 8: 0 0
>> -7.32233
>>
>>
>> At:
>> 0: 0
>> 0 0
>> 1: 2500
>> 0 0
>> 2: -4.54747e-13
>> 0 0
>> 3: 0
>> 0 0
>> 4: 1250
>> 0 0
>> 5: 1250
>> 0 0
>> 6: 0
>> 0 0
>> 7: 1250
>> 0 0
>> 8: 3750
>> 0 0
>>
>> Ah+At
>> 0: 60.3553 -0.249975
>> 2.77556e-17
>> 1: 2500
>> 60.3553 0
>> 2: -4.54747e-13 0
>> 60.3553
>> 3: 17.6777 0.374962
>> 0.124987
>> 4: 0
>> 17.6777 0
>> 5: 0 0
>> 17.6777
>> 6: -7.32233 -0.124987
>> 0.374962
>> 7: 0
>> -7.32233 0
>> 8: 0 0
>> -7.32233
>>
>> you can see the first 3 rows of the resulting matrix looks exactly like
>> what I would expect, but the last 6 rows only consists of the values of Ah.
>> When you would also look on the matrix A and A+Ah+At you would also see,
>> that values where both matrices have an nonzero entry are not sum correctly.
>>
>> Best
>>
>> Klaus
>>
>> On 03/27/2015 03:59 PM, Matthew Knepley wrote:
>>
>> On Fri, Mar 27, 2015 at 9:48 AM, Klaus Kaiser <
>> kaiser at igpm.rwth-aachen.de> wrote:
>>
>>> Hallo,
>>>
>>> I have a strange behavior in my code concerning the function MatAXPY. I
>>> create 3 different Matrices
>>>
>>> ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size, local_size,
>>> local_size, system_size, system_size, 0, d_nnz, 0, o_nnz,&A);
>>> ierr = MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
>>> ierr = MatSetOption(A,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
>>> ierr = MatSetOption(A,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);
>>>
>>> ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size, local_size,
>>> local_size, system_size, system_size, 0, d_nnz, 0, o_nnz,&At);
>>> ierr = MatSetOption(At,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
>>> ierr = MatSetOption(At,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
>>> ierr = MatSetOption(At,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);
>>>
>>> ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size, local_size,
>>> local_size, system_size, system_size, 0, d_nnz, 0, o_nnz,&Ah);
>>> ierr = MatSetOption(Ah,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
>>> ierr = MatSetOption(Ah,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
>>> ierr = MatSetOption(Ah,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);
>>>
>>
>> These creations are superfluous since you use MatDuplicate() below.
>>
>>
>>> and want to sum these three matrixes with different factors. First I
>>> fill the Matrix A with some values, and duplicate the structure of A to the
>>> other two matrices:
>>>
>>> MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);
>>> MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);
>>> MatDuplicate(A,MAT_DO_NOT_COPY_VALUES,&Ah);
>>> MatDuplicate(A,MAT_DO_NOT_COPY_VALUES,&At);
>>> MatAssemblyBegin(Ah,MAT_FINAL_ASSEMBLY);
>>> MatAssemblyEnd(Ah,MAT_FINAL_ASSEMBLY);
>>> MatAssemblyBegin(At,MAT_FINAL_ASSEMBLY);
>>> MatAssemblyEnd(At,MAT_FINAL_ASSEMBLY);
>>>
>>
>>
>>> After this I fill the matrices At and Ah with some other values, which
>>> are not beside the non zero structure (I also tried with just copying the
>>> Matrix A). Now after another MatAssembly
>>
>>
>> I do not understand "no beside the nonzero structure". Do you mean that
>> the nonzero structure is the same?
>>
>> Can you first test with MAT_COPY_VALUES?
>>
>>
>>> I want to add these Matrices in the form A+c*(Ah+d*At):
>>
>>
>>> MatAXPY(Ah,c,At,SAME_NONZERO_PATTERN);
>>> MatAXPY(A,d,Ah,SAME_NONZERO_PATTERN);
>>>
>>> When I run the method with mpi and one core everything works fine.
>>> Starting the same method with more cores, the sum of the matrices fails. It
>>> seems like some values are added
>>
>>
>> Please send the full output of a failure when you use MAT_COPY_VALUES
>> on 2 procs.
>>
>> Thanks,
>>
>> Matt
>>
>>
>>> correctly and many values are missed. Using DIFFERENT_NONZERO_STRUCTURE
>>> leads to the right behavior in the multi-core case, but is very slow. I
>>> checked with a viewer if all matrices have the same nonzero structure and
>>> this is the case.
>>>
>>> Does anyone know why this fails, or do I have made any wrong thoughts?
>>>
>>> I'm corrently working with a petsc version (Petsc Release Version 3.3.0,
>>> Patch 5, Sat Dec 1 15:10:41 CST 2012), I looked into the changelogs up to
>>> the current version and did not find any note about MatAXPY or MatAYPX.
>>>
>>>
>>> Best and Thanks a lot for your help
>>>
>>> Klaus
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>
--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150409/2c953ac3/attachment-0001.html>
More information about the petsc-users
mailing list