[petsc-users] Problem with MPI, MatAXPY and SAME_NONZERO_PATTERN
Klaus Kaiser
kaiser at igpm.rwth-aachen.de
Wed Apr 8 01:46:40 CDT 2015
Hallo Metthew,
you were albe to reconstruct this problem on your petsc installation?
Best
Klaus
On 03/30/2015 01:43 PM, Matthew Knepley wrote:
> On Mon, Mar 30, 2015 at 5:18 AM, Klaus Kaiser
> <kaiser at igpm.rwth-aachen.de <mailto:kaiser at igpm.rwth-aachen.de>> wrote:
>
> Hallo Metthew,
>
> at the weekend I updated petsc to the newest release version 3.5.2
> and I still got this strange behavior.
>
>
> Okay, I will check it out. The first thing I am going to do is try
> with AIJ matrices instead of BAIJ, which
> you can probably do before me. If you get the right result, then its a
> bug in the BAIJ implementation,
> which I am confident has not been tested as thoroughly.
>
> Thanks,
>
> Matt
>
> Best
>
> Klaus
>
>
> On 03/27/2015 05:07 PM, Klaus Kaiser wrote:
>> Hallo Matthew,
>>
>> here is a short example I used for 8 cores:
>>
>> int d_nnz[3];
>> int o_nnz[3];
>> d_nnz[0] = 3; d_nnz[1] = 3; d_nnz[2] = 3;
>> o_nnz[0] = 6; o_nnz[1] = 6; o_nnz[2] = 6;
>> ierr = MatCreateBAIJ(PETSC_COMM_WORLD, 3, 9, 9, 72, 72, 0,
>> d_nnz, 0, o_nnz,&A);
>> ierr = MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
>> ierr = MatSetOption(A,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
>> ierr = MatSetOption(A,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);
>> ierr = MatCreateBAIJ(PETSC_COMM_WORLD, 3, 9, 9, 72, 72, 0,
>> d_nnz, 0, o_nnz,&Ah);
>> ierr = MatCreateBAIJ(PETSC_COMM_WORLD, 3, 9, 9, 72, 72, 0,
>> d_nnz, 0, o_nnz,&At);
>>
>> std::vector<double> insert(3*3*3*3, 1.0);
>> for(int i=0;i<8;++i)
>> {
>> int rows[3] = {i,i+1,i+3};
>> int cols[3] = {i,i+1,i+3};
>>
>> MatSetValuesBlocked(A, 3, rows, 3, cols, &insert[0],
>> ADD_VALUES);
>> }
>>
>> MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);
>> MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);
>> MatDuplicate(A,MAT_COPY_VALUES,&Ah);
>> MatDuplicate(A,MAT_COPY_VALUES,&At);
>> MatAssemblyBegin(Ah,MAT_FINAL_ASSEMBLY);
>> MatAssemblyEnd(Ah,MAT_FINAL_ASSEMBLY);
>> MatAssemblyBegin(At,MAT_FINAL_ASSEMBLY);
>> MatAssemblyEnd(At,MAT_FINAL_ASSEMBLY);
>>
>> MatAXPY(Ah,1.,At,SAME_NONZERO_PATTERN);
>> MatAXPY(A,1.,Ah,SAME_NONZERO_PATTERN);
>>
>> MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);
>> MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);
>>
>> The result is, that only some values are computed correctly.
>>
>> Best and Thank you a lot
>>
>> Klaus
>>
>>
>>
>> On 03/27/2015 04:26 PM, Matthew Knepley wrote:
>>> On Fri, Mar 27, 2015 at 10:16 AM, Klaus Kaiser
>>> <kaiser at igpm.rwth-aachen.de <mailto:kaiser at igpm.rwth-aachen.de>>
>>> wrote:
>>>
>>> Hallo Matthew,
>>>
>>> thanks for your fast response. With "no beside the nonzero
>>> structure" I meant, that I do not create a different
>>> non-zero structure while I'm adding values to my matrices.
>>>
>>> I also tried MAT_COPY_VALUES and the result was the same.
>>>
>>>
>>> Please send the small example and we will figure out what is
>>> going wrong.
>>>
>>> Thanks,
>>>
>>> Matt
>>>
>>> I do not get a error message but the resulting matrix is
>>> wrong. Here I have a short example. The first 9 rows and
>>> first 3 colomn of the matrices Ah, At and Ah+At on the first
>>> processor (of 8):
>>>
>>> Ah:
>>> 0: 60.3553 -0.249975 2.77556e-17
>>> 1: 0 60.3553 0
>>> 2: 0 0 60.3553
>>> 3: 17.6777 0.374962 0.124987
>>> 4: 0 17.6777 0
>>> 5: 0 0 17.6777
>>> 6: -7.32233 -0.124987 0.374962
>>> 7: 0 -7.32233 0
>>> 8: 0 0 -7.32233
>>>
>>>
>>> At:
>>> 0: 0 0 0
>>> 1: 2500 0 0
>>> 2: -4.54747e-13 0 0
>>> 3: 0 0 0
>>> 4: 1250 0 0
>>> 5: 1250 0 0
>>> 6: 0 0 0
>>> 7: 1250 0 0
>>> 8: 3750 0 0
>>>
>>> Ah+At
>>> 0: 60.3553 -0.249975 2.77556e-17
>>> 1: 2500 60.3553 0
>>> 2: -4.54747e-13 0 60.3553
>>> 3: 17.6777 0.374962 0.124987
>>> 4: 0 17.6777 0
>>> 5: 0 0 17.6777
>>> 6: -7.32233 -0.124987 0.374962
>>> 7: 0 -7.32233 0
>>> 8: 0 0 -7.32233
>>>
>>> you can see the first 3 rows of the resulting matrix looks
>>> exactly like what I would expect, but the last 6 rows only
>>> consists of the values of Ah. When you would also look on
>>> the matrix A and A+Ah+At you would also see, that values
>>> where both matrices have an nonzero entry are not sum correctly.
>>>
>>> Best
>>>
>>> Klaus
>>>
>>> On 03/27/2015 03:59 PM, Matthew Knepley wrote:
>>>> On Fri, Mar 27, 2015 at 9:48 AM, Klaus Kaiser
>>>> <kaiser at igpm.rwth-aachen.de
>>>> <mailto:kaiser at igpm.rwth-aachen.de>> wrote:
>>>>
>>>> Hallo,
>>>>
>>>> I have a strange behavior in my code concerning the
>>>> function MatAXPY. I create 3 different Matrices
>>>>
>>>> ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size,
>>>> local_size, local_size, system_size, system_size, 0,
>>>> d_nnz, 0, o_nnz,&A);
>>>> ierr =
>>>> MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
>>>> ierr =
>>>> MatSetOption(A,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
>>>> ierr =
>>>> MatSetOption(A,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);
>>>>
>>>> ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size,
>>>> local_size, local_size, system_size, system_size, 0,
>>>> d_nnz, 0, o_nnz,&At);
>>>> ierr =
>>>> MatSetOption(At,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
>>>> ierr =
>>>> MatSetOption(At,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
>>>> ierr =
>>>> MatSetOption(At,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);
>>>>
>>>> ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size,
>>>> local_size, local_size, system_size, system_size, 0,
>>>> d_nnz, 0, o_nnz,&Ah);
>>>> ierr =
>>>> MatSetOption(Ah,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
>>>> ierr =
>>>> MatSetOption(Ah,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
>>>> ierr =
>>>> MatSetOption(Ah,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);
>>>>
>>>>
>>>> These creations are superfluous since you use
>>>> MatDuplicate() below.
>>>>
>>>> and want to sum these three matrixes with different
>>>> factors. First I fill the Matrix A with some values,
>>>> and duplicate the structure of A to the other two matrices:
>>>>
>>>> MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);
>>>> MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);
>>>> MatDuplicate(A,MAT_DO_NOT_COPY_VALUES,&Ah);
>>>> MatDuplicate(A,MAT_DO_NOT_COPY_VALUES,&At);
>>>> MatAssemblyBegin(Ah,MAT_FINAL_ASSEMBLY);
>>>> MatAssemblyEnd(Ah,MAT_FINAL_ASSEMBLY);
>>>> MatAssemblyBegin(At,MAT_FINAL_ASSEMBLY);
>>>> MatAssemblyEnd(At,MAT_FINAL_ASSEMBLY);
>>>>
>>>>
>>>> After this I fill the matrices At and Ah with some
>>>> other values, which are not beside the non zero
>>>> structure (I also tried with just copying the Matrix
>>>> A). Now after another MatAssembly
>>>>
>>>>
>>>> I do not understand "no beside the nonzero structure". Do
>>>> you mean that the nonzero structure is the same?
>>>>
>>>> Can you first test with MAT_COPY_VALUES?
>>>>
>>>> I want to add these Matrices in the form A+c*(Ah+d*At):
>>>>
>>>>
>>>> MatAXPY(Ah,c,At,SAME_NONZERO_PATTERN);
>>>> MatAXPY(A,d,Ah,SAME_NONZERO_PATTERN);
>>>>
>>>> When I run the method with mpi and one core everything
>>>> works fine. Starting the same method with more cores,
>>>> the sum of the matrices fails. It seems like some
>>>> values are added
>>>>
>>>>
>>>> Please send the full output of a failure when you use
>>>> MAT_COPY_VALUES on 2 procs.
>>>>
>>>> Thanks,
>>>>
>>>> Matt
>>>>
>>>> correctly and many values are missed. Using
>>>> DIFFERENT_NONZERO_STRUCTURE leads to the right behavior
>>>> in the multi-core case, but is very slow. I checked
>>>> with a viewer if all matrices have the same nonzero
>>>> structure and this is the case.
>>>>
>>>> Does anyone know why this fails, or do I have made any
>>>> wrong thoughts?
>>>>
>>>> I'm corrently working with a petsc version (Petsc
>>>> Release Version 3.3.0, Patch 5, Sat Dec 1 15:10:41 CST
>>>> 2012), I looked into the changelogs up to the current
>>>> version and did not find any note about MatAXPY or MatAYPX.
>>>>
>>>>
>>>> Best and Thanks a lot for your help
>>>>
>>>> Klaus
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin
>>>> their experiments is infinitely more interesting than any
>>>> results to which their experiments lead.
>>>> -- Norbert Wiener
>>>
>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to
>>> which their experiments lead.
>>> -- Norbert Wiener
>>
>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which
> their experiments lead.
> -- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150408/1b710b11/attachment-0001.html>
More information about the petsc-users
mailing list