[petsc-users] MatMult() returning different values depending on # of processors?

[Arthur Kurlej]

Hello,

I'm sorry, but this is a bit new for me, and I'm still not quite sure I
follow.

Are you recommending that opposed to doing this:
if(procs!=1){
for(i=0;i<procs;i++){
if(rank==i){
VecGetLocalSize(*x,&length);
VecGetOwnershipRange(*x,&div,NULL);
ISCreateStride(PETSC_COMM_WORLD,length,div+begin,1,&iscol);
ISCreateStride(PETSC_COMM_WORLD,length,div+begin,1,&isrow);
ierr = MatGetSubMatrix(*A,isrow,iscol,MAT_INITIAL_MATRIX,AA); CHKERRQ(ierr);
}
}
}
else{
ISCreateStride(PETSC_COMM_SELF,final_size,begin,1,&iscol);
ISCreateStride(PETSC_COMM_SELF,final_size,begin,1,&isrow);
ierr = MatGetSubMatrix(*A,isrow,iscol,MAT_INITIAL_MATRIX,AA); CHKERRQ(ierr);
}


The proper implementation would instead just be the following:
VecGetLocalSize(*x,&length);
VecGetOwnershipRange(*x,&div,NULL);
ISCreateStride(PETSC_COMM_WORLD,length,div+begin,1,&iscol);
ISCreateStride(PETSC_COMM_WORLD,length,div+begin,1,&isrow);
ierr = MatGetSubMatrix(*A,isrow,iscol,MAT_INITIAL_MATRIX,AA); CHKERRQ(ierr);
?





On Thu, Jun 26, 2014 at 5:32 PM, Peter Brune <prbrune at gmail.com> wrote:

> MatGetSubMatrix() is collective on Mat and ISCreateXXX is collective on
> the provided comm, so the logic you have built to call it on one proc at a
> time is unnecessary at best and most likely incorrect and likely to produce
> strange results.  You can forgo the if statement and loop over processors,
> create the ISes on the same comm as x, and then call MatGetSubMatrix() once.
>
> - Peter
>
>
> On Thu, Jun 26, 2014 at 4:26 PM, Arthur Kurlej <akurlej at gmail.com> wrote:
>
>> I cannot send the original code, but I reproduced the problem in another
>> code. I have attached a makefile the code, and the data for the x vector
>> and A matrix.
>>
>> I think the problem may be with my ShortenMatrix function, but it's not
>> clear to me what exactly is going wrong and how to fix it. So I would
>> appreciate some assistance there.
>>
>>
>> Thanks,
>> Arthur
>>
>>
>>
>> On Wed, Jun 25, 2014 at 6:24 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>
>>>
>>>    Can you send the code that reproduces this behavior?
>>>
>>>    Barry
>>>
>>> On Jun 25, 2014, at 4:37 PM, Arthur Kurlej <akurlej at gmail.com> wrote:
>>>
>>> > Hi Barry,
>>> >
>>> > So for the matrix C that I am currently testing (size 162x162), the
>>> condition number is roughly 10^4.
>>> >
>>> > For reference, I'm porting MATLAB code into PETSc, and for one
>>> processor, the PETSc b vector is roughly equivalent to the MATLAB b vector.
>>> So I know that for one processor, my program is performing as expected.
>>> >
>>> > I've included examples below of values for b (also of size 162),
>>> ranging from indices 131 to 141.
>>> >
>>> > #processors=1:
>>> >                          0
>>> >      1.315217173959314e-20
>>> >      1.315217173959314e-20
>>> >      4.843201487740107e-17
>>> >      4.843201487740107e-17
>>> >      8.166104700666665e-14
>>> >      8.166104700666665e-14
>>> >      6.303834267553249e-11
>>> >      6.303834267553249e-11
>>> >      2.227932688485483e-08
>>> >      2.227932688485483e-08
>>> >
>>> > # processors=2:
>>> >      5.480410831461926e-22
>>> >      2.892553944350444e-22
>>> >      2.892553944350444e-22
>>> >      7.524038923310717e-24
>>> >      7.524038923214420e-24
>>> >     -3.340766769043093e-26
>>> >     -7.558372155761972e-27
>>> >      5.551561288838557e-25
>>> >      5.550551546879874e-25
>>> >     -1.579397982093437e-22
>>> >      2.655766754178065e-22
>>> >
>>> > # processors = 4:
>>> >      5.480410831461926e-22
>>> >      2.892553944351728e-22
>>> >      2.892553944351728e-22
>>> >      7.524092205125593e-24
>>> >      7.524092205125593e-24
>>> >     -2.584939414228212e-26
>>> >     -2.584939414228212e-26
>>> >                          0
>>> >                          0
>>> >     -1.245940797657998e-23
>>> >     -1.245940797657998e-23
>>> >
>>> > # processors = 8:
>>> >      5.480410831461926e-22
>>> >      2.892553944023035e-22
>>> >      2.892553944023035e-22
>>> >      7.524065744581494e-24
>>> >      7.524065744581494e-24
>>> >     -2.250265175188197e-26
>>> >     -2.250265175188197e-26
>>> >     -6.543127892265160e-26
>>> >     1.544288143499193e-317
>>> >      8.788794008375919e-25
>>> >      8.788794008375919e-25
>>> >
>>> >
>>> > Thanks,
>>> > Arthur
>>> >
>>> >
>>> >
>>> > On Wed, Jun 25, 2014 at 4:06 PM, Barry Smith <bsmith at mcs.anl.gov>
>>> wrote:
>>> >
>>> >    How different are the values in b? Can you send back a few examples
>>> of the different b’s? Any idea of the condition number of C?
>>> >
>>> >    Barry
>>> >
>>> > On Jun 25, 2014, at 3:10 PM, Arthur Kurlej <akurlej at gmail.com> wrote:
>>> >
>>> > > Hi all,
>>> > >
>>> > > While running my code, I have found that MatMult() returns different
>>> values depending on the number of processors I use (and there is quite the
>>> variance in the values).
>>> > >
>>> > > The setup of my code is as follows (I can go into more
>>> depth/background if needed):
>>> > > -Generate parallel AIJ matrix of size NxN, denoted as A
>>> > > -Retrieve parallel AIJ submatrix from the last N-1 rows&columns from
>>> A, denoted as C
>>> > > -Generate vector of length N-1, denoted as x
>>> > > -Find C*x=b
>>> > >
>>> > > I have already checked that A, C, and x are all equivalent when ran
>>> for any number of processors, it is only the values of vector b that varies.
>>> > >
>>> > > Does anyone have an idea about what's going on?
>>> > >
>>> > >
>>> > > Thanks,
>>> > > Arthur
>>> > >
>>> >
>>> >
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20140626/67c1e64e/attachment.html>