<div dir="ltr">Hmm, so I made that change, but my program still exhibits the same problem with it's output.</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jun 26, 2014 at 9:05 PM, Peter Brune <span dir="ltr"><<a href="mailto:prbrune@gmail.com" target="_blank">prbrune@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Yep! Looks good. One other thing is that you can pre/postface all PETSc calls with ierr = ... CHKERRQ(ierr); and it will properly trace other problems if they arise.<span class="HOEnZb"><font color="#888888"><div>
<br></div><div>- Peter</div></font></span></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra">
<br><br><div class="gmail_quote">On Thu, Jun 26, 2014 at 8:56 PM, Arthur Kurlej <span dir="ltr"><<a href="mailto:akurlej@gmail.com" target="_blank">akurlej@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Hello,<div><br></div><div>I'm sorry, but this is a bit new for me, and I'm still not quite sure I follow.</div><div><br></div><div>Are you recommending that opposed to doing this:<br></div><div><div>
if(procs!=1){</div><div><span style="white-space:pre-wrap"> </span>for(i=0;i<procs;i++){</div><div><span style="white-space:pre-wrap"> </span>if(rank==i){</div><div><span style="white-space:pre-wrap"> </span>VecGetLocalSize(*x,&length);</div>
<div><span style="white-space:pre-wrap"> </span>VecGetOwnershipRange(*x,&div,NULL);</div><div><span style="white-space:pre-wrap"> </span>ISCreateStride(PETSC_COMM_WORLD,length,div+begin,1,&iscol);</div>
<div><span style="white-space:pre-wrap"> </span>ISCreateStride(PETSC_COMM_WORLD,length,div+begin,1,&isrow);</div><div><span style="white-space:pre-wrap"> </span>ierr = MatGetSubMatrix(*A,isrow,iscol,MAT_INITIAL_MATRIX,AA); CHKERRQ(ierr);</div>
<div><span style="white-space:pre-wrap"> </span>}</div><div><span style="white-space:pre-wrap"> </span>}</div><div>}</div><div>else{</div><div>ISCreateStride(PETSC_COMM_SELF,final_size,begin,1,&iscol);</div><div>
ISCreateStride(PETSC_COMM_SELF,final_size,begin,1,&isrow);</div><div>ierr = MatGetSubMatrix(*A,isrow,iscol,MAT_INITIAL_MATRIX,AA); CHKERRQ(ierr);</div><div>}</div></div><div class="gmail_extra"><br></div><div class="gmail_extra">
<br></div><div class="gmail_extra">The proper implementation would instead just be the following:</div><div class="gmail_extra"><div class="gmail_extra">VecGetLocalSize(*x,&length);</div><div class="gmail_extra">VecGetOwnershipRange(*x,&div,NULL);</div>
<div class="gmail_extra">ISCreateStride(PETSC_COMM_WORLD,length,div+begin,1,&iscol);</div><div class="gmail_extra">ISCreateStride(PETSC_COMM_WORLD,length,div+begin,1,&isrow);</div><div class="gmail_extra">ierr = MatGetSubMatrix(*A,isrow,iscol,MAT_INITIAL_MATRIX,AA); CHKERRQ(ierr);</div>
<div>?</div><div><br></div><div><br></div></div><div><div><div class="gmail_extra"><br></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jun 26, 2014 at 5:32 PM, Peter Brune <span dir="ltr"><<a href="mailto:prbrune@gmail.com" target="_blank">prbrune@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">MatGetSubMatrix() is collective on Mat and ISCreateXXX is collective on the provided comm, so the logic you have built to call it on one proc at a time is unnecessary at best and most likely incorrect and likely to produce strange results. You can forgo the if statement and loop over processors, create the ISes on the same comm as x, and then call MatGetSubMatrix() once.<span><font color="#888888"><div>
<br></div><div>- Peter</div></font></span></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jun 26, 2014 at 4:26 PM, Arthur Kurlej <span dir="ltr"><<a href="mailto:akurlej@gmail.com" target="_blank">akurlej@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div>I cannot send the original code, but I reproduced the problem in another code. I have attached a makefile the code, and the data for the x vector and A matrix.<br>
<br></div>I think the problem may be with my ShortenMatrix function, but it's not clear to me what exactly is going wrong and how to fix it. So I would appreciate some assistance there.<br>
<br><br>Thanks,<br>Arthur<br><div><br></div></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Jun 25, 2014 at 6:24 PM, Barry Smith <span dir="ltr"><<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><br>
Can you send the code that reproduces this behavior?<br>
<span><font color="#888888"><br>
Barry<br>
</font></span><div><div><br>
On Jun 25, 2014, at 4:37 PM, Arthur Kurlej <<a href="mailto:akurlej@gmail.com" target="_blank">akurlej@gmail.com</a>> wrote:<br>
<br>
> Hi Barry,<br>
><br>
> So for the matrix C that I am currently testing (size 162x162), the condition number is roughly 10^4.<br>
><br>
> For reference, I'm porting MATLAB code into PETSc, and for one processor, the PETSc b vector is roughly equivalent to the MATLAB b vector. So I know that for one processor, my program is performing as expected.<br>
><br>
> I've included examples below of values for b (also of size 162), ranging from indices 131 to 141.<br>
><br>
> #processors=1:<br>
> 0<br>
> 1.315217173959314e-20<br>
> 1.315217173959314e-20<br>
> 4.843201487740107e-17<br>
> 4.843201487740107e-17<br>
> 8.166104700666665e-14<br>
> 8.166104700666665e-14<br>
> 6.303834267553249e-11<br>
> 6.303834267553249e-11<br>
> 2.227932688485483e-08<br>
> 2.227932688485483e-08<br>
><br>
> # processors=2:<br>
> 5.480410831461926e-22<br>
> 2.892553944350444e-22<br>
> 2.892553944350444e-22<br>
> 7.524038923310717e-24<br>
> 7.524038923214420e-24<br>
> -3.340766769043093e-26<br>
> -7.558372155761972e-27<br>
> 5.551561288838557e-25<br>
> 5.550551546879874e-25<br>
> -1.579397982093437e-22<br>
> 2.655766754178065e-22<br>
><br>
> # processors = 4:<br>
> 5.480410831461926e-22<br>
> 2.892553944351728e-22<br>
> 2.892553944351728e-22<br>
> 7.524092205125593e-24<br>
> 7.524092205125593e-24<br>
> -2.584939414228212e-26<br>
> -2.584939414228212e-26<br>
> 0<br>
> 0<br>
> -1.245940797657998e-23<br>
> -1.245940797657998e-23<br>
><br>
> # processors = 8:<br>
> 5.480410831461926e-22<br>
> 2.892553944023035e-22<br>
> 2.892553944023035e-22<br>
> 7.524065744581494e-24<br>
> 7.524065744581494e-24<br>
> -2.250265175188197e-26<br>
> -2.250265175188197e-26<br>
> -6.543127892265160e-26<br>
> 1.544288143499193e-317<br>
> 8.788794008375919e-25<br>
> 8.788794008375919e-25<br>
><br>
><br>
> Thanks,<br>
> Arthur<br>
><br>
><br>
><br>
> On Wed, Jun 25, 2014 at 4:06 PM, Barry Smith <<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>> wrote:<br>
><br>
> How different are the values in b? Can you send back a few examples of the different b’s? Any idea of the condition number of C?<br>
><br>
> Barry<br>
><br>
> On Jun 25, 2014, at 3:10 PM, Arthur Kurlej <<a href="mailto:akurlej@gmail.com" target="_blank">akurlej@gmail.com</a>> wrote:<br>
><br>
> > Hi all,<br>
> ><br>
> > While running my code, I have found that MatMult() returns different values depending on the number of processors I use (and there is quite the variance in the values).<br>
> ><br>
> > The setup of my code is as follows (I can go into more depth/background if needed):<br>
> > -Generate parallel AIJ matrix of size NxN, denoted as A<br>
> > -Retrieve parallel AIJ submatrix from the last N-1 rows&columns from A, denoted as C<br>
> > -Generate vector of length N-1, denoted as x<br>
> > -Find C*x=b<br>
> ><br>
> > I have already checked that A, C, and x are all equivalent when ran for any number of processors, it is only the values of vector b that varies.<br>
> ><br>
> > Does anyone have an idea about what's going on?<br>
> ><br>
> ><br>
> > Thanks,<br>
> > Arthur<br>
> ><br>
><br>
><br>
<br>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div></div></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>