<div dir="ltr"><div><div><div><div><div>Progress Report:<br><br></div> So I modified my code to remove all the wrong usages of MatCopy where I was trying to AllScatter/AllGather a parallel matrix, and replaced with usage of MatGetRedundantMatrix().<a href="http://goo.gl/yyJVcV"> The highlighted diff is here</a>.<br>
<br></div>Now, running it with 1 proc crashes, I guess because t<a href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetRedundantMatrix.html">he documentation says</a> that only parallel matrices are supported by MatGetRedundantMatrix(), and running with 1 proc means the parallel matrices are sequantial (from what I read in the PetSc users manual).<br>
<br></div>However, running it with 2,3,4,5+ procs now yields identical and consistent results (unlike before), so the problem appears to have been solved.<br><br></div>I would be very grateful if an extra pair of eyes could take a quick gander at the diff and confirm if my changes are correct.<br>
<br></div>Thanks a bunch and Regards,<br><br><div><br><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Sep 25, 2013 at 12:55 PM, Analabha Roy <span dir="ltr"><<a href="mailto:hariseldon99@gmail.com" target="_blank">hariseldon99@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote"><div><div class="h5">On Wed, Sep 25, 2013 at 12:12 PM, Jed Brown <span dir="ltr"><<a href="mailto:jedbrown@mcs.anl.gov" target="_blank">jedbrown@mcs.anl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>Analabha Roy <<a href="mailto:hariseldon99@gmail.com" target="_blank">hariseldon99@gmail.com</a>> writes:<br>
<br>
> There is one thing<br>
><br>
> In the code, the evaluation of each element of AVG_BIBJ requires a<br>
> read-only matrix U_parallel that I input from another program, and a<br>
> writeable sequential vector BDB_AA that is different for each element.<br>
><br>
> I sequentiate U_parallel to U_seq by using MatCopy here in lines<br>
</div>> 242+<<a href="https://code.google.com/p/daneelrepo/source/browse/eth_question/eth.c#242" target="_blank">https://code.google.com/p/daneelrepo/source/browse/eth_question/eth.c#242</a>><br>
<br>
/*<br>
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -<br>
Create a sequential matrix and copy U_parallel to it<br>
so that each processor has complete local copy of U<br>
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -<br>
*/<br>
ierr = MatCreateSeqAIJ (PETSC_COMM_SELF, dim, dim, 0, PETSC_NULL, &U_seq);<br>
CHKERRQ (ierr);<br>
ierr = MatSetOption (U_seq, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);<br>
CHKERRQ (ierr);<br>
ierr = MatCopy (U_parallel, U_seq, DIFFERENT_NONZERO_PATTERN);<br>
<br>
This is wrong and should raise an error (definitely in a debugging<br>
build). Are you intending to create a redundant matrix containing all<br>
of U_parallel replicated onto each process? If so, you may as well<br>
assemble the whole thing redundantly, but if you'd rather distribute the<br>
computation of entries, you can use MatGetRedundantMatrix(). If you<br>
want something else, what is it?<br>
<br></blockquote></div></div><div><br><br>Thanks for the informative post. You're right in that I need to distribute the computation of entries, but need the data scattered/gathered to all procs afterwards. I thought MatCopy would do this. I'm not surprised that this is wrong. This part is probably what's wrong with the code.<br>
<br></div><div>I'm looking at MatGetRedundantMatrix() docs and using it to fix the problem. Will post results as soon as I'm done...<br><br></div><div class="im"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
CHKERRQ (ierr);<br>
ierr = MatAssemblyBegin (U_seq, MAT_FINAL_ASSEMBLY);<br>
CHKERRQ (ierr);<br>
ierr = MatAssemblyEnd (U_seq, MAT_FINAL_ASSEMBLY);<br>
CHKERRQ (ierr);<br>
<div><br>
<br>
> and each process is supposed to update its copy of BDB_AA at every loop<br>
> iteration here in line<br>
</div>> 347+<<a href="https://code.google.com/p/daneelrepo/source/browse/eth_question/eth.c#347" target="_blank">https://code.google.com/p/daneelrepo/source/browse/eth_question/eth.c#347</a>><br>
<div>><br>
><br>
> Is this right? Or are sequential vectors/matrices handled by the root<br>
> process only?<br>
<br>
</div>PETSC_COMM_SELF means the current process, so every process does that stuff.<br>
<div><br>
> I know how to scatter a parallel vector to all processes using PetSc<br>
> scatter contexts but don't see any way to do so to a matrix other than<br>
> MatCopy. How do I ensure that each process has its own private<br>
> writeable copy of a sequential vector?<br>
<br>
</div>Is the last sentence supposed to say "vector"? If so, see<br>
VecScatterCreateToAll.<br>
</blockquote></div></div><br><br clear="all"><div class="im"><br>-- <br><font face="tahoma, sans-serif">---</font><div><font face="tahoma, sans-serif"><b>Analabha Roy</b></font></div><div><font face="tahoma, sans-serif"><a href="http://www.csir.res.in" target="_blank">C.S.I.R</a> <a href="http://csirhrdg.res.in/poolsra.htm" target="_blank">Senior Research Associate</a></font></div>
<div><font face="tahoma, sans-serif"><a href="http://www.saha.ac.in" target="_blank">Saha Institute of Nuclear Physics</a></font></div><div><font face="tahoma, sans-serif">Section 1, Block AF</font></div><div><font face="tahoma, sans-serif">Bidhannagar, Calcutta 700064</font></div>
<div><font face="tahoma, sans-serif">India</font></div><div><b style="font-family:tahoma,sans-serif">Emails</b><span style="font-family:tahoma,sans-serif">: </span><a href="mailto:daneel@physics.utexas.edu" style="font-family:tahoma,sans-serif" target="_blank">daneel@physics.utexas.edu</a><span style="font-family:tahoma,sans-serif">, </span><a href="mailto:hariseldon99@gmail.com" style="font-family:tahoma,sans-serif" target="_blank">hariseldon99@gmail.com</a></div>
<div><font face="tahoma, sans-serif"><b>Webpage</b>: <a href="http://www.ph.utexas.edu/~daneel/" target="_blank">http://www.ph.utexas.edu/~daneel/</a></font></div>
</div></div></div>
</blockquote></div><br><br clear="all"><br>-- <br><font face="tahoma, sans-serif">---</font><div><font face="tahoma, sans-serif"><b>Analabha Roy</b></font></div><div><font face="tahoma, sans-serif"><a href="http://www.csir.res.in" target="_blank">C.S.I.R</a> <a href="http://csirhrdg.res.in/poolsra.htm" target="_blank">Senior Research Associate</a></font></div>
<div><font face="tahoma, sans-serif"><a href="http://www.saha.ac.in" target="_blank">Saha Institute of Nuclear Physics</a></font></div><div><font face="tahoma, sans-serif">Section 1, Block AF</font></div><div><font face="tahoma, sans-serif">Bidhannagar, Calcutta 700064</font></div>
<div><font face="tahoma, sans-serif">India</font></div><div><b style="font-family:tahoma,sans-serif">Emails</b><span style="font-family:tahoma,sans-serif">: </span><a href="mailto:daneel@physics.utexas.edu" style="font-family:tahoma,sans-serif" target="_blank">daneel@physics.utexas.edu</a><span style="font-family:tahoma,sans-serif">, </span><a href="mailto:hariseldon99@gmail.com" style="font-family:tahoma,sans-serif" target="_blank">hariseldon99@gmail.com</a></div>
<div><font face="tahoma, sans-serif"><b>Webpage</b>: <a href="http://www.ph.utexas.edu/~daneel/" target="_blank">http://www.ph.utexas.edu/~daneel/</a></font></div>
</div>