<div dir="ltr"><div><div><div><div><div>Hi,<br></div></div></div></div></div><span class=""><br></span><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Sep 24, 2013 at 9:33 PM, Matthew Knepley <span dir="ltr"><<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div class="h5">On Tue, Sep 24, 2013 at 8:35 AM, Analabha Roy <span dir="ltr"><<a href="mailto:hariseldon99@gmail.com" target="_blank">hariseldon99@gmail.com</a>></span> wrote:<br>
</div></div><div class="gmail_extra"><div class="gmail_quote"><div><div class="h5">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Sep 24, 2013 at 8:41 PM, Matthew Knepley <span dir="ltr"><<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>On Tue, Sep 24, 2013 at 8:08 AM, Analabha Roy <span dir="ltr"><<a href="mailto:hariseldon99@gmail.com" target="_blank">hariseldon99@gmail.com</a>></span> wrote:<br>
</div><div class="gmail_extra"><div class="gmail_quote"><div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_quote"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">
<div>On Tue, Sep 24, 2013 at 1:42 PM, Jed Brown <span dir="ltr"><<a href="mailto:jedbrown@mcs.anl.gov" target="_blank">jedbrown@mcs.anl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>Analabha Roy <<a href="mailto:hariseldon99@gmail.com" target="_blank">hariseldon99@gmail.com</a>> writes:<br>
<br>
> Hi all,<br>
><br>
><br>
> Compiling and running this<br>
</div>> code<<a href="https://code.google.com/p/daneelrepo/source/browse/eth_question/eth.c" target="_blank">https://code.google.com/p/daneelrepo/source/browse/eth_question/eth.c</a>>that<br>
<div>> builds a petsc matrix gives different results when run with different<br>
> number of processors.<br>
<br></div></blockquote><div><br></div></div><div>Thanks for the reply.<br></div><div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
</div>Uh, if you call rand() on different processors, why would you expect it<br>
to give the same results?<br>
<br></blockquote></div><div>Right, I get that. The rand() was a placeholder.<br><br></div><div><a href="https://code.google.com/p/daneelrepo/source/browse/eth_question/eth.c" target="_blank">This original much larger code</a> replicates the same loop structure and runs the same Petsc subroutines, but running it by<br>
<br>mpirun -np $N ./eth -lattice_size 5 -vector_size 1 -repulsion 0.0 -draw_out -draw_pause -1<br></div><div><br></div><div>with N=1,2,3,4 gives different results for the matrix dumped out by lines <a href="https://code.google.com/p/daneelrepo/source/browse/eth_question/eth.c#514" target="_blank">514-519</a>. The matrix itself is evaluated in parallel, created in lines<a href="https://code.google.com/p/daneelrepo/source/browse/eth_question/eth.c#263" target="_blank"> 263-275 </a>and evaluated in lines <a href="https://code.google.com/p/daneelrepo/source/browse/eth_question/eth.c#294" target="_blank">294-356</a><br>
<br></div><div>(you can click on the line numbers above to navigate directly to them)<br></div><div><br></div><div><a href="http://i43.tinypic.com/zyhf2f.jpg" target="_blank">Here is a sample</a> of the output of lines <a href="https://code.google.com/p/daneelrepo/source/browse/eth_question/eth.c#514" target="_blank">514-519</a> for N=1,2,3,4 procs left to right.<br>
<br></div><div>Thty're different for different procs. They should be the same, since none of my input parameters are numprocs dependent, and I don't explicitly use the size or rank anywhere in the code.<br></div>
</div></div></div></div></div></blockquote><div><br></div></div><div>You are likely not dividing the rows you loop over so you are redundantly computing.</div></div></div></div></blockquote><div><br></div><div>Thanks for the reply.<br>
<br></div><div><a href="https://code.google.com/p/daneelrepo/source/browse/eth_question/eth.c#274" target="_blank">Line 274</a> gets the local row indices of Petsc Matrix <span>AVG_BDIBJ</span></div><div><br></div><div><a href="https://code.google.com/p/daneelrepo/source/browse/eth_question/eth.c#295" target="_blank">Line 295 </a>iterates over the local rows and the lines below get the column <br>
elements. For each row, the column elements are assigned by the lines up to <a href="https://code.google.com/p/daneelrepo/source/browse/eth_question/eth.c#344" target="_blank">Line 344</a> and stored locally in colvalues[]. Dunno if the details are relevant.<br>
<br></div><div><a href="https://code.google.com/p/daneelrepo/source/browse/eth_question/eth.c#347" target="_blank">Line 347</a> inserts the sitestride1^th row into the matrix<br></div><div><br></div><div><a href="https://code.google.com/p/daneelrepo/source/browse/eth_question/eth.c#353" target="_blank">Line 353+</a> does the mat assembly<br>
<br></div><div>Then, after a lot of currently irrelevant code, <br><br></div><div><a href="https://code.google.com/p/daneelrepo/source/browse/eth_question/eth.c#514" target="_blank">Line 514+</a> dumps the mat plot to graphics<br>
<br><br>
</div><div>Different numprocs give different matrices.<br><br></div><div>Can somebody suggest what I did wrong (or didn't do)?</div></div></div></div></blockquote><div><br></div></div></div><div>Different values are being given to MatSetValues() for different numbers of processes. So</div>
<div><br></div><div> 1) Reduce this to the smallest problem size possible</div><div><br></div><div> 2) Print out all rows/cols/values for each call</div><div><br></div><div> 3) Compare 2 procs to the serial case</div>
<div>
<br></div></div></div></div></blockquote><div><br><br>Thanks for your excellent suggestion.<br><br><a href="https://code.google.com/p/daneelrepo/source/diff?spec=svn1435&r=1435&format=side&path=/eth_question/eth.c">I modified my code</a> to dump the matrix in binary<br>
<br>Then I<a href="https://code.google.com/p/daneelrepo/source/browse/eth_question/mat_bin2ascii.py"> used this python script I had</a> to convert to ascii<br><br><br><a href="http://pastebin.ca/2457842">Here are the values of </a><span class=""><a href="http://pastebin.ca/2457842">AVG_BDIBJ</a>, a 9X9 matrix (the smallest possible problem size) run with the exact same input parameters with 1,2,3 and 4 procs<br>
<br></span><span class="">As you can see, the 1 and 2 procs match up, but the 3 and 4 procs do not.<br></span><br></div><div>Serious wierdness.<br></div><div><br> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div></div><div> Matt</div><div><div class="h5"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><br></div><div> Matt</div><div><div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr"><div class="gmail_quote"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div></div><div><div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
for (sitestride1 = Istart; sitestride1 < Iend; sitestride1++)<br>
{<br>
for (sitestride2 = 0; sitestride2 < matsize; sitestride2++)<br>
{<br>
for (alpha = 0; alpha < dim; alpha++)<br>
{<br>
for (mu = 0; mu < dim; mu++)<br>
for (lambda = 0; lambda < dim; lambda++)<br>
{<br>
vecval = rand () / rand ();<br>
}<br>
<br>
VecSetValue (BDB_AA, alpha, vecval, INSERT_VALUES);<br>
<br>
}<br>
VecAssemblyBegin (BDB_AA);<br>
VecAssemblyEnd (BDB_AA);<br>
VecSum (BDB_AA, &element);<br>
colvalues[sitestride2] = element;<br>
<br>
}<br>
//Insert the array of colvalues to the sitestride1^th row of H<br>
MatSetValues (AVG_BDIBJ, 1, &sitestride1, matsize, idx, colvalues,<br>
INSERT_VALUES);<br>
<div><br>
}<br>
<br>
> The code is large and complex, so I have created a smaller program<br>
> with the same<br>
</div>> loop structure here. <<a href="http://pastebin.ca/2457643" target="_blank">http://pastebin.ca/2457643</a>><br>
<div>><br>
> Compile it and run it with "mpirun -np $N ./test -draw_pause -1" gives<br>
> different results for different values of N even though it's not supposed<br>
> to.<br>
<br>
</div>What do you expect to see?<br>
<br>
> Here is a sample output <<a href="http://i42.tinypic.com/2s16ccw.jpg" target="_blank">http://i42.tinypic.com/2s16ccw.jpg</a>> for N=1,2,3,4<br>
<div>> from left to right.<br>
><br>
> Can anyone guide me as to what I'm doing wrong? Are any of the petssc<br>
> routines used not parallelizable?<br>
><br>
> Thanks in advance,<br>
><br>
> Regards.<br>
><br>
> --<br>
> ---<br>
</div>> *Analabha Roy*<br>
> C.S.I.R <<a href="http://www.csir.res.in" target="_blank">http://www.csir.res.in</a>> Senior Research<br>
> Associate<<a href="http://csirhrdg.res.in/poolsra.htm" target="_blank">http://csirhrdg.res.in/poolsra.htm</a>><br>
> Saha Institute of Nuclear Physics <<a href="http://www.saha.ac.in" target="_blank">http://www.saha.ac.in</a>><br>
<div>> Section 1, Block AF<br>
> Bidhannagar, Calcutta 700064<br>
> India<br>
</div>> *Emails*: <a href="mailto:daneel@physics.utexas.edu" target="_blank">daneel@physics.utexas.edu</a>, <a href="mailto:hariseldon99@gmail.com" target="_blank">hariseldon99@gmail.com</a><br>
> *Webpage*: <a href="http://www.ph.utexas.edu/~daneel/" target="_blank">http://www.ph.utexas.edu/~daneel/</a><span><font color="#888888"><br>
</font></span></blockquote></div></div></div><span><font color="#888888"><br><br clear="all"><div><br>-- <br><font face="tahoma, sans-serif">---</font><div><font face="tahoma, sans-serif"><b>Analabha Roy</b></font></div>
<div><font face="tahoma, sans-serif"><a href="http://www.csir.res.in" target="_blank">C.S.I.R</a> <a href="http://csirhrdg.res.in/poolsra.htm" target="_blank">Senior Research Associate</a></font></div>
</div><div><div><font face="tahoma, sans-serif"><a href="http://www.saha.ac.in" target="_blank">Saha Institute of Nuclear Physics</a></font></div></div><div><div><font face="tahoma, sans-serif">Section 1, Block AF</font></div>
<div><font face="tahoma, sans-serif">Bidhannagar, Calcutta 700064</font></div>
<div><font face="tahoma, sans-serif">India</font></div></div><div><div><b style="font-family:tahoma,sans-serif">Emails</b><span style="font-family:tahoma,sans-serif">: </span><a href="mailto:daneel@physics.utexas.edu" style="font-family:tahoma,sans-serif" target="_blank">daneel@physics.utexas.edu</a><span style="font-family:tahoma,sans-serif">, </span><a href="mailto:hariseldon99@gmail.com" style="font-family:tahoma,sans-serif" target="_blank">hariseldon99@gmail.com</a></div>
<div><font face="tahoma, sans-serif"><b>Webpage</b>: <a href="http://www.ph.utexas.edu/~daneel/" target="_blank">http://www.ph.utexas.edu/~daneel/</a></font><br></div></div></font></span></div></div></div></div>
</blockquote></div></div></div><span><font color="#888888"><br><br clear="all"><span><font color="#888888"><div><br></div>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener
</font></span></font></span></div></div><span><font color="#888888">
</font></span></blockquote></div><span><font color="#888888"><br><br clear="all"><br>-- <br><font face="tahoma, sans-serif">---</font><div><font face="tahoma, sans-serif"><b>Analabha Roy</b></font></div><div>
<font face="tahoma, sans-serif"><a href="http://www.csir.res.in" target="_blank">C.S.I.R</a> <a href="http://csirhrdg.res.in/poolsra.htm" target="_blank">Senior Research Associate</a></font></div>
<div><font face="tahoma, sans-serif"><a href="http://www.saha.ac.in" target="_blank">Saha Institute of Nuclear Physics</a></font></div><div><font face="tahoma, sans-serif">Section 1, Block AF</font></div><div><font face="tahoma, sans-serif">Bidhannagar, Calcutta 700064</font></div>
<div><font face="tahoma, sans-serif">India</font></div><div><b style="font-family:tahoma,sans-serif">Emails</b><span style="font-family:tahoma,sans-serif">: </span><a href="mailto:daneel@physics.utexas.edu" style="font-family:tahoma,sans-serif" target="_blank">daneel@physics.utexas.edu</a><span style="font-family:tahoma,sans-serif">, </span><a href="mailto:hariseldon99@gmail.com" style="font-family:tahoma,sans-serif" target="_blank">hariseldon99@gmail.com</a></div>
<div><font face="tahoma, sans-serif"><b>Webpage</b>: <a href="http://www.ph.utexas.edu/~daneel/" target="_blank">http://www.ph.utexas.edu/~daneel/</a></font></div>
</font></span></div></div>
</blockquote></div></div></div><div><div class="h5"><br><br clear="all"><div><br></div>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener
</div></div></div></div>
</blockquote></div><br><br clear="all"><br>-- <br><font face="tahoma, sans-serif">---</font><div><font face="tahoma, sans-serif"><b>Analabha Roy</b></font></div><div><font face="tahoma, sans-serif"><a href="http://www.csir.res.in" target="_blank">C.S.I.R</a> <a href="http://csirhrdg.res.in/poolsra.htm" target="_blank">Senior Research Associate</a></font></div>
<div><font face="tahoma, sans-serif"><a href="http://www.saha.ac.in" target="_blank">Saha Institute of Nuclear Physics</a></font></div><div><font face="tahoma, sans-serif">Section 1, Block AF</font></div><div><font face="tahoma, sans-serif">Bidhannagar, Calcutta 700064</font></div>
<div><font face="tahoma, sans-serif">India</font></div><div><b style="font-family:tahoma,sans-serif">Emails</b><span style="font-family:tahoma,sans-serif">: </span><a href="mailto:daneel@physics.utexas.edu" style="font-family:tahoma,sans-serif" target="_blank">daneel@physics.utexas.edu</a><span style="font-family:tahoma,sans-serif">, </span><a href="mailto:hariseldon99@gmail.com" style="font-family:tahoma,sans-serif" target="_blank">hariseldon99@gmail.com</a></div>
<div><font face="tahoma, sans-serif"><b>Webpage</b>: <a href="http://www.ph.utexas.edu/~daneel/" target="_blank">http://www.ph.utexas.edu/~daneel/</a></font></div>
</div></div>