Ravi,<div><br></div><div>I need more info for debugging. Can you provide a simple stand alone code and matrices in petsc</div><div>binary format that reproduce the error?</div><div><br></div><div>MatTransposeMatMult() for mpiaij is a newly developed subroutine - less than one month old </div>
<div>and not well tested yet :-(</div><div>I used petsc-dev/src/mat/examples/tests/ex94.c for testing.</div><div><br></div><div>Thanks,</div><div><br></div><div>Hong<br><br><div class="gmail_quote">On Thu, Jan 12, 2012 at 9:17 PM, Mark F. Adams <span dir="ltr"><<a href="mailto:mark.adams@columbia.edu">mark.adams@columbia.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">It looks like the problem is in MatTransposeMatMult and Hong (cc'ed) is working on it.<div>
<br></div><div>I'm hoping that your output will be enough for Hong to figure this out but I could not reproduce this problem with any of my tests.</div><div><br></div><div>If Hong can not figure this out then we will need to get the matrix from you to reproduce this.</div>
<span class="HOEnZb"><font color="#888888"><div><br></div></font></span><div><span class="HOEnZb"><font color="#888888">Mark</font></span><div><div class="h5"><br><div><br></div><div><div><div>On Jan 12, 2012, at 6:25 PM, Ravi Kannan wrote:</div>
<br><blockquote type="cite"><span style="border-collapse:separate;font-family:Helvetica;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:medium"><div lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:14pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Hi Mark,<u></u><u></u></span></div>
<div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:14pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></div>
<div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:14pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Any luck with the gamg bug fix?<u></u><u></u></span></div>
<div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:14pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></div>
<div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:14pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Thanks,<u></u><u></u></span></div>
<div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:14pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Ravi.<u></u><u></u></span></div>
<div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:14pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></div>
<div><div style="border-right-style:none;border-bottom-style:none;border-left-style:none;border-width:initial;border-color:initial;border-top-style:solid;border-top-color:rgb(181,196,223);border-top-width:1pt;padding-top:3pt;padding-right:0in;padding-bottom:0in;padding-left:0in">
<div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><b><span style="font-size:10pt;font-family:Tahoma,sans-serif">From:</span></b><span style="font-size:10pt;font-family:Tahoma,sans-serif"><span> </span><a href="mailto:petsc-dev-bounces@mcs.anl.gov" style="color:blue;text-decoration:underline" target="_blank">petsc-dev-bounces@mcs.anl.gov</a><span> </span>[mailto:<a href="mailto:petsc-dev-bounces@mcs.anl.gov" target="_blank">petsc-dev-bounces@mcs.anl.gov</a>]<span> </span><b>On Behalf Of<span> </span></b>Mark F. Adams<br>
<b>Sent:</b><span> </span>Wednesday, January 11, 2012 1:54 PM<br><b>To:</b><span> </span>For users of the development version of PETSc<br><b>Subject:</b><span> </span>Re: [petsc-dev] boomerAmg scalability<u></u><u></u></span></div>
</div></div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><u></u> <u></u></div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif">
This seems to be dying earlier than it was last week, so it looks like a new bug in MatTransposeMatMult.<u></u><u></u></div><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif">
<u></u> <u></u></div></div><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif">Mark<u></u><u></u></div><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif">
<u></u> <u></u></div><div><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif">On Jan 11, 2012, at 1:59 PM, Matthew Knepley wrote:<u></u><u></u></div>
</div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><br><br><u></u><u></u></div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif">
On Wed, Jan 11, 2012 at 12:23 PM, Ravi Kannan <<a href="mailto:rxk@cfdrc.com" style="color:blue;text-decoration:underline" target="_blank">rxk@cfdrc.com</a>> wrote:<u></u><u></u></div><div><blockquote style="border-top-style:none;border-right-style:none;border-bottom-style:none;border-width:initial;border-color:initial;border-left-style:solid;border-left-color:rgb(204,204,204);border-left-width:1pt;padding-top:0in;padding-right:0in;padding-bottom:0in;padding-left:6pt;margin-left:4.8pt;margin-right:0in">
<div><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:14pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Hi Mark,</span><u></u><u></u></div>
<div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:14pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span><u></u><u></u></div>
<div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:14pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">I downloaded the dev version again. This time, the program crashes even earlier. Attached is the serial and parallel info outputs.</span><u></u><u></u></div>
<div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:14pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span><u></u><u></u></div>
<div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:14pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Could you kindly take a look.</span><u></u><u></u></div>
</div></div></blockquote><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><u></u> <u></u></div></div><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif">
It looks like this is a problem with MatMatMult(). Can you try to reproduce this using KSP ex10? You put<u></u><u></u></div></div><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif">
your matrix in binary format and use -pc_type gamg. Then you can send us the matrix and we can track<u></u><u></u></div></div><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif">
it down. Or are you running an example there?<u></u><u></u></div></div><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><u></u> <u></u></div>
</div><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif">  Thanks,<u></u><u></u></div></div><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif">
<u></u> <u></u></div></div><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif">    Matt<u></u><u></u></div></div><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif">
<u></u> <u></u></div></div><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"> <u></u><u></u></div></div><blockquote style="border-top-style:none;border-right-style:none;border-bottom-style:none;border-width:initial;border-color:initial;border-left-style:solid;border-left-color:rgb(204,204,204);border-left-width:1pt;padding-top:0in;padding-right:0in;padding-bottom:0in;padding-left:6pt;margin-left:4.8pt;margin-right:0in">
<div><div><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:14pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span><u></u><u></u></div>
</div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:14pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Thanks,</span><u></u><u></u></div>
<div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:14pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Ravi.</span><u></u><u></u></div>
<div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:14pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span><u></u><u></u></div>
<div><div style="border-right-style:none;border-bottom-style:none;border-left-style:none;border-width:initial;border-color:initial;border-top-style:solid;border-top-color:rgb(181,196,223);border-top-width:1pt;padding-top:3pt;padding-right:0in;padding-bottom:0in;padding-left:0in">
<div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><b><span style="font-size:10pt;font-family:Tahoma,sans-serif">From:</span></b><span style="font-size:10pt;font-family:Tahoma,sans-serif"><span> </span><a href="mailto:petsc-dev-bounces@mcs.anl.gov" style="color:blue;text-decoration:underline" target="_blank">petsc-dev-bounces@mcs.anl.gov</a><span> </span>[mailto:<a href="mailto:petsc-dev-bounces@mcs.anl.gov" style="color:blue;text-decoration:underline" target="_blank">petsc-dev-bounces@mcs.anl.gov</a>]<span> </span><b>On Behalf Of<span> </span></b>Mark F. Adams<br>
<b>Sent:</b><span> </span>Monday, January 09, 2012 3:08 PM</span><u></u><u></u></div><div><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif">
<br><b>To:</b><span> </span>For users of the development version of PETSc<br><b>Subject:</b><span> </span>Re: [petsc-dev] boomerAmg scalability<u></u><u></u></div></div></div><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif">
<u></u> <u></u></div></div></div></div><div><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"> <u></u><u></u></div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif">
Yes its all checked it, just pull from dev.<u></u><u></u></div><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif">Mark<u></u><u></u></div>
</div><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"> <u></u><u></u></div><div><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif">
On Jan 9, 2012, at 2:54 PM, Ravi Kannan wrote:<u></u><u></u></div></div><p class="MsoNormal" style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:12pt;font-size:12pt;font-family:'Times New Roman',serif">
<u></u> <u></u></p><div><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:14pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Hi Mark,</span><u></u><u></u></div>
</div><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:14pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span><u></u><u></u></div>
</div><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:14pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Thanks for your efforts.</span><u></u><u></u></div>
</div><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:14pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span><u></u><u></u></div>
</div><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:14pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Do I need to do the install from scratch once again? Or some particular files (check out gamg.c for instance)?</span><u></u><u></u></div>
</div><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:14pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span><u></u><u></u></div>
</div><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:14pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Thanks,</span><u></u><u></u></div>
</div><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:14pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Ravi.</span><u></u><u></u></div>
</div><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:14pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span><u></u><u></u></div>
</div><div><div style="border-right-style:none;border-bottom-style:none;border-left-style:none;border-width:initial;border-color:initial;border-top-style:solid;padding-top:3pt;padding-right:0in;padding-bottom:0in;padding-left:0in;border-width:initial;border-color:initial">
<div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><b><span style="font-size:10pt;font-family:Tahoma,sans-serif">From:</span></b><span style="font-size:10pt;font-family:Tahoma,sans-serif"> <a href="mailto:petsc-dev-bounces@mcs.anl.gov" style="color:blue;text-decoration:underline" target="_blank">petsc-dev-bounces@mcs.anl.gov</a> [mailto:<a href="mailto:petsc-dev-bounces@mcs.anl.gov" style="color:blue;text-decoration:underline" target="_blank">petsc-dev-bounces@mcs.anl.gov</a>] <b>On Behalf Of </b>Mark F. Adams<br>
<b>Sent:</b> Friday, January 06, 2012 10:30 AM<br><b>To:</b> For users of the development version of PETSc<br><b>Subject:</b> Re: [petsc-dev] boomerAmg scalability</span><u></u><u></u></div></div></div></div><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif">
 <u></u><u></u></div></div><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif">I think I found the problem.  You will need to use petsc-dev to get the fix.<u></u><u></u></div>
</div><div><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"> <u></u><u></u></div></div></div><div><div><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif">
Mark<u></u><u></u></div></div></div><div><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"> <u></u><u></u></div></div><div>
<div><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif">On Jan 6, 2012, at 8:55 AM, Mark F. Adams wrote:<u></u><u></u></div>
</div></div><div><p class="MsoNormal" style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:12pt;font-size:12pt;font-family:'Times New Roman',serif"><br><br><u></u><u></u></p></div><div><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif">
Ravi, I forgot but you can just use -ksp_view_binary to output the matrix data (two files).  You could run it with two procs and a Jacobi solver to get it past the solve, where it writes the matrix (I believe).<u></u><u></u></div>
</div><div><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif">Mark<u></u><u></u></div></div></div><div><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif">
 <u></u><u></u></div></div><div><div><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif">On Jan 5, 2012, at 6:19 PM, Ravi Kannan wrote:<u></u><u></u></div>
</div></div><div><p class="MsoNormal" style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:12pt;font-size:12pt;font-family:'Times New Roman',serif"><br><br><u></u><u></u></p></div><div><div><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif">
<span style="font-size:14pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Just send in another email with the attachment.</span><u></u><u></u></div></div></div><div><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif">
<span style="font-size:14pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span><u></u><u></u></div></div></div><div style="border-right-style:none;border-bottom-style:none;border-left-style:none;border-width:initial;border-color:initial;border-top-style:solid;padding-top:3pt;padding-right:0in;padding-bottom:0in;padding-left:0in;border-width:initial;border-color:initial;border-width:initial;border-color:initial">
<div><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><b><span style="font-size:10pt;font-family:Tahoma,sans-serif">From:</span></b><span style="font-size:10pt;font-family:Tahoma,sans-serif"> <a href="mailto:petsc-dev-bounces@mcs.anl.gov" style="color:blue;text-decoration:underline" target="_blank">petsc-dev-bounces@mcs.anl.gov</a> [mailto:<a href="mailto:petsc-dev-bounces@mcs.anl.gov" style="color:blue;text-decoration:underline" target="_blank">petsc-dev-bounces@mcs.anl.gov</a>] <b>On Behalf Of </b>Jed Brown<br>
<b>Sent:</b> Thursday, January 05, 2012 5:15 PM<br><b>To:</b> For users of the development version of PETSc<br><b>Subject:</b> Re: [petsc-dev] boomerAmg scalability</span><u></u><u></u></div></div></div></div><div><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif">
 <u></u><u></u></div></div></div><div><div><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif">On Thu, Jan 5, 2012 at 17:12, Ravi Kannan <<a href="mailto:rxk@cfdrc.com" style="color:blue;text-decoration:underline" target="_blank">rxk@cfdrc.com</a>> wrote:<u></u><u></u></div>
</div></div><div><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif">I have attached the verbose+info outputs for both the serial and the parallel (2 partitions). NOTE: the serial output at some location says PC=Jacobi! Is it implicitly converting the PC to a Jacobi?<u></u><u></u></div>
</div></div></div><div><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"> <u></u><u></u></div></div></div><div><div><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif">
Looks like you forgot the attachment.<u></u><u></u></div></div></div></div></div></div><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif">
 <u></u><u></u></div></div></div></div></div><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"> <u></u><u></u></div></div></div>
</div></div></div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"> <u></u><u></u></div></div></div></div></div></div></blockquote>
</div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><br><br clear="all"><u></u><u></u></div><div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif">
<u></u> <u></u></div></div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif">--<span> </span><br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<u></u><u></u></div></div><div style="margin-top:0in;margin-right:0in;margin-left:0in;margin-bottom:0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><u></u> <u></u></div></div></div></div>
</div></span></blockquote></div><br></div></div></div></div></div></blockquote></div><br></div>