<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 12 (filtered medium)"><base href="x-msg://77/"><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.apple-style-span
{mso-style-name:apple-style-span;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple style='word-wrap: break-word;-webkit-nbsp-mode: space;-webkit-line-break: after-white-space'><div class=WordSection1><p class=MsoNormal><span style='font-size:14.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Hi Mark,<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:14.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:14.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> petsc-dev-bounces@mcs.anl.gov [mailto:petsc-dev-bounces@mcs.anl.gov] <b>On Behalf Of </b>Mark F. Adams<br><b>Sent:</b> Wednesday, January 04, 2012 12:09 PM<br><b>To:</b> For users of the development version of PETSc<br><b>Subject:</b> Re: [petsc-dev] boomerAmg scalability<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Is your problem unsymmetric? I don't think this is the problem, it was fixed recently.<o:p></o:p></p><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>If not, can you run with -pc_gamg_verbose and -info and send me the output?<o:p></o:p></p><p class=MsoNormal><span style='font-size:14.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>I have attached the verbose+info outputs for both the serial and the parallel (2 partitions). NOTE: the serial output at some location says PC=Jacobi! Is it implicitly converting the PC to a Jacobi?<o:p></o:p></span></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>You don't need to set the block size if it is 1.<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>I have a feeling it has to do with not setting the buffer sizes correctly. There are some heuristics in there and this needs to be improved. What kind of discretization are you using? What order, how many non-zeros per row? Maybe try the smallest case that you have with more processors. I have a feeling it is trashing around mallocing stuff in MatSetValues so it just "looks" hung.<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Mark<o:p></o:p></p><p class=MsoNormal><span style='font-size:14.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:14.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Thanks,<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:14.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Ravi.<o:p></o:p></span></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><div><div><p class=MsoNormal>On Jan 4, 2012, at 12:14 PM, Ravi Kannan wrote:<o:p></o:p></p></div><p class=MsoNormal><br><br><o:p></o:p></p><div><div><p class=MsoNormal><span style='font-size:14.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Hi Mark, Matt,</span><o:p></o:p></p></div><div><p class=MsoNormal><span style='font-size:14.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p></div><div><p class=MsoNormal><span style='font-size:14.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>We recently downloaded the petsc development version, to test the gamg package.</span><o:p></o:p></p></div><div><p class=MsoNormal><span style='font-size:14.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p></div><div><p class=MsoNormal><span style='font-size:14.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>This works in serial : we tried for small cases. The parallel case (even with 2 partitions) just hangs. As of now, we have not set any parameters. So I guess that the default parameters are being used.</span><o:p></o:p></p></div><div><p class=MsoNormal><span style='font-size:14.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p></div><div><p class=MsoNormal><span style='font-size:14.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Do we need to explicitly set the block size to MatSetBlockSize(mat,1) for a PARALLEL run? Our solver solves U,V,W and P separately.</span><o:p></o:p></p></div><div><p class=MsoNormal><span style='font-size:14.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p></div><div><p class=MsoNormal><span style='font-size:14.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Any input on this would be great.</span><o:p></o:p></p></div><div><p class=MsoNormal><span style='font-size:14.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p></div><div><p class=MsoNormal><span style='font-size:14.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Thanks,</span><o:p></o:p></p></div><div><p class=MsoNormal><span style='font-size:14.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Ravi.</span><o:p></o:p></p></div><div><p class=MsoNormal><span style='font-size:14.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p></div><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in;border-width:initial;border-color:initial'><div><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span class=apple-converted-space><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> </span></span><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'><a href="mailto:petsc-dev-bounces@mcs.anl.gov">petsc-dev-bounces@mcs.anl.gov</a><span class=apple-converted-space> </span>[mailto:petsc-dev-bounces@mcs.anl.gov]<span class=apple-converted-space> </span><b>On Behalf Of<span class=apple-converted-space> </span></b>Mark F. Adams<br><b>Sent:</b><span class=apple-converted-space> </span>Thursday, December 15, 2011 1:18 PM<br><b>To:</b><span class=apple-converted-space> </span>For users of the development version of PETSc<br><b>Subject:</b><span class=apple-converted-space> </span>Re: [petsc-dev] boomerAmg scalability</span><o:p></o:p></p></div></div></div><div><p class=MsoNormal> <o:p></o:p></p></div><div><p class=MsoNormal> <o:p></o:p></p></div><div><div><div><p class=MsoNormal>On Dec 15, 2011, at 1:56 PM, Matthew Knepley wrote:<o:p></o:p></p></div></div><div><p class=MsoNormal><br><br><br><o:p></o:p></p></div><div><p class=MsoNormal>On Thu, Dec 15, 2011 at 10:23 AM, Ravi Kannan <<a href="mailto:rxk@cfdrc.com">rxk@cfdrc.com</a>> wrote:<o:p></o:p></p></div><div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt;border-width:initial;border-color:initial;z-index:auto'><div><div><div><p class=MsoNormal><span style='font-size:14.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Dear All,</span><o:p></o:p></p></div><div><p class=MsoNormal><span style='font-size:14.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p></div><div><p class=MsoNormal><span style='font-size:14.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>This is Ravi Kannan from CFD Research Corporation. Recently, we are experimenting with the BoomerAMG preconditioner for some “stiff” CFD problems. In that regard, all the other standard solver-preconditioner combinations failed for the current CFD problem. The boomer is the only one which is able to provide with “converged” solutions.</span><o:p></o:p></p></div><div><p class=MsoNormal><span style='font-size:14.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p></div><div><p class=MsoNormal><span style='font-size:14.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>We noticed that the scalability of this boomer preconditioner is really poor. For instance, even with a cell size of 2 million, we cannot scale to even 16 partitions (in contrast, the other solver-preconditioner combinations like the BI-CGS/BJacobi gave good enough scalability).</span><o:p></o:p></p></div><div><p class=MsoNormal><span style='font-size:14.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p></div><div><p class=MsoNormal><span style='font-size:14.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Are we missing something? Do we need to use a more latest version of boomer?</span><o:p></o:p></p></div></div></div></blockquote><div><div><p class=MsoNormal> <o:p></o:p></p></div></div><div><div><p class=MsoNormal>Have you tried -pc_type gamg in petsc-dev?<o:p></o:p></p></div></div></div><div><div><p class=MsoNormal> <o:p></o:p></p></div></div><div><div><p class=MsoNormal>For gamg you also want to use MPIAIJ matrices and set the block size MatSetBlockSize(mat,3), for a 3D velocity field, for instance. You can also try '-pc_gamg_type pa' or '-pc_gamg_type sa'. "pa", for plain aggregation might be better for CFD problems.<o:p></o:p></p></div></div><div><div><p class=MsoNormal> <o:p></o:p></p></div></div><div><div><p class=MsoNormal>Mark<o:p></o:p></p></div></div><div><p class=MsoNormal><br><br><br><o:p></o:p></p></div><div><div><div><p class=MsoNormal> <o:p></o:p></p></div></div><div><div><p class=MsoNormal> Matt<o:p></o:p></p></div></div><div><div><p class=MsoNormal> <span class=apple-style-span><span style='font-size:14.5pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span></span><o:p></o:p></p></div></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt;border-width:initial;border-color:initial'><div><div><div><p class=MsoNormal><span style='font-size:14.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Thanks,</span><o:p></o:p></p></div><div><p class=MsoNormal><span style='font-size:14.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Ravi.</span><o:p></o:p></p></div><div><p class=MsoNormal> <o:p></o:p></p></div></div></div></blockquote></div><div><p class=MsoNormal><br clear=all><o:p></o:p></p></div><div><div><p class=MsoNormal> <o:p></o:p></p></div></div><div><p class=MsoNormal>--<span class=apple-converted-space> </span><br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener<o:p></o:p></p></div></div><div><p class=MsoNormal> <o:p></o:p></p></div></div></div><p class=MsoNormal><o:p> </o:p></p></div></div></body></html>