<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<style>
<!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.Section1
{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Matt,<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Sorry, I thought you meant the ones with iteration counts in the
hundreds. I had a typo it was 46 actually (not 43). That's very interesting
that you think it should converge in less than 10 iterations, all I can say is
I wish I could get there. This mesh is a graded mesh and does have some poor
aspect ratio elements. For a uniform mesh I see 20 iterations or so.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Thanks,<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Harun<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'>
<p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span
style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>
petsc-users-bounces@mcs.anl.gov [mailto:petsc-users-bounces@mcs.anl.gov] <b>On
Behalf Of </b>Matthew Knepley<br>
<b>Sent:</b> Wednesday, July 29, 2009 5:23 PM<br>
<b>To:</b> PETSc users list<br>
<b>Subject:</b> Re: Smoother settings for AMG<o:p></o:p></span></p>
</div>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>On Wed, Jul 29, 2009 at 4:17 PM, BAYRAKTAR Harun <<a
href="mailto:Harun.BAYRAKTAR@3ds.com">Harun.BAYRAKTAR@3ds.com</a>> wrote:<o:p></o:p></p>
<div>
<blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;
margin-left:4.8pt;margin-right:0in'>
<div>
<div>
<p><span style='font-size:11.0pt;color:#1F497D'>Matt,</span><o:p></o:p></p>
<p><span style='font-size:11.0pt;color:#1F497D'>It is from the pressure poisson
equation for incompressible navier-stokes so it is elliptic. Also on 1 cpu, I
am able to solve it with reason able iteration count (i.e., 43 to 1.e-5 true
res norm rel tolerance). It is the parallel runs that really concern me.</span><o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class=MsoNormal>Actually, it was the 43 that really concerned me. In my
experience, an MG that is doing what it is supposed<br>
to on Poisson takes < 10 iterations. However, if your grid is pretty
distorted, maybe it can get this bad.<br>
<br>
Matt<o:p></o:p></p>
</div>
<blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;
margin-left:4.8pt;margin-right:0in'>
<div>
<div>
<p><o:p> </o:p></p>
<p><span style='font-size:11.0pt;color:#1F497D'>Thanks,</span><o:p></o:p></p>
<p><span style='font-size:11.0pt;color:#1F497D'>Harun</span><o:p></o:p></p>
<p><span style='font-size:11.0pt;color:#1F497D'> </span><o:p></o:p></p>
<p><span style='font-size:11.0pt;color:#1F497D'> </span><o:p></o:p></p>
<div style='border:none;border-top:solid windowtext 1.0pt;padding:3.0pt 0in 0in 0in;
border-color:-moz-use-text-color -moz-use-text-color'>
<p><b><span style='font-size:10.0pt'>From:</span></b><span style='font-size:
10.0pt'> <a href="mailto:petsc-users-bounces@mcs.anl.gov" target="_blank">petsc-users-bounces@mcs.anl.gov</a>
[mailto:<a href="mailto:petsc-users-bounces@mcs.anl.gov" target="_blank">petsc-users-bounces@mcs.anl.gov</a>]
<b>On Behalf Of </b>Matthew Knepley<br>
<b>Sent:</b> Wednesday, July 29, 2009 5:00 PM<br>
<b>To:</b> PETSc users list<br>
<b>Subject:</b> Re: Smoother settings for AMG</span><o:p></o:p></p>
</div>
<p> <o:p></o:p></p>
<p>On Wed, Jul 29, 2009 at 3:54 PM, BAYRAKTAR Harun <<a
href="mailto:Harun.BAYRAKTAR@3ds.com" target="_blank">Harun.BAYRAKTAR@3ds.com</a>>
wrote:<o:p></o:p></p>
<div>
<blockquote style='border:none;border-left:solid windowtext 1.0pt;padding:0in 0in 0in 6.0pt;
margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt;
border-color:-moz-use-text-color -moz-use-text-color -moz-use-text-color rgb(204, 204, 204)'>
<p>Hi,<br>
<br>
I am trying to solve a system of equations and I am having difficulty<br>
picking the right smoothers for AMG (using ML as pc_type) in PETSc for<br>
parallel execution. First here is what happens in terms of CG (ksp_type)<br>
iteration counts (both columns use block jacobi):<o:p></o:p></p>
</blockquote>
<div>
<p><br>
Are you sure you have an elliptic system? These iteration counts are extremely<br>
high.<br>
<br>
Matt<br>
<o:p></o:p></p>
</div>
<blockquote style='border:none;border-left:solid windowtext 1.0pt;padding:0in 0in 0in 6.0pt;
margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt;
border-color:-moz-use-text-color -moz-use-text-color -moz-use-text-color rgb(204, 204, 204)'>
<p style='margin-bottom:12.0pt'><br>
cpus | AMG w/ ICC(0) x1
| AMG w/ SOR x4<br>
------------------------------------------------------<br>
1 | 43
|
243<br>
4 | 699
|
379<br>
<br>
x1 or x4 means 1 or 4 iterations of smoother application at each AMG<br>
level (all details from ksp view for the 4 cpu run are below). The main<br>
observation is that on 1 cpu, AMG w/ ICC(0) is a clear winner but falls<br>
apart in parallel. SOR on the other hand experiences a 1.5X increase in<br>
iteration count which is totally expected from the quality of coarsening<br>
ML delivers in parallel.<br>
<br>
I basically would like to find a way (if possible) to have the number of<br>
iterations in parallel stay with 1-2X of 1 cpu iteration count for the<br>
AMG w/ ICC case. Is there a way to achieve this?<br>
<br>
Thanks,<br>
Harun<br>
<br>
%%%%%%%%%%%%%%%%%%%%%%%%%<br>
AMG w/ ICC(0) x1 ksp_view<br>
%%%%%%%%%%%%%%%%%%%%%%%%%<br>
KSP Object:<br>
type: cg<br>
maximum iterations=10000<br>
tolerances: relative=1e-05, absolute=1e-50, divergence=10000<br>
left preconditioning<br>
PC Object:<br>
type: ml<br>
MG: type is MULTIPLICATIVE, levels=3 cycles=v, pre-smooths=1,<br>
post-smooths=1<br>
Coarse gride solver -- level 0 -------------------------------<br>
KSP Object:(mg_coarse_)<br>
type: preonly<br>
maximum iterations=1, initial guess is zero<br>
tolerances: relative=1e-05, absolute=1e-50,
divergence=10000<br>
left preconditioning<br>
PC Object:(mg_coarse_)<br>
type: redundant<br>
Redundant preconditioner: First (color=0) of 4 PCs
follows<br>
KSP Object:(mg_coarse_redundant_)<br>
type: preonly<br>
maximum iterations=10000, initial guess is zero<br>
tolerances: relative=1e-05, absolute=1e-50,
divergence=10000<br>
left preconditioning<br>
PC Object:(mg_coarse_redundant_)<br>
type: lu<br>
LU: out-of-place factorization<br>
matrix ordering: nd<br>
LU: tolerance for zero pivot 1e-12<br>
LU: factor fill ratio needed 2.17227<br>
Factored matrix follows<br>
Matrix Object:<br>
type=seqaij, rows=283,
cols=283<br>
total: nonzeros=21651,
allocated nonzeros=21651<br>
using I-node
routines: found 186 nodes, limit used is<br>
5<br>
linear system matrix = precond matrix:<br>
Matrix Object:<br>
type=seqaij, rows=283, cols=283<br>
total: nonzeros=9967, allocated
nonzeros=14150<br>
not using I-node routines<br>
linear system matrix = precond matrix:<br>
Matrix Object:<br>
type=mpiaij, rows=283, cols=283<br>
total: nonzeros=9967, allocated nonzeros=9967<br>
not using I-node (on process 0) routines<br>
Down solver (pre-smoother) on level 1 -------------------------------<br>
KSP Object:(mg_levels_1_)<br>
type: richardson<br>
Richardson: damping factor=0.9<br>
maximum iterations=1, initial guess is zero<br>
tolerances: relative=1e-05, absolute=1e-50,
divergence=10000<br>
left preconditioning<br>
PC Object:(mg_levels_1_)<br>
type: bjacobi<br>
block Jacobi: number of blocks = 4<br>
Local solve is same for all blocks, in the following
KSP and PC<br>
objects:<br>
KSP Object:(mg_levels_1_sub_)<br>
type: preonly<br>
maximum iterations=10000, initial guess is zero<br>
tolerances: relative=1e-05, absolute=1e-50,
divergence=10000<br>
left preconditioning<br>
PC Object:(mg_levels_1_sub_)<br>
type: icc<br>
ICC: 0 levels of fill<br>
ICC: factor fill ratio allocated 1<br>
ICC: using Manteuffel shift<br>
ICC: factor fill ratio needed 0.514899<br>
Factored matrix follows<br>
Matrix Object:<br>
type=seqsbaij,
rows=2813, cols=2813<br>
total: nonzeros=48609,
allocated nonzeros=48609<br>
block size
is 1<br>
linear system matrix = precond matrix:<br>
Matrix Object:<br>
type=seqaij, rows=2813, cols=2813<br>
total: nonzeros=94405, allocated
nonzeros=94405<br>
not using I-node routines<br>
linear system matrix = precond matrix:<br>
Matrix Object:<br>
type=mpiaij, rows=10654, cols=10654<br>
total: nonzeros=376634, allocated nonzeros=376634<br>
not using I-node (on process 0) routines<br>
Up solver (post-smoother) on level 1 -------------------------------<br>
KSP Object:(mg_levels_1_)<br>
type: richardson<br>
Richardson: damping factor=0.9<br>
maximum iterations=1<br>
tolerances: relative=1e-05, absolute=1e-50,
divergence=10000<br>
left preconditioning<br>
PC Object:(mg_levels_1_)<br>
type: bjacobi<br>
block Jacobi: number of blocks = 4<br>
Local solve is same for all blocks, in the following
KSP and PC<br>
objects:<br>
KSP Object:(mg_levels_1_sub_)<br>
type: preonly<br>
maximum iterations=10000, initial guess is zero<br>
tolerances: relative=1e-05, absolute=1e-50,
divergence=10000<br>
left preconditioning<br>
PC Object:(mg_levels_1_sub_)<br>
type: icc<br>
ICC: 0 levels of fill<br>
ICC: factor fill ratio allocated 1<br>
ICC: using Manteuffel shift<br>
ICC: factor fill ratio needed 0.514899<br>
Factored matrix follows<br>
Matrix Object:<br>
type=seqsbaij,
rows=2813, cols=2813<br>
total: nonzeros=48609,
allocated nonzeros=48609<br>
block size
is 1<br>
linear system matrix = precond matrix:<br>
Matrix Object:<br>
type=seqaij, rows=2813, cols=2813<br>
total: nonzeros=94405, allocated
nonzeros=94405<br>
not using I-node routines<br>
linear system matrix = precond matrix:<br>
Matrix Object:<br>
type=mpiaij, rows=10654, cols=10654<br>
total: nonzeros=376634, allocated nonzeros=376634<br>
not using I-node (on process 0) routines<br>
Down solver (pre-smoother) on level 2 -------------------------------<br>
KSP Object:(mg_levels_2_)<br>
type: richardson<br>
Richardson: damping factor=0.9<br>
maximum iterations=1, initial guess is zero<br>
tolerances: relative=1e-05, absolute=1e-50,
divergence=10000<br>
left preconditioning<br>
PC Object:(mg_levels_2_)<br>
type: bjacobi<br>
block Jacobi: number of blocks = 4<br>
Local solve is same for all blocks, in the following
KSP and PC<br>
objects:<br>
KSP Object:(mg_levels_2_sub_)<br>
type: preonly<br>
maximum iterations=10000, initial guess is zero<br>
tolerances: relative=1e-05, absolute=1e-50,
divergence=10000<br>
left preconditioning<br>
PC Object:(mg_levels_2_sub_)<br>
type: icc<br>
ICC: 0 levels of fill<br>
ICC: factor fill ratio allocated 1<br>
ICC: using Manteuffel shift<br>
ICC: factor fill ratio needed 0.519045<br>
Factored matrix follows<br>
Matrix Object:<br>
type=seqsbaij,
rows=101164, cols=101164<br>
total: nonzeros=1378558,
allocated nonzeros=1378558<br>
block size
is 1<br>
linear system matrix = precond matrix:<br>
Matrix Object:<br>
type=seqaij, rows=101164, cols=101164<br>
total: nonzeros=2655952, allocated
nonzeros=5159364<br>
not using I-node routines<br>
linear system matrix = precond matrix:<br>
Matrix Object:<br>
type=mpiaij, rows=411866, cols=411866<br>
total: nonzeros=10941434, allocated
nonzeros=42010332<br>
not using I-node (on process 0) routines<br>
Up solver (post-smoother) on level 2 -------------------------------<br>
KSP Object:(mg_levels_2_)<br>
type: richardson<br>
Richardson: damping factor=0.9<br>
maximum iterations=1<br>
tolerances: relative=1e-05, absolute=1e-50,
divergence=10000<br>
left preconditioning<br>
PC Object:(mg_levels_2_)<br>
type: bjacobi<br>
block Jacobi: number of blocks = 4<br>
Local solve is same for all blocks, in the following
KSP and PC<br>
objects:<br>
KSP Object:(mg_levels_2_sub_)<br>
type: preonly<br>
maximum iterations=10000, initial guess is zero<br>
tolerances: relative=1e-05, absolute=1e-50,
divergence=10000<br>
left preconditioning<br>
PC Object:(mg_levels_2_sub_)<br>
type: icc<br>
ICC: 0 levels of fill<br>
ICC: factor fill ratio allocated 1<br>
ICC: using Manteuffel shift<br>
ICC: factor fill ratio needed 0.519045<br>
Factored matrix follows<br>
Matrix Object:<br>
type=seqsbaij,
rows=101164, cols=101164<br>
total: nonzeros=1378558,
allocated nonzeros=1378558<br>
block size
is 1<br>
linear system matrix = precond matrix:<br>
Matrix Object:<br>
type=seqaij, rows=101164, cols=101164<br>
total: nonzeros=2655952, allocated
nonzeros=5159364<br>
not using I-node routines<br>
linear system matrix = precond matrix:<br>
Matrix Object:<br>
type=mpiaij, rows=411866, cols=411866<br>
total: nonzeros=10941434, allocated
nonzeros=42010332<br>
not using I-node (on process 0) routines<br>
linear system matrix = precond matrix:<br>
Matrix Object:<br>
type=mpiaij, rows=411866, cols=411866<br>
total: nonzeros=10941434, allocated nonzeros=42010332<br>
not using I-node (on process 0) routines<br>
<br>
%%%%%%%%%%%%%%%%%%%%%%<br>
AMG w/ SOR x4 ksp_view<br>
%%%%%%%%%%%%%%%%%%%%%%<br>
<br>
KSP Object:<br>
type: cg<br>
maximum iterations=10000<br>
tolerances: relative=1e-05, absolute=1e-50, divergence=10000<br>
left preconditioning<br>
PC Object:<br>
type: ml<br>
MG: type is MULTIPLICATIVE, levels=3 cycles=v, pre-smooths=1,<br>
post-smooths=1<br>
Coarse gride solver -- level 0 -------------------------------<br>
KSP Object:(mg_coarse_)<br>
type: preonly<br>
maximum iterations=1, initial guess is zero<br>
tolerances: relative=1e-05, absolute=1e-50,
divergence=10000<br>
left preconditioning<br>
PC Object:(mg_coarse_)<br>
type: redundant<br>
Redundant preconditioner: First (color=0) of 4 PCs
follows<br>
KSP Object:(mg_coarse_redundant_)<br>
type: preonly<br>
maximum iterations=10000, initial guess is zero<br>
tolerances: relative=1e-05, absolute=1e-50,
divergence=10000<br>
left preconditioning<br>
PC Object:(mg_coarse_redundant_)<br>
type: lu<br>
LU: out-of-place factorization<br>
matrix ordering: nd<br>
LU: tolerance for zero pivot 1e-12<br>
LU: factor fill ratio needed 2.17227<br>
Factored matrix follows<br>
Matrix Object:<br>
type=seqaij, rows=283,
cols=283<br>
total: nonzeros=21651,
allocated nonzeros=21651<br>
using I-node
routines: found 186 nodes, limit used is<br>
5<br>
linear system matrix = precond matrix:<br>
Matrix Object:<br>
type=seqaij, rows=283, cols=283<br>
total: nonzeros=9967, allocated
nonzeros=14150<br>
not using I-node routines<br>
linear system matrix = precond matrix:<br>
Matrix Object:<br>
type=mpiaij, rows=283, cols=283<br>
total: nonzeros=9967, allocated nonzeros=9967<br>
not using I-node (on process 0) routines<br>
Down solver (pre-smoother) on level 1 -------------------------------<br>
KSP Object:(mg_levels_1_)<br>
type: richardson<br>
Richardson: damping factor=1<br>
maximum iterations=4, initial guess is zero<br>
tolerances: relative=1e-05, absolute=1e-50,
divergence=10000<br>
left preconditioning<br>
PC Object:(mg_levels_1_)<br>
type: sor<br>
SOR: type = local_symmetric, iterations = 1, omega =
1<br>
linear system matrix = precond matrix:<br>
Matrix Object:<br>
type=mpiaij, rows=10654, cols=10654<br>
total: nonzeros=376634, allocated nonzeros=376634<br>
not using I-node (on process 0) routines<br>
Up solver (post-smoother) on level 1 -------------------------------<br>
KSP Object:(mg_levels_1_)<br>
type: richardson<br>
Richardson: damping factor=1<br>
maximum iterations=4<br>
tolerances: relative=1e-05, absolute=1e-50,
divergence=10000<br>
left preconditioning<br>
PC Object:(mg_levels_1_)<br>
type: sor<br>
SOR: type = local_symmetric, iterations = 1, omega =
1<br>
linear system matrix = precond matrix:<br>
Matrix Object:<br>
type=mpiaij, rows=10654, cols=10654<br>
total: nonzeros=376634, allocated nonzeros=376634<br>
not using I-node (on process 0) routines<br>
Down solver (pre-smoother) on level 2 -------------------------------<br>
KSP Object:(mg_levels_2_)<br>
type: richardson<br>
Richardson: damping factor=1<br>
maximum iterations=4, initial guess is zero<br>
tolerances: relative=1e-05, absolute=1e-50,
divergence=10000<br>
left preconditioning<br>
PC Object:(mg_levels_2_)<br>
type: sor<br>
SOR: type = local_symmetric, iterations = 1, omega =
1<br>
linear system matrix = precond matrix:<br>
Matrix Object:<br>
type=mpiaij, rows=411866, cols=411866<br>
total: nonzeros=10941434, allocated
nonzeros=42010332<br>
not using I-node (on process 0) routines<br>
Up solver (post-smoother) on level 2 -------------------------------<br>
KSP Object:(mg_levels_2_)<br>
type: richardson<br>
Richardson: damping factor=1<br>
maximum iterations=4<br>
tolerances: relative=1e-05, absolute=1e-50,
divergence=10000<br>
left preconditioning<br>
PC Object:(mg_levels_2_)<br>
type: sor<br>
SOR: type = local_symmetric, iterations = 1, omega =
1<br>
linear system matrix = precond matrix:<br>
Matrix Object:<br>
type=mpiaij, rows=411866, cols=411866<br>
total: nonzeros=10941434, allocated
nonzeros=42010332<br>
not using I-node (on process 0) routines<br>
linear system matrix = precond matrix:<br>
Matrix Object:<br>
type=mpiaij, rows=411866, cols=411866<br>
total: nonzeros=10941434, allocated nonzeros=42010332<br>
not using I-node (on process 0) routines<o:p></o:p></p>
</blockquote>
</div>
<p><br>
<br clear=all>
<br>
-- <br>
What most experimenters take for granted before they begin their experiments is
infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
<p class=MsoNormal><br>
<br clear=all>
<br>
-- <br>
What most experimenters take for granted before they begin their experiments is
infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<o:p></o:p></p>
</div>
</body>
</html>