<div dir="ltr">So with -pc_gamg_square_graph 10 I get the following:<div><br></div><div><p style="margin:0px;font-size:12px;line-height:normal;font-family:Courier;color:rgb(76,47,45);background-color:rgb(223,219,196)">[0] PCSetUp_<span style="color:rgb(177,37,18)"><b>GAMG</b></span>(): level 0) N=48000, n data rows=1, n data cols=1, nnz/row (ave)=9, np=1</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Courier;color:rgb(76,47,45);background-color:rgb(223,219,196)">[0] PC<span style="color:rgb(177,37,18)"><b>GAMG</b></span>FilterGraph(): <span class="" style="white-space:pre"> </span> 55.7114% nnz after filtering, with threshold 0., 8.79533 nnz ave. (N=48000)</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Courier;color:rgb(76,47,45);background-color:rgb(223,219,196)">[0] PC<span style="color:rgb(177,37,18)"><b>GAMG</b></span>Coarsen_AGG(): Square Graph on level 1 of 1 to square</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Courier;color:rgb(76,47,45);background-color:rgb(223,219,196)">[0] PC<span style="color:rgb(177,37,18)"><b>GAMG</b></span>Prolongator_AGG(): New grid 6672 nodes</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Courier;color:rgb(76,47,45);background-color:rgb(223,219,196)">[0] PC<span style="color:rgb(177,37,18)"><b>GAMG</b></span>OptProlongator_AGG(): Smooth P0: max eigen=1.954700e+00 min=1.040410e-02 PC=jacobi</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Courier;color:rgb(76,47,45);background-color:rgb(223,219,196)">[0] PCSetUp_<span style="color:rgb(177,37,18)"><b>GAMG</b></span>(): 1) N=6672, n data cols=1, nnz/row (ave)=623, 1 active pes</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Courier;color:rgb(76,47,45);background-color:rgb(223,219,196)">[0] PC<span style="color:rgb(177,37,18)"><b>GAMG</b></span>FilterGraph(): <span class="" style="white-space:pre"> </span> 3.40099% nnz after filtering, with threshold 0., 623.135 nnz ave. (N=6672)</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Courier;color:rgb(76,47,45);background-color:rgb(223,219,196)">[0] PC<span style="color:rgb(177,37,18)"><b>GAMG</b></span>Prolongator_AGG(): New grid 724 nodes</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Courier;color:rgb(76,47,45);background-color:rgb(223,219,196)">[0] PC<span style="color:rgb(177,37,18)"><b>GAMG</b></span>OptProlongator_AGG(): Smooth P0: max eigen=1.313339e+00 min=2.474586e-02 PC=jacobi</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Courier;color:rgb(76,47,45);background-color:rgb(223,219,196)">[0] PCSetUp_<span style="color:rgb(177,37,18)"><b>GAMG</b></span>(): 2) N=724, n data cols=1, nnz/row (ave)=724, 1 active pes</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Courier;color:rgb(76,47,45);background-color:rgb(223,219,196)">[0] PC<span style="color:rgb(177,37,18)"><b>GAMG</b></span>FilterGraph(): <span class="" style="white-space:pre"> </span> 9.82914% nnz after filtering, with threshold 0., 724. nnz ave. (N=724)</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Courier;color:rgb(76,47,45);background-color:rgb(223,219,196)">[0] PC<span style="color:rgb(177,37,18)"><b>GAMG</b></span>Prolongator_AGG(): New grid 37 nodes</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Courier;color:rgb(76,47,45);background-color:rgb(223,219,196)">[0] PC<span style="color:rgb(177,37,18)"><b>GAMG</b></span>OptProlongator_AGG(): Smooth P0: max eigen=2.011784e+00 min=2.759552e-01 PC=jacobi</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Courier;color:rgb(76,47,45);background-color:rgb(223,219,196)">[0] PCSetUp_<span style="color:rgb(177,37,18)"><b>GAMG</b></span>(): 3) N=37, n data cols=1, nnz/row (ave)=37, 1 active pes</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Courier;color:rgb(76,47,45);background-color:rgb(223,219,196)">[0] PCSetUp_<span style="color:rgb(177,37,18)"><b>GAMG</b></span>(): 4 levels, grid complexity = 12.0928</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Courier;color:rgb(76,47,45);background-color:rgb(223,219,196)">[0] PCSetUp_<span style="color:rgb(177,37,18)"><b>GAMG</b></span>(): level 0) N=48000, n data rows=1, n data cols=1, nnz/row (ave)=9, np=1</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Courier;color:rgb(76,47,45);background-color:rgb(223,219,196)">[0] PC<span style="color:rgb(177,37,18)"><b>GAMG</b></span>FilterGraph(): <span class="" style="white-space:pre"> </span> 55.7114% nnz after filtering, with threshold 0., 8.79533 nnz ave. (N=48000)</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Courier;color:rgb(76,47,45);background-color:rgb(223,219,196)">[0] PC<span style="color:rgb(177,37,18)"><b>GAMG</b></span>Coarsen_AGG(): Square Graph on level 1 of 1 to square</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Courier;color:rgb(76,47,45);background-color:rgb(223,219,196)">[0] PC<span style="color:rgb(177,37,18)"><b>GAMG</b></span>Prolongator_AGG(): New grid 6672 nodes</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Courier;color:rgb(76,47,45);background-color:rgb(223,219,196)">[0] PC<span style="color:rgb(177,37,18)"><b>GAMG</b></span>OptProlongator_AGG(): Smooth P0: max eigen=1.954700e+00 min=1.040410e-02 PC=jacobi</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Courier;color:rgb(76,47,45);background-color:rgb(223,219,196)">[0] PCSetUp_<span style="color:rgb(177,37,18)"><b>GAMG</b></span>(): 1) N=6672, n data cols=1, nnz/row (ave)=623, 1 active pes</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Courier;color:rgb(76,47,45);background-color:rgb(223,219,196)">[0] PC<span style="color:rgb(177,37,18)"><b>GAMG</b></span>FilterGraph(): <span class="" style="white-space:pre"> </span> 3.40099% nnz after filtering, with threshold 0., 623.135 nnz ave. (N=6672)</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Courier;color:rgb(76,47,45);background-color:rgb(223,219,196)">[0] PC<span style="color:rgb(177,37,18)"><b>GAMG</b></span>Prolongator_AGG(): New grid 724 nodes</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Courier;color:rgb(76,47,45);background-color:rgb(223,219,196)">[0] PC<span style="color:rgb(177,37,18)"><b>GAMG</b></span>OptProlongator_AGG(): Smooth P0: max eigen=1.313339e+00 min=2.474586e-02 PC=jacobi</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Courier;color:rgb(76,47,45);background-color:rgb(223,219,196)">[0] PCSetUp_<span style="color:rgb(177,37,18)"><b>GAMG</b></span>(): 2) N=724, n data cols=1, nnz/row (ave)=724, 1 active pes</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Courier;color:rgb(76,47,45);background-color:rgb(223,219,196)">[0] PC<span style="color:rgb(177,37,18)"><b>GAMG</b></span>FilterGraph(): <span class="" style="white-space:pre"> </span> 9.82914% nnz after filtering, with threshold 0., 724. nnz ave. (N=724)</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Courier;color:rgb(76,47,45);background-color:rgb(223,219,196)">[0] PC<span style="color:rgb(177,37,18)"><b>GAMG</b></span>Prolongator_AGG(): New grid 37 nodes</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Courier;color:rgb(76,47,45);background-color:rgb(223,219,196)">[0] PC<span style="color:rgb(177,37,18)"><b>GAMG</b></span>OptProlongator_AGG(): Smooth P0: max eigen=2.011784e+00 min=2.759552e-01 PC=jacobi</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Courier;color:rgb(76,47,45);background-color:rgb(223,219,196)">[0] PCSetUp_<span style="color:rgb(177,37,18)"><b>GAMG</b></span>(): 3) N=37, n data cols=1, nnz/row (ave)=37, 1 active pes</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Courier;color:rgb(76,47,45);background-color:rgb(223,219,196)">[0] PCSetUp_<span style="color:rgb(177,37,18)"><b>GAMG</b></span>(): 4 levels, grid complexity = 12.0928</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Courier;color:rgb(76,47,45);background-color:rgb(223,219,196)">[0] PCSetUp_<span style="color:rgb(177,37,18)"><b>GAMG</b></span>(): level 0) N=162000, n data rows=1, n data cols=1, nnz/row (ave)=9, np=1</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Courier;color:rgb(76,47,45);background-color:rgb(223,219,196)">[0] PC<span style="color:rgb(177,37,18)"><b>GAMG</b></span>FilterGraph(): <span class="" style="white-space:pre"> </span> 55.6621% nnz after filtering, with threshold 0., 8.863 nnz ave. (N=162000)</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Courier;color:rgb(76,47,45);background-color:rgb(223,219,196)">[0] PC<span style="color:rgb(177,37,18)"><b>GAMG</b></span>Coarsen_AGG(): Square Graph on level 1 of 1 to square</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Courier;color:rgb(76,47,45);background-color:rgb(223,219,196)">[0] PC<span style="color:rgb(177,37,18)"><b>GAMG</b></span>Prolongator_AGG(): New grid 22085 nodes</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Courier;color:rgb(76,47,45);background-color:rgb(223,219,196)">[0] PC<span style="color:rgb(177,37,18)"><b>GAMG</b></span>OptProlongator_AGG(): Smooth P0: max eigen=1.955376e+00 min=8.260696e-03 PC=jacobi</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Courier;color:rgb(76,47,45);background-color:rgb(223,219,196)">[0] PCSetUp_<span style="color:rgb(177,37,18)"><b>GAMG</b></span>(): 1) N=22085, n data cols=1, nnz/row (ave)=704, 1 active pes</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Courier;color:rgb(76,47,45);background-color:rgb(223,219,196)">[0] PC<span style="color:rgb(177,37,18)"><b>GAMG</b></span>FilterGraph(): <span class="" style="white-space:pre"> </span> 3.1314% nnz after filtering, with threshold 0., 704.128 nnz ave. (N=22085)</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Courier;color:rgb(76,47,45);background-color:rgb(223,219,196)">[0] PC<span style="color:rgb(177,37,18)"><b>GAMG</b></span>Prolongator_AGG(): New grid 2283 nodes</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Courier;color:rgb(76,47,45);background-color:rgb(223,219,196)">[0] PC<span style="color:rgb(177,37,18)"><b>GAMG</b></span>OptProlongator_AGG(): Smooth P0: max eigen=1.311291e+00 min=1.484874e-02 PC=jacobi</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Courier;color:rgb(76,47,45);background-color:rgb(223,219,196)">[0] PCSetUp_<span style="color:rgb(177,37,18)"><b>GAMG</b></span>(): 2) N=2283, n data cols=1, nnz/row (ave)=2283, 1 active pes</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Courier;color:rgb(76,47,45);background-color:rgb(223,219,196)">[0] PC<span style="color:rgb(177,37,18)"><b>GAMG</b></span>FilterGraph(): <span class="" style="white-space:pre"> </span> 3.64497% nnz after filtering, with threshold 0., 2283. nnz ave. (N=2283)</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Courier;color:rgb(76,47,45);background-color:rgb(223,219,196)">[0] PC<span style="color:rgb(177,37,18)"><b>GAMG</b></span>Prolongator_AGG(): New grid 97 nodes</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Courier;color:rgb(76,47,45);background-color:rgb(223,219,196)">[0] PC<span style="color:rgb(177,37,18)"><b>GAMG</b></span>OptProlongator_AGG(): Smooth P0: max eigen=2.043254e+00 min=1.321528e-01 PC=jacobi</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Courier;color:rgb(76,47,45);background-color:rgb(223,219,196)">[0] PCSetUp_<span style="color:rgb(177,37,18)"><b>GAMG</b></span>(): 3) N=97, n data cols=1, nnz/row (ave)=97, 1 active pes</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Courier;color:rgb(76,47,45);background-color:rgb(223,219,196)">[0] PC<span style="color:rgb(177,37,18)"><b>GAMG</b></span>FilterGraph(): <span class="" style="white-space:pre"> </span> 66.8403% nnz after filtering, with threshold 0., 97. nnz ave. (N=97)</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Courier;color:rgb(76,47,45);background-color:rgb(223,219,196)">[0] PC<span style="color:rgb(177,37,18)"><b>GAMG</b></span>Prolongator_AGG(): New grid 5 nodes</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Courier;color:rgb(76,47,45);background-color:rgb(223,219,196)">[0] PC<span style="color:rgb(177,37,18)"><b>GAMG</b></span>OptProlongator_AGG(): Smooth P0: max eigen=1.653762e+00 min=4.460582e-01 PC=jacobi</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Courier;color:rgb(76,47,45);background-color:rgb(223,219,196)">[0] PCSetUp_<span style="color:rgb(177,37,18)"><b>GAMG</b></span>(): 4) N=5, n data cols=1, nnz/row (ave)=5, 1 active pes</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Courier;color:rgb(76,47,45);background-color:rgb(223,219,196)">[0] PCSetUp_<span style="color:rgb(177,37,18)"><b>GAMG</b></span>(): 5 levels, grid complexity = 15.4673</p><br>Btw i did a smaller problem. Unit cube of 30x30x30 not 40x40x40.</div><div><br></div><div>I used those smaller threshold values you mentioned but nothing really changed</div><div><br></div><div>Thanks,</div><div>Justin</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Mar 4, 2016 at 7:40 AM, Mark Adams <span dir="ltr"><<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span class="">On Fri, Mar 4, 2016 at 2:05 AM, Justin Chang <span dir="ltr"><<a href="mailto:jychang48@gmail.com" target="_blank">jychang48@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Mark,<div><br></div><div>Using "-pc_gamg_square_graph 10" didn't change anything. I used values of 1, 10, 100, and 1000 and the performance seemed unaffected.</div></blockquote><div><br></div></span><div>Humm. please run with -info and grep on GAMG, This will be about 20 lines but -info is very noisy. Perhaps if you can do the same thing for ML (I'm not sure if the ML interface supports verbose output like this) that would be useful.</div><div><br></div><div>BTW, -pc_gamg_square_graph N is the number of levels to square the graph. You will see in the verbose output the number of non-zeros per row in your problems starts at 9 and goes up to ~740, and ~5370, and then 207 the the coarse grid where there are 207 columns.</div><span class=""><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div><br></div><div>Changing the threshold of -pc_gamg_threshold to 0.8 did decrease wall-clock time but it required more iterations.</div></blockquote><div><br></div></span><div>That is very large. A more reasonable scan would be: 0, 0.01, 0.04, 0.08.</div><span class=""><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div><br></div><div>I am not really sure how I go about tinkering around with GAMG or even ML. Do you have a manual/reference/paper/etc that describes what's going on within gamg?</div></blockquote><div><br></div></span><div>There is a section in the manual. It goes through some of these troubleshooting techniques.</div><div><div class="h5"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div><br></div><div>Thanks,</div><div>Justin<div><div><br><br>On Thursday, March 3, 2016, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">You have a very sparse 3D problem, with 9 non-zeros per row. It is coarsening very slowly and creating huge coarse grids. which are expensive to construct. The superlinear speedup is from cache effects, most likely. First try with:<div><br></div><div>-pc_gamg_square_graph 10</div><div><br></div><div>ML must have some AI in there to do this automatically, because gamg are pretty similar algorithmically. There is a threshold parameter that is important (-pc_gamg_threshold <0.0>) and I think ML has the same default. ML is doing OK, but I would guess that if you use like 0.02 for MLs threshold you would see some improvement. </div><div><br></div><div>Hypre is doing pretty bad also. I suspect that it is getting confused as well. I know less about how to deal with hypre.</div><div><br></div><div>If you use -info and grep on GAMG you will see about 20 lines that will tell you the number of equations on level and the average number of non-zeros per row. In 3D the reduction per level should be -- very approximately -- 30x and the number of non-zeros per row should not explode, but getting up to several hundred is OK.</div><div><br></div><div>If you care to test this we should be able to get ML and GAMG to agree pretty well. ML is a nice solver, but our core numerics should be about the same. I tested this on a 3D elasticity problem a few years ago. That said, I think your ML solve is pretty good.</div><div><br></div><div>Mark</div><div><br></div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Mar 3, 2016 at 4:36 AM, Lawrence Mitchell <span dir="ltr"><<a>lawrence.mitchell@imperial.ac.uk</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">On 02/03/16 22:28, Justin Chang wrote:<br>
...<br>
<span><br>
<br>
> Down solver (pre-smoother) on level 3<br>
><br>
</span>> KSP Object: (solver_fieldsplit_1_mg_levels_3_)<br>
<span>> linear system matrix = precond matrix:<br>
</span>...<br>
<span>> Mat Object: 1 MPI processes<br>
><br>
> type: seqaij<br>
><br>
> rows=52147, cols=52147<br>
><br>
> total: nonzeros=38604909, allocated nonzeros=38604909<br>
><br>
> total number of mallocs used during MatSetValues calls =2<br>
><br>
> not using I-node routines<br>
><br>
</span><span>> Down solver (pre-smoother) on level 4<br>
><br>
</span>> KSP Object: (solver_fieldsplit_1_mg_levels_4_)<br>
<span>> linear system matrix followed by preconditioner matrix:<br>
><br>
> Mat Object: (solver_fieldsplit_1_)<br>
<br>
</span>...<br>
<span>><br>
> Mat Object: 1 MPI processes<br>
><br>
> type: seqaij<br>
><br>
> rows=384000, cols=384000<br>
><br>
> total: nonzeros=3416452, allocated nonzeros=3416452<br>
<br>
<br>
</span>This looks pretty suspicious to me. The original matrix on the finest<br>
level has 3.8e5 rows and ~3.4e6 nonzeros. The next level up, the<br>
coarsening produces 5.2e4 rows, but 38e6 nonzeros.<br>
<br>
FWIW, although Justin's PETSc is from Oct 2015, I get the same<br>
behaviour with:<br>
<br>
ad5697c (Master as of 1st March).<br>
<br>
If I compare with the coarse operators that ML produces on the same<br>
problem:<br>
<br>
The original matrix has, again:<br>
<span><br>
Mat Object: 1 MPI processes<br>
type: seqaij<br>
rows=384000, cols=384000<br>
total: nonzeros=3416452, allocated nonzeros=3416452<br>
total number of mallocs used during MatSetValues calls=0<br>
not using I-node routines<br>
<br>
</span>While the next finest level has:<br>
<span><br>
Mat Object: 1 MPI processes<br>
type: seqaij<br>
</span> rows=65258, cols=65258<br>
total: nonzeros=1318400, allocated nonzeros=1318400<br>
<span> total number of mallocs used during MatSetValues calls=0<br>
not using I-node routines<br>
<br>
</span>So we have 6.5e4 rows and 1.3e6 nonzeros, which seems more plausible.<br>
<br>
Cheers,<br>
<br>
Lawrence<br>
<br>
</blockquote></div><br></div>
</blockquote></div></div></div>
</blockquote></div></div></div><br></div></div>
</blockquote></div><br></div>