[petsc-users] Strange GAMG performance for mixed FE formulation

Fri Mar 4 16:04:05 CST 2016

You seem to have 3 of one type of solve that is give 'square_graph 1':

0] PC*GAMG*Coarsen_AGG(): Square Graph on level 1 of 1 to square

This has 9 nnz-row and 44% are zero:

[0] PC*GAMG*FilterGraph():   55.7114% nnz after filtering, with threshold
0., 8.79533 nnz ave.

So you  want to use a _negative_ threshold.  This will keep zeros in the
graph and help it to coarsen faster.  And you could try adding more levels
to square the graph.

The second type of solve has the 'square_graph 10'.  It looks like the
first solve.  It should also use a negative threshold also.

ML has a default of zero for the threshold, but it seems to keep zeros
whereas GAMG does not.

Mark

On Fri, Mar 4, 2016 at 10:38 AM, Justin Chang <jychang48 at gmail.com> wrote:

> Time to solution went from 100 seconds to 30 seconds once i used 10
> graphs. Using 20 graphs started to increase in time slightly
>
> On Fri, Mar 4, 2016 at 8:35 AM, Justin Chang <jychang48 at gmail.com> wrote:
>
>> You're right. This is what I have:
>>
>> [0] PCSetUp_*GAMG*(): level 0) N=48000, n data rows=1, n data cols=1,
>> nnz/row (ave)=9, np=1
>>
>> [0] PC*GAMG*FilterGraph():  55.7114% nnz after filtering, with threshold
>> 0., 8.79533 nnz ave. (N=48000)
>>
>> [0] PC*GAMG*Coarsen_AGG(): Square Graph on level 1 of 1 to square
>>
>> [0] PC*GAMG*Prolongator_AGG(): New grid 6672 nodes
>>
>> [0] PC*GAMG*OptProlongator_AGG(): Smooth P0: max eigen=1.954700e+00
>> min=1.040410e-02 PC=jacobi
>>
>> [0] PCSetUp_*GAMG*(): 1) N=6672, n data cols=1, nnz/row (ave)=623, 1
>> active pes
>>
>> [0] PC*GAMG*FilterGraph():  3.40099% nnz after filtering, with threshold
>> 0., 623.135 nnz ave. (N=6672)
>>
>> [0] PC*GAMG*Prolongator_AGG(): New grid 724 nodes
>>
>> [0] PC*GAMG*OptProlongator_AGG(): Smooth P0: max eigen=1.313339e+00
>> min=2.474586e-02 PC=jacobi
>>
>> [0] PCSetUp_*GAMG*(): 2) N=724, n data cols=1, nnz/row (ave)=724, 1
>> active pes
>>
>> [0] PC*GAMG*FilterGraph():  9.82914% nnz after filtering, with threshold
>> 0., 724. nnz ave. (N=724)
>>
>> [0] PC*GAMG*Prolongator_AGG(): New grid 37 nodes
>>
>> [0] PC*GAMG*OptProlongator_AGG(): Smooth P0: max eigen=2.011784e+00
>> min=2.759552e-01 PC=jacobi
>>
>> [0] PCSetUp_*GAMG*(): 3) N=37, n data cols=1, nnz/row (ave)=37, 1 active
>> pes
>>
>> [0] PCSetUp_*GAMG*(): 4 levels, grid complexity = 12.0928
>>
>> [0] PCSetUp_*GAMG*(): level 0) N=48000, n data rows=1, n data cols=1,
>> nnz/row (ave)=9, np=1
>>
>> [0] PC*GAMG*FilterGraph():  55.7114% nnz after filtering, with threshold
>> 0., 8.79533 nnz ave. (N=48000)
>>
>> [0] PC*GAMG*Coarsen_AGG(): Square Graph on level 1 of 1 to square
>>
>> [0] PC*GAMG*Prolongator_AGG(): New grid 6672 nodes
>>
>> [0] PC*GAMG*OptProlongator_AGG(): Smooth P0: max eigen=1.954700e+00
>> min=1.040410e-02 PC=jacobi
>>
>> [0] PCSetUp_*GAMG*(): 1) N=6672, n data cols=1, nnz/row (ave)=623, 1
>> active pes
>>
>> [0] PC*GAMG*FilterGraph():  3.40099% nnz after filtering, with threshold
>> 0., 623.135 nnz ave. (N=6672)
>>
>> [0] PC*GAMG*Prolongator_AGG(): New grid 724 nodes
>>
>> [0] PC*GAMG*OptProlongator_AGG(): Smooth P0: max eigen=1.313339e+00
>> min=2.474586e-02 PC=jacobi
>>
>> [0] PCSetUp_*GAMG*(): 2) N=724, n data cols=1, nnz/row (ave)=724, 1
>> active pes
>>
>> [0] PC*GAMG*FilterGraph():  9.82914% nnz after filtering, with threshold
>> 0., 724. nnz ave. (N=724)
>>
>> [0] PC*GAMG*Prolongator_AGG(): New grid 37 nodes
>>
>> [0] PC*GAMG*OptProlongator_AGG(): Smooth P0: max eigen=2.011784e+00
>> min=2.759552e-01 PC=jacobi
>>
>> [0] PCSetUp_*GAMG*(): 3) N=37, n data cols=1, nnz/row (ave)=37, 1 active
>> pes
>>
>> [0] PCSetUp_*GAMG*(): 4 levels, grid complexity = 12.0928
>>
>> [0] PCSetUp_*GAMG*(): level 0) N=162000, n data rows=1, n data cols=1,
>> nnz/row (ave)=9, np=1
>>
>> [0] PC*GAMG*FilterGraph():  55.6621% nnz after filtering, with threshold
>> 0., 8.863 nnz ave. (N=162000)
>>
>> [0] PC*GAMG*Coarsen_AGG(): Square Graph on level 1 of 1 to square
>>
>> [0] PC*GAMG*Prolongator_AGG(): New grid 22085 nodes
>>
>> [0] PC*GAMG*OptProlongator_AGG(): Smooth P0: max eigen=1.955376e+00
>> min=8.260696e-03 PC=jacobi
>>
>> [0] PCSetUp_*GAMG*(): 1) N=22085, n data cols=1, nnz/row (ave)=704, 1
>> active pes
>>
>> [0] PC*GAMG*FilterGraph():  3.1314% nnz after filtering, with threshold
>> 0., 704.128 nnz ave. (N=22085)
>>
>> [0] PC*GAMG*Prolongator_AGG(): New grid 2283 nodes
>>
>> [0] PC*GAMG*OptProlongator_AGG(): Smooth P0: max eigen=1.311291e+00
>> min=1.484874e-02 PC=jacobi
>>
>> [0] PCSetUp_*GAMG*(): 2) N=2283, n data cols=1, nnz/row (ave)=2283, 1
>> active pes
>>
>> [0] PC*GAMG*FilterGraph():  3.64497% nnz after filtering, with threshold
>> 0., 2283. nnz ave. (N=2283)
>>
>> [0] PC*GAMG*Prolongator_AGG(): New grid 97 nodes
>>
>> [0] PC*GAMG*OptProlongator_AGG(): Smooth P0: max eigen=2.043254e+00
>> min=1.321528e-01 PC=jacobi
>>
>> [0] PCSetUp_*GAMG*(): 3) N=97, n data cols=1, nnz/row (ave)=97, 1 active
>> pes
>>
>> [0] PC*GAMG*FilterGraph():  66.8403% nnz after filtering, with threshold
>> 0., 97. nnz ave. (N=97)
>>
>> [0] PC*GAMG*Prolongator_AGG(): New grid 5 nodes
>>
>> [0] PC*GAMG*OptProlongator_AGG(): Smooth P0: max eigen=1.653762e+00
>> min=4.460582e-01 PC=jacobi
>>
>> [0] PCSetUp_*GAMG*(): 4) N=5, n data cols=1, nnz/row (ave)=5, 1 active
>> pes
>>
>> [0] PCSetUp_*GAMG*(): 5 levels, grid complexity = 15.4673
>>
>> [0] PCSetUp_*GAMG*(): level 0) N=162000, n data rows=1, n data cols=1,
>> nnz/row (ave)=9, np=1
>>
>> [0] PC*GAMG*FilterGraph():  55.6621% nnz after filtering, with threshold
>> 0., 8.863 nnz ave. (N=162000)
>>
>> [0] PC*GAMG*Coarsen_AGG(): Square Graph on level 1 of 10 to square
>>
>> [0] PC*GAMG*Prolongator_AGG(): New grid 22085 nodes
>>
>> [0] PC*GAMG*OptProlongator_AGG(): Smooth P0: max eigen=1.955376e+00
>> min=8.260696e-03 PC=jacobi
>>
>> [0] PCSetUp_*GAMG*(): 1) N=22085, n data cols=1, nnz/row (ave)=704, 1
>> active pes
>>
>> [0] PC*GAMG*FilterGraph():  3.1314% nnz after filtering, with threshold
>> 0., 704.128 nnz ave. (N=22085)
>>
>> [0] PC*GAMG*Coarsen_AGG(): Square Graph on level 2 of 10 to square
>>
>> [0] PC*GAMG*Prolongator_AGG(): New grid 545 nodes
>>
>> [0] PC*GAMG*OptProlongator_AGG(): Smooth P0: max eigen=1.311291e+00
>> min=1.484874e-02 PC=jacobi
>>
>> [0] PCSetUp_*GAMG*(): 2) N=545, n data cols=1, nnz/row (ave)=545, 1
>> active pes
>>
>> [0] PC*GAMG*FilterGraph():  7.55997% nnz after filtering, with threshold
>> 0., 545. nnz ave. (N=545)
>>
>> [0] PC*GAMG*Coarsen_AGG(): Square Graph on level 3 of 10 to square
>>
>> [0] PC*GAMG*Prolongator_AGG(): New grid 11 nodes
>>
>> [0] PC*GAMG*OptProlongator_AGG(): Smooth P0: max eigen=1.368729e+00
>> min=1.563750e-01 PC=jacobi
>>
>> [0] PCSetUp_*GAMG*(): 3) N=11, n data cols=1, nnz/row (ave)=11, 1 active
>> pes
>>
>> [0] PCSetUp_*GAMG*(): 4 levels, grid complexity = 12.0376
>>
>> On Fri, Mar 4, 2016 at 8:31 AM, Lawrence Mitchell <
>> lawrence.mitchell at imperial.ac.uk> wrote:
>>
>>>
>>> > On 4 Mar 2016, at 15:24, Justin Chang <jychang48 at gmail.com> wrote:
>>> >
>>> > So with -pc_gamg_square_graph 10 I get the following:
>>>
>>> Because you're using gamg inside the fieldsplit, I think you need:
>>>
>>> -fieldsplit_1_pc_gamg_square_graph 10
>>>
>>>
>>>
>>> > [0] PCSetUp_GAMG(): level 0) N=48000, n data rows=1, n data cols=1,
>>> nnz/row (ave)=9, np=1
>>> > [0] PCGAMGFilterGraph():       55.7114% nnz after filtering, with
>>> threshold 0., 8.79533 nnz ave. (N=48000)
>>> > [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
>>>                                                  ^^^^^
>>>
>>> Cheers,
>>>
>>> Lawrence
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20160304/de6c848c/attachment.html>