[petsc-users] Strange GAMG performance for mixed FE formulation
Justin Chang
jychang48 at gmail.com
Tue Mar 8 15:50:41 CST 2016
Mark,
So I had the following options:
-solver_fieldsplit_1_pc_type gamg
-solver_fieldsplit_1_pc_gamg_square_graph 10
-solver_fieldsplit_1_mg_levels_ksp_type richardson
-solver_fieldsplit_1_pc_gamg_threshold <-0.02/0.00/0.02>
-info
-log_summary
where I varied the gamg threshold from -0.02, 0.00, and 0.02. Richardson
for the mg_levels_ksp_type option did improve the performance, but it seems
using a non-zero threshold worsens the overall solver.
for "-solver_fieldsplit_1_pc_gamg_threshold 0.02":
>> grep 'GAMG' positive_threshold
[0] PCSetUp_*GAMG*(): level 0) N=162000, n data rows=1, n data cols=1,
nnz/row (ave)=9, np=1
[0] PC*GAMG*FilterGraph(): 55.6621% nnz after filtering, with threshold
0.02, 8.863 nnz ave. (N=162000)
[0] PC*GAMG*Coarsen_AGG(): Square Graph on level 1 of 10 to square
[0] PC*GAMG*Prolongator_AGG(): New grid 22085 nodes
[0] PC*GAMG*OptProlongator_AGG(): Smooth P0: max eigen=1.955376e+00
min=8.260696e-03 PC=jacobi
[0] PCSetUp_*GAMG*(): 1) N=22085, n data cols=1, nnz/row (ave)=704, 1
active pes
[0] PC*GAMG*FilterGraph(): 2.20542% nnz after filtering, with threshold
0.02, 704.128 nnz ave. (N=22085)
[0] PC*GAMG*Coarsen_AGG(): Square Graph on level 2 of 10 to square
[0] PC*GAMG*Prolongator_AGG(): New grid 798 nodes
[0] PC*GAMG*OptProlongator_AGG(): Smooth P0: max eigen=1.311291e+00
min=1.484874e-02 PC=jacobi
[0] PCSetUp_*GAMG*(): 2) N=798, n data cols=1, nnz/row (ave)=798, 1 active
pes
[0] PC*GAMG*FilterGraph(): 1.75564% nnz after filtering, with threshold
0.02, 798. nnz ave. (N=798)
[0] PC*GAMG*Coarsen_AGG(): Square Graph on level 3 of 10 to square
[0] PC*GAMG*Prolongator_AGG(): New grid 44 nodes
[0] PC*GAMG*OptProlongator_AGG(): Smooth P0: max eigen=1.668188e+00
min=1.402892e-01 PC=jacobi
[0] PCSetUp_*GAMG*(): 3) N=44, n data cols=1, nnz/row (ave)=44, 1 active pes
[0] PCSetUp_*GAMG*(): 4 levels, grid complexity = 12.2755
For "-solver_fieldsplit_1_pc_gamg_threshold 0.00":
[0] PCSetUp_*GAMG*(): level 0) N=162000, n data rows=1, n data cols=1,
nnz/row (ave)=9, np=1
[0] PC*GAMG*FilterGraph(): 55.6621% nnz after filtering, with threshold
0., 8.863 nnz ave. (N=162000)
[0] PC*GAMG*Coarsen_AGG(): Square Graph on level 1 of 10 to square
[0] PC*GAMG*Prolongator_AGG(): New grid 22085 nodes
[0] PC*GAMG*OptProlongator_AGG(): Smooth P0: max eigen=1.955376e+00
min=8.260696e-03 PC=jacobi
[0] PCSetUp_*GAMG*(): 1) N=22085, n data cols=1, nnz/row (ave)=704, 1
active pes
[0] PC*GAMG*FilterGraph(): 3.1314% nnz after filtering, with threshold 0.,
704.128 nnz ave. (N=22085)
[0] PC*GAMG*Coarsen_AGG(): Square Graph on level 2 of 10 to square
[0] PC*GAMG*Prolongator_AGG(): New grid 545 nodes
[0] PC*GAMG*OptProlongator_AGG(): Smooth P0: max eigen=1.311291e+00
min=1.484874e-02 PC=jacobi
[0] PCSetUp_*GAMG*(): 2) N=545, n data cols=1, nnz/row (ave)=545, 1 active
pes
[0] PC*GAMG*FilterGraph(): 7.55997% nnz after filtering, with threshold
0., 545. nnz ave. (N=545)
[0] PC*GAMG*Coarsen_AGG(): Square Graph on level 3 of 10 to square
[0] PC*GAMG*Prolongator_AGG(): New grid 11 nodes
[0] PC*GAMG*OptProlongator_AGG(): Smooth P0: max eigen=1.368729e+00
min=1.563750e-01 PC=jacobi
[0] PCSetUp_*GAMG*(): 3) N=11, n data cols=1, nnz/row (ave)=11, 1 active pes
[0] PCSetUp_*GAMG*(): 4 levels, grid complexity = 12.0376
and for "-solver_fieldsplit_1_pc_gamg_threshold -0.02":
[0] PCSetUp_*GAMG*(): level 0) N=162000, n data rows=1, n data cols=1,
nnz/row (ave)=9, np=1
[0] PC*GAMG*Coarsen_AGG(): Square Graph on level 1 of 10 to square
[0] PC*GAMG*Prolongator_AGG(): New grid 10406 nodes
[0] PC*GAMG*OptProlongator_AGG(): Smooth P0: max eigen=1.955376e+00
min=8.260696e-03 PC=jacobi
[0] PCSetUp_*GAMG*(): 1) N=10406, n data cols=1, nnz/row (ave)=945, 1
active pes
[0] PC*GAMG*Coarsen_AGG(): Square Graph on level 2 of 10 to square
[0] PC*GAMG*Prolongator_AGG(): New grid 1 nodes
[0] PC*GAMG*OptProlongator_AGG(): Smooth P0: max eigen=1.248316e+00
min=9.020787e-02 PC=jacobi
[0] PCSetUp_*GAMG*(): 2) N=1, n data cols=1, nnz/row (ave)=1, 1 active pes
[0] PCSetUp_*GAMG*(): HARD stop of coarsening on level 1. Grid too small:
1 block nodes
[0] PCSetUp_*GAMG*(): 3 levels, grid complexity = 7.85162
Attached are the log summaries for the three respective cases (positive,
zero, and negative).
1) From the log summaries, it seems positive negative threshold is
extremely slow. Though not as slow as if I had no "square_graph 10" and
zero threshold has the "best" performance.
2) I tried mg_levels_pc_type sor but it wasn't as good as richardson
3) I tried increasing the square_graph number to 100 and 1000 but saw no
improvement
4) I could not find any way to identify the ML information from -info, so I
simply attached the log_summary using ML for reference
5) Is there anything else I could try? Thanks again for all your help
Thanks,
Justin
On Fri, Mar 4, 2016 at 4:36 PM, Mark Adams <mfadams at lbl.gov> wrote:
> And it looks like you have a well behaved Laplacian here (M-matrix) so I
> would guess 'richardson' would be faster as the smoother, instead of
> 'chebyshev'.
>
> On Fri, Mar 4, 2016 at 5:04 PM, Mark Adams <mfadams at lbl.gov> wrote:
>
>> You seem to have 3 of one type of solve that is give 'square_graph 1':
>>
>> 0] PC*GAMG*Coarsen_AGG(): Square Graph on level 1 of 1 to square
>>
>> This has 9 nnz-row and 44% are zero:
>>
>> [0] PC*GAMG*FilterGraph(): 55.7114% nnz after filtering, with
>> threshold 0., 8.79533 nnz ave.
>>
>> So you want to use a _negative_ threshold. This will keep zeros in the
>> graph and help it to coarsen faster. And you could try adding more levels
>> to square the graph.
>>
>> The second type of solve has the 'square_graph 10'. It looks like the
>> first solve. It should also use a negative threshold also.
>>
>> ML has a default of zero for the threshold, but it seems to keep zeros
>> whereas GAMG does not.
>>
>> Mark
>>
>>
>> On Fri, Mar 4, 2016 at 10:38 AM, Justin Chang <jychang48 at gmail.com>
>> wrote:
>>
>>> Time to solution went from 100 seconds to 30 seconds once i used 10
>>> graphs. Using 20 graphs started to increase in time slightly
>>>
>>> On Fri, Mar 4, 2016 at 8:35 AM, Justin Chang <jychang48 at gmail.com>
>>> wrote:
>>>
>>>> You're right. This is what I have:
>>>>
>>>> [0] PCSetUp_*GAMG*(): level 0) N=48000, n data rows=1, n data cols=1,
>>>> nnz/row (ave)=9, np=1
>>>>
>>>> [0] PC*GAMG*FilterGraph(): 55.7114% nnz after filtering, with
>>>> threshold 0., 8.79533 nnz ave. (N=48000)
>>>>
>>>> [0] PC*GAMG*Coarsen_AGG(): Square Graph on level 1 of 1 to square
>>>>
>>>> [0] PC*GAMG*Prolongator_AGG(): New grid 6672 nodes
>>>>
>>>> [0] PC*GAMG*OptProlongator_AGG(): Smooth P0: max eigen=1.954700e+00
>>>> min=1.040410e-02 PC=jacobi
>>>>
>>>> [0] PCSetUp_*GAMG*(): 1) N=6672, n data cols=1, nnz/row (ave)=623, 1
>>>> active pes
>>>>
>>>> [0] PC*GAMG*FilterGraph(): 3.40099% nnz after filtering, with
>>>> threshold 0., 623.135 nnz ave. (N=6672)
>>>>
>>>> [0] PC*GAMG*Prolongator_AGG(): New grid 724 nodes
>>>>
>>>> [0] PC*GAMG*OptProlongator_AGG(): Smooth P0: max eigen=1.313339e+00
>>>> min=2.474586e-02 PC=jacobi
>>>>
>>>> [0] PCSetUp_*GAMG*(): 2) N=724, n data cols=1, nnz/row (ave)=724, 1
>>>> active pes
>>>>
>>>> [0] PC*GAMG*FilterGraph(): 9.82914% nnz after filtering, with
>>>> threshold 0., 724. nnz ave. (N=724)
>>>>
>>>> [0] PC*GAMG*Prolongator_AGG(): New grid 37 nodes
>>>>
>>>> [0] PC*GAMG*OptProlongator_AGG(): Smooth P0: max eigen=2.011784e+00
>>>> min=2.759552e-01 PC=jacobi
>>>>
>>>> [0] PCSetUp_*GAMG*(): 3) N=37, n data cols=1, nnz/row (ave)=37, 1
>>>> active pes
>>>>
>>>> [0] PCSetUp_*GAMG*(): 4 levels, grid complexity = 12.0928
>>>>
>>>> [0] PCSetUp_*GAMG*(): level 0) N=48000, n data rows=1, n data cols=1,
>>>> nnz/row (ave)=9, np=1
>>>>
>>>> [0] PC*GAMG*FilterGraph(): 55.7114% nnz after filtering, with
>>>> threshold 0., 8.79533 nnz ave. (N=48000)
>>>>
>>>> [0] PC*GAMG*Coarsen_AGG(): Square Graph on level 1 of 1 to square
>>>>
>>>> [0] PC*GAMG*Prolongator_AGG(): New grid 6672 nodes
>>>>
>>>> [0] PC*GAMG*OptProlongator_AGG(): Smooth P0: max eigen=1.954700e+00
>>>> min=1.040410e-02 PC=jacobi
>>>>
>>>> [0] PCSetUp_*GAMG*(): 1) N=6672, n data cols=1, nnz/row (ave)=623, 1
>>>> active pes
>>>>
>>>> [0] PC*GAMG*FilterGraph(): 3.40099% nnz after filtering, with
>>>> threshold 0., 623.135 nnz ave. (N=6672)
>>>>
>>>> [0] PC*GAMG*Prolongator_AGG(): New grid 724 nodes
>>>>
>>>> [0] PC*GAMG*OptProlongator_AGG(): Smooth P0: max eigen=1.313339e+00
>>>> min=2.474586e-02 PC=jacobi
>>>>
>>>> [0] PCSetUp_*GAMG*(): 2) N=724, n data cols=1, nnz/row (ave)=724, 1
>>>> active pes
>>>>
>>>> [0] PC*GAMG*FilterGraph(): 9.82914% nnz after filtering, with
>>>> threshold 0., 724. nnz ave. (N=724)
>>>>
>>>> [0] PC*GAMG*Prolongator_AGG(): New grid 37 nodes
>>>>
>>>> [0] PC*GAMG*OptProlongator_AGG(): Smooth P0: max eigen=2.011784e+00
>>>> min=2.759552e-01 PC=jacobi
>>>>
>>>> [0] PCSetUp_*GAMG*(): 3) N=37, n data cols=1, nnz/row (ave)=37, 1
>>>> active pes
>>>>
>>>> [0] PCSetUp_*GAMG*(): 4 levels, grid complexity = 12.0928
>>>>
>>>> [0] PCSetUp_*GAMG*(): level 0) N=162000, n data rows=1, n data cols=1,
>>>> nnz/row (ave)=9, np=1
>>>>
>>>> [0] PC*GAMG*FilterGraph(): 55.6621% nnz after filtering, with
>>>> threshold 0., 8.863 nnz ave. (N=162000)
>>>>
>>>> [0] PC*GAMG*Coarsen_AGG(): Square Graph on level 1 of 1 to square
>>>>
>>>> [0] PC*GAMG*Prolongator_AGG(): New grid 22085 nodes
>>>>
>>>> [0] PC*GAMG*OptProlongator_AGG(): Smooth P0: max eigen=1.955376e+00
>>>> min=8.260696e-03 PC=jacobi
>>>>
>>>> [0] PCSetUp_*GAMG*(): 1) N=22085, n data cols=1, nnz/row (ave)=704, 1
>>>> active pes
>>>>
>>>> [0] PC*GAMG*FilterGraph(): 3.1314% nnz after filtering, with
>>>> threshold 0., 704.128 nnz ave. (N=22085)
>>>>
>>>> [0] PC*GAMG*Prolongator_AGG(): New grid 2283 nodes
>>>>
>>>> [0] PC*GAMG*OptProlongator_AGG(): Smooth P0: max eigen=1.311291e+00
>>>> min=1.484874e-02 PC=jacobi
>>>>
>>>> [0] PCSetUp_*GAMG*(): 2) N=2283, n data cols=1, nnz/row (ave)=2283, 1
>>>> active pes
>>>>
>>>> [0] PC*GAMG*FilterGraph(): 3.64497% nnz after filtering, with
>>>> threshold 0., 2283. nnz ave. (N=2283)
>>>>
>>>> [0] PC*GAMG*Prolongator_AGG(): New grid 97 nodes
>>>>
>>>> [0] PC*GAMG*OptProlongator_AGG(): Smooth P0: max eigen=2.043254e+00
>>>> min=1.321528e-01 PC=jacobi
>>>>
>>>> [0] PCSetUp_*GAMG*(): 3) N=97, n data cols=1, nnz/row (ave)=97, 1
>>>> active pes
>>>>
>>>> [0] PC*GAMG*FilterGraph(): 66.8403% nnz after filtering, with
>>>> threshold 0., 97. nnz ave. (N=97)
>>>>
>>>> [0] PC*GAMG*Prolongator_AGG(): New grid 5 nodes
>>>>
>>>> [0] PC*GAMG*OptProlongator_AGG(): Smooth P0: max eigen=1.653762e+00
>>>> min=4.460582e-01 PC=jacobi
>>>>
>>>> [0] PCSetUp_*GAMG*(): 4) N=5, n data cols=1, nnz/row (ave)=5, 1 active
>>>> pes
>>>>
>>>> [0] PCSetUp_*GAMG*(): 5 levels, grid complexity = 15.4673
>>>>
>>>> [0] PCSetUp_*GAMG*(): level 0) N=162000, n data rows=1, n data cols=1,
>>>> nnz/row (ave)=9, np=1
>>>>
>>>> [0] PC*GAMG*FilterGraph(): 55.6621% nnz after filtering, with
>>>> threshold 0., 8.863 nnz ave. (N=162000)
>>>>
>>>> [0] PC*GAMG*Coarsen_AGG(): Square Graph on level 1 of 10 to square
>>>>
>>>> [0] PC*GAMG*Prolongator_AGG(): New grid 22085 nodes
>>>>
>>>> [0] PC*GAMG*OptProlongator_AGG(): Smooth P0: max eigen=1.955376e+00
>>>> min=8.260696e-03 PC=jacobi
>>>>
>>>> [0] PCSetUp_*GAMG*(): 1) N=22085, n data cols=1, nnz/row (ave)=704, 1
>>>> active pes
>>>>
>>>> [0] PC*GAMG*FilterGraph(): 3.1314% nnz after filtering, with
>>>> threshold 0., 704.128 nnz ave. (N=22085)
>>>>
>>>> [0] PC*GAMG*Coarsen_AGG(): Square Graph on level 2 of 10 to square
>>>>
>>>> [0] PC*GAMG*Prolongator_AGG(): New grid 545 nodes
>>>>
>>>> [0] PC*GAMG*OptProlongator_AGG(): Smooth P0: max eigen=1.311291e+00
>>>> min=1.484874e-02 PC=jacobi
>>>>
>>>> [0] PCSetUp_*GAMG*(): 2) N=545, n data cols=1, nnz/row (ave)=545, 1
>>>> active pes
>>>>
>>>> [0] PC*GAMG*FilterGraph(): 7.55997% nnz after filtering, with
>>>> threshold 0., 545. nnz ave. (N=545)
>>>>
>>>> [0] PC*GAMG*Coarsen_AGG(): Square Graph on level 3 of 10 to square
>>>>
>>>> [0] PC*GAMG*Prolongator_AGG(): New grid 11 nodes
>>>>
>>>> [0] PC*GAMG*OptProlongator_AGG(): Smooth P0: max eigen=1.368729e+00
>>>> min=1.563750e-01 PC=jacobi
>>>>
>>>> [0] PCSetUp_*GAMG*(): 3) N=11, n data cols=1, nnz/row (ave)=11, 1
>>>> active pes
>>>>
>>>> [0] PCSetUp_*GAMG*(): 4 levels, grid complexity = 12.0376
>>>>
>>>> On Fri, Mar 4, 2016 at 8:31 AM, Lawrence Mitchell <
>>>> lawrence.mitchell at imperial.ac.uk> wrote:
>>>>
>>>>>
>>>>> > On 4 Mar 2016, at 15:24, Justin Chang <jychang48 at gmail.com> wrote:
>>>>> >
>>>>> > So with -pc_gamg_square_graph 10 I get the following:
>>>>>
>>>>> Because you're using gamg inside the fieldsplit, I think you need:
>>>>>
>>>>> -fieldsplit_1_pc_gamg_square_graph 10
>>>>>
>>>>>
>>>>>
>>>>> > [0] PCSetUp_GAMG(): level 0) N=48000, n data rows=1, n data cols=1,
>>>>> nnz/row (ave)=9, np=1
>>>>> > [0] PCGAMGFilterGraph(): 55.7114% nnz after filtering, with
>>>>> threshold 0., 8.79533 nnz ave. (N=48000)
>>>>> > [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
>>>>> ^^^^^
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Lawrence
>>>>>
>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20160308/aa0c0318/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: negative_threshold
Type: application/octet-stream
Size: 17987 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20160308/aa0c0318/attachment-0004.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: positive_threshold
Type: application/octet-stream
Size: 17977 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20160308/aa0c0318/attachment-0005.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: zero_threshold
Type: application/octet-stream
Size: 17976 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20160308/aa0c0318/attachment-0006.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ml_summary
Type: application/octet-stream
Size: 14850 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20160308/aa0c0318/attachment-0007.obj>
More information about the petsc-users
mailing list