<div dir="ltr"><div class="gmail_quote"><div dir="ltr">On Thu, Jul 26, 2018 at 2:43 PM Jed Brown <<a href="mailto:jed@jedbrown.org">jed@jedbrown.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> writes:<br>
<br>
> On Thu, Jul 26, 2018 at 12:56 PM Fande Kong <<a href="mailto:fdkong.jd@gmail.com" target="_blank">fdkong.jd@gmail.com</a>> wrote:<br>
><br>
>><br>
>><br>
>> On Thu, Jul 26, 2018 at 10:35 AM, Junchao Zhang <<a href="mailto:jczhang@mcs.anl.gov" target="_blank">jczhang@mcs.anl.gov</a>><br>
>> wrote:<br>
>><br>
>>> On Thu, Jul 26, 2018 at 11:15 AM, Fande Kong <<a href="mailto:fdkong.jd@gmail.com" target="_blank">fdkong.jd@gmail.com</a>> wrote:<br>
>>><br>
>>>><br>
>>>><br>
>>>> On Thu, Jul 26, 2018 at 9:51 AM, Junchao Zhang <<a href="mailto:jczhang@mcs.anl.gov" target="_blank">jczhang@mcs.anl.gov</a>><br>
>>>> wrote:<br>
>>>><br>
>>>>> Hi, Pierre,<br>
>>>>> From your log_view files, I see you did strong scaling. You used 4X<br>
>>>>> more cores, but the execution time only dropped from 3.9143e+04<br>
>>>>> to 1.6910e+04.<br>
>>>>> From my previous analysis of a GAMG weak scaling test, it looks<br>
>>>>> communication is one of the reasons that caused poor scaling. In your<br>
>>>>> case, VecScatterEnd time was doubled from 1.5575e+03 to 3.2413e+03. Its<br>
>>>>> time percent jumped from 1% to 17%. This time can contribute to the big<br>
>>>>> time ratio in MatMultAdd ant MatMultTranspose, misleading you guys thinking<br>
>>>>> there was load-imbalance computation-wise.<br>
>>>>> The reason is that I found in the interpolation and restriction<br>
>>>>> phases of gamg, the communication pattern is very bad. Few processes<br>
>>>>> communicate with hundreds of neighbors with message sizes of a few bytes.<br>
>>>>><br>
>>>><br>
>>>> We may need to truncate interpolation/restriction operators. Also do<br>
>>>> some aggressive coarsening. Unfortunately, GAMG currently does not support.<br>
>>>><br>
>>><br>
>>> Are these gamg options the truncation you thought?<br>
>>><br>
>><br>
>>> -pc_gamg_threshold[] <thresh,default=0> - Before aggregating the graph<br>
>>> GAMG will remove small values from the graph on each level<br>
>>> -pc_gamg_threshold_scale <scale,default=1> - Scaling of threshold on each<br>
>>> coarser grid if not specified<br>
>>><br>
>><br>
>> Nope. Totally different things.<br>
>><br>
><br>
> Well, you could use _threshold to do more aggressive coarsening, but not<br>
> for thinning out<br>
> the interpolation. <br>
<br>
Increasing the threshold results in slower coarsening.<br></blockquote><div><br></div><div>Hmm, I think we have to change the webpage then:</div><div><br></div><div> <a href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html">http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html</a></div><div><br></div><div>I read it the opposite way.</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Note that square_graph 10 is very unusual.<br>
<br>
> There are some simple filters we might be able to use (Luke Olson<br>
> talked about it today), but Mark is the expert.<br>
><br>
> Matt<br>
><br>
><br>
>> Fande<br>
>><br>
>><br>
>>><br>
>>><br>
>>>> Fande,<br>
>>>><br>
>>>><br>
>>>>> If we can avoid this pattern algorithmically (which I don't know), or<br>
>>>>> find ways with faster communication (which I am working), then we can get<br>
>>>>> better scalability.<br>
>>>>><br>
>>>>> --Junchao Zhang<br>
>>>>><br>
>>>>> On Thu, Jul 26, 2018 at 10:02 AM, Pierre Jolivet <<br>
>>>>> <a href="mailto:pierre.jolivet@enseeiht.fr" target="_blank">pierre.jolivet@enseeiht.fr</a>> wrote:<br>
>>>>><br>
>>>>>><br>
>>>>>><br>
>>>>>> > On 26 Jul 2018, at 4:24 PM, Karl Rupp <<a href="mailto:rupp@iue.tuwien.ac.at" target="_blank">rupp@iue.tuwien.ac.at</a>> wrote:<br>
>>>>>> ><br>
>>>>>> > Hi Pierre,<br>
>>>>>> ><br>
>>>>>> >> I’m using GAMG on a shifted Laplacian with these options:<br>
>>>>>> >> -st_fieldsplit_pressure_ksp_type preonly<br>
>>>>>> >> -st_fieldsplit_pressure_pc_composite_type additive<br>
>>>>>> >> -st_fieldsplit_pressure_pc_type composite<br>
>>>>>> >> -st_fieldsplit_pressure_sub_0_ksp_pc_type jacobi<br>
>>>>>> >> -st_fieldsplit_pressure_sub_0_pc_type ksp<br>
>>>>>> >> -st_fieldsplit_pressure_sub_1_ksp_pc_gamg_square_graph 10<br>
>>>>>> >> -st_fieldsplit_pressure_sub_1_ksp_pc_type gamg<br>
>>>>>> >> -st_fieldsplit_pressure_sub_1_pc_type ksp<br>
>>>>>> >> and I end up with the following logs on 512 (top) and 2048 (bottom)<br>
>>>>>> processes:<br>
>>>>>> >> MatMult 1577790 1.0 3.1967e+03 1.2 4.48e+12 1.6 7.6e+09<br>
>>>>>> 5.6e+03 0.0e+00 7 71 75 63 0 7 71 75 63 0 650501<br>
>>>>>> >> MatMultAdd 204786 1.0 1.3412e+02 5.5 1.50e+10 1.7 5.5e+08<br>
>>>>>> 2.7e+02 0.0e+00 0 0 5 0 0 0 0 5 0 0 50762<br>
>>>>>> >> MatMultTranspose 204786 1.0 4.6790e+01 4.3 1.50e+10 1.7 5.5e+08<br>
>>>>>> 2.7e+02 0.0e+00 0 0 5 0 0 0 0 5 0 0 145505<br>
>>>>>> >> [..]<br>
>>>>>> >> KSPSolve_FS_3 7286 1.0 7.5506e+02 1.0 9.14e+11 1.8 7.3e+09<br>
>>>>>> 1.5e+03 2.6e+05 2 14 71 16 34 2 14 71 16 34 539009<br>
>>>>>> >> MatMult 1778795 1.0 3.5511e+03 4.1 1.46e+12 1.9 4.0e+10<br>
>>>>>> 2.4e+03 0.0e+00 7 66 75 61 0 7 66 75 61 0 728371<br>
>>>>>> >> MatMultAdd 222360 1.0 2.5904e+0348.0 4.31e+09 1.9 2.4e+09<br>
>>>>>> 1.3e+02 0.0e+00 14 0 4 0 0 14 0 4 0 0 2872<br>
>>>>>> >> MatMultTranspose 222360 1.0 1.8736e+03421.8 4.31e+09 1.9 2.4e+09<br>
>>>>>> 1.3e+02 0.0e+00 0 0 4 0 0 0 0 4 0 0 3970<br>
>>>>>> >> [..]<br>
>>>>>> >> KSPSolve_FS_3 7412 1.0 2.8939e+03 1.0 2.66e+11 2.1 3.5e+10<br>
>>>>>> 6.1e+02 2.7e+05 17 11 67 14 28 17 11 67 14 28 148175<br>
>>>>>> >> MatMultAdd and MatMultTranspose (performed by GAMG) somehow ruin<br>
>>>>>> the scalability of the overall solver. The pressure space “only” has 3M<br>
>>>>>> unknowns so I’m guessing that’s why GAMG is having a hard time strong<br>
>>>>>> scaling.<br>
>>>>>> ><br>
>>>>>> > 3M unknowns divided by 512 processes implies less than 10k unknowns<br>
>>>>>> per process. It is not unusual to see strong scaling roll off at this size.<br>
>>>>>> Also note that the time per call(!) for "MatMult" is the same for both<br>
>>>>>> cases, indicating that your run into a latency-limited regime.<br>
>>>>>> ><br>
>>>>>> > Also, have a look at the time ratios: With 2048 processes,<br>
>>>>>> MatMultAdd and MatMultTranspose show a time ratio of 48 and 421,<br>
>>>>>> respectively. Maybe one of your MPI ranks is getting a huge workload?<br>
>>>>>><br>
>>>>>> Maybe inside GAMG itself (how could I check this?), but since the<br>
>>>>>> timing and ratio of the MatMult look OK and the distribution of the<br>
>>>>>> pressure space is the same as the other three fields, I’m guessing this<br>
>>>>>> does not come from my global Mat, but I may be wrong.<br>
>>>>>><br>
>>>>>> >> For the other fields, the matrix is somehow distributed nicely,<br>
>>>>>> i.e., I don’t want to change the overall distribution of the matrix.<br>
>>>>>> >> Do you have any suggestion to improve the performance of GAMG in<br>
>>>>>> that scenario? I had two ideas in mind but please correct me if I’m wrong<br>
>>>>>> or if this is not doable:<br>
>>>>>> >> 1) before setting up GAMG, first use a PCTELESCOPE to avoid having<br>
>>>>>> too many processes work on this small problem<br>
>>>>>> >> 2) have the sub_0_ and the sub_1_ work on two different<br>
>>>>>> nonoverlapping communicators of size PETSC_COMM_WORLD/2, do the solve<br>
>>>>>> concurrently, and then sum the solutions (only worth doing because of<br>
>>>>>> -pc_composite_type additive). I have no idea if this easily doable with<br>
>>>>>> PETSc command line arguments<br>
>>>>>> ><br>
>>>>>> > 1) is the more flexible approach, as you have better control over<br>
>>>>>> the system sizes after 'telescoping’.<br>
>>>>>><br>
>>>>>> Right, but the advantage of 2) is that I wouldn't have one half or<br>
>>>>>> more of processes idling and I could overlap the solves of both subpc in<br>
>>>>>> the PCCOMPOSITE.<br>
>>>>>><br>
>>>>>> I’m attaching the -log_view for both runs (I trimmed some options).<br>
>>>>>><br>
>>>>>> Thanks for your help,<br>
>>>>>> Pierre<br>
>>>>>><br>
>>>>>><br>
>>>>>><br>
>>>>>> > Best regards,<br>
>>>>>> > Karli<br>
>>>>>><br>
>>>>>><br>
>>>>>><br>
>>>>><br>
>>>><br>
>>><br>
>><br>
><br>
> -- <br>
> What most experimenters take for granted before they begin their<br>
> experiments is infinitely more interesting than any results to which their<br>
> experiments lead.<br>
> -- Norbert Wiener<br>
><br>
> <a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a> <<a href="http://www.caam.rice.edu/~mk51/" rel="noreferrer" target="_blank">http://www.caam.rice.edu/~mk51/</a>><br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.caam.rice.edu/~mk51/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div>