[petsc-users] MPI Communication times

[Manuel Valera]

Thanks for your answer, so for example i have a log for 200 cores across 10
nodes that reads:

------------------------------------------------------------------------------------------------------------------------
Event                           Count      Time (sec)     Flop
               --- Global ---  --- Stage ---   Total
                                Max Ratio  Max     Ratio   Max  Ratio
Mess   Avg len Reduct  %T %F %M %L %R  %T %F %M %L %R Mflop/s
----------------------------------------------------------------------------------------------------------------------
VecScatterBegin     3014 1.0 4.5550e+01 2.6 0.00e+00 0.0 *4.2e+06 1.1e+06
2.8e+01  4*  0 63 56  0   4  0 63 56  0     0
VecScatterEnd       2976 1.0 1.2143e+02 1.7 0.00e+00 0.0 *0.0e+00 0.0e+00
0.0e+00 14*  0  0  0  0  14  0  0  0  0     0

While for 20 nodes at one node i have:

VecScatterBegin     2596 1.0 2.9142e+01 2.1 0.00e+00 0.0 *1.2e+05 4.0e+06
3.0e+01  2*  0 81 61  0   2  0 81 61  0     0
VecScatterEnd       2558 1.0 8.0344e+01 7.9 0.00e+00 0.0  *0.0e+00 0.0e+00
0.0e+00  3*  0  0  0  0   3  0  0  0  0     0

Where do i see the max/min ratio in here? and why End step is all 0.0e00 in
both but still grows from 3% to 14% of total time? It seems i would need to
run again with the -log_sync option, is this correct?

Different question, can't i estimate the total communication time if i had
a typical communication time per MPI message times the number of MPI
messages reported in the log? or it doesn't work like that?

Thanks.





On Wed, Mar 20, 2019 at 2:02 PM Zhang, Junchao <jczhang at mcs.anl.gov> wrote:

> See the "Mess   AvgLen  Reduct" number in each log stage.  Mess is the
> total number of messages sent in an event over all processes.  AvgLen is
> average message len. Reduct is the number of global reduction.
> Each event like VecScatterBegin/End has a maximal execution time over all
> processes, and a max/min ratio.  %T is sum(execution time of the event on
> each process)/sum(execution time of the stage on each process). %T
> indicates how expensive the event is. It is a number you should pay
> attention to.
> If your code is imbalanced (i.e., with a big max/min ratio), then the
> performance number is skewed and becomes misleading because some processes
> are just waiting for others. Then, besides -log_view, you can add
> -log_sync, which adds an extra MPI_Barrier for each event to let them start
> at the same time. With that, it is easier to interpret the number.
> src/vec/vscat/examples/ex4.c is a tiny example for VecScatter logging.
>
> --Junchao Zhang
>
>
> On Wed, Mar 20, 2019 at 2:58 PM Manuel Valera via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> Hello,
>>
>> I am working on timing my model, which we made MPI scalable using petsc
>> DMDAs, i want to know more about the output log and how to calculate a
>> total communication times for my runs, so far i see we have "MPI Messages"
>> and "MPI Messages Lengths" in the log, along VecScatterEnd and
>> VecScatterBegin reports.
>>
>> My question is, how do i interpret these number to get a rough estimate
>> on how much overhead we have just from MPI communications times in my model
>> runs?
>>
>> Thanks,
>>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190320/d0a46480/attachment.html>