[petsc-users] Smaller assemble time with increasing processors

[Barry Smith]

   You cannot look just at the VecAssemblyEnd() time, that will very likely give the wrong impression of the total time it takes to put the values in.

   You need to register a new Event and put a PetscLogEvent() just before you start generating the vector entries and calling VecSetValues() and put the PetscLogEventEnd() just after the VecAssemblyEnd() this is the only way to get an accurate accounting of the time.

  Barry


> On Jun 30, 2023, at 11:21 AM, Runfeng Jin <jsfaraway at gmail.com> wrote:
> 
> Hello!
> 
> When I use PETSc build a sbaij matrix, I find a strange thing. When I increase the number of processors, the assemble time become smaller. All these are totally same matrix. The assemble time mainly arouse from message passing, which because I use dynamic workload that it is random for which elements are computed by which processor.
> But from instinct, if use more processors, then more possible that the processor computes elements storing in other processors. But from the output of log_view, It seems when use more processors, the processors compute more elements storing in its local(infer from that, with more processors, less total amount of passed messages).
> 
> What could cause this happened? Thank you!
> 
> 
>  Following is the output of log_view for 64\128\256 processors. Every row is time profiler of VecAssemblyEnd.
> 
> ------------------------------------------------------------------------------------------------------------------------
> processors                Count                      Time (sec)                                      Flop                                                               --- Global ---                               --- Stage ----                Total
>                               Max    Ratio         Max                  Ratio                 Max  Ratio      Mess        AvgLen         Reduct               %T %F %M %L %R         %T %F %M %L %R       Mflop/s
> 64                            1     1.0            2.3775e+02      1.0                   0.00e+00 0.0      6.2e+03    2.3e+04     9.0e+00                 52  0      1    1    2             52   0    1      1     2             0
> 128                          1     1.0            6.9945e+01      1.0                   0.00e+00 0.0      2.5e+04    1.1e+04     9.0e+00                30   0      1     1  2              30   0    1       1    2             0
> 256                          1     1.0           1.7445e+01        1.0                  0.00e+00 0.0      9.9e+04     5.2e+03    9.0e+00                10   0      1     1  2              10   0    1        1   2             0
> 
> Runfeng Jin