[petsc-users] Reductions in various vector and matrix kernels

[Barry Smith]

  The debug version does extra reductions for error checking, you should never look at -log_summary with a debug build.

   Normally MatMult() for MPIAIJ matrices has nearest neighbor communication so neither MatMult() or MatMultTranspose() has global reductions but if some scatters involve all entries then VecScatter does use global reductions in those cases.  You could run a slimmed down run on 2 process with one process in the debugger and put a break point in MPI_Allreduce() and MPI_Reduce() to see when it is being triggered inside the MatMultTranspose(). Use -start_in_debugger -debugger_nodes 0


   Barry

On Jan 4, 2014, at 1:32 PM, R. Oğuz Selvitopi <roguzsel at gmail.com> wrote:

> Hello,
> 
> I am trying to understand the output generated by PETSc with -log_summary option.
> 
> Using PetscLogStageRegister/PetscLogStagePush/PetscLogStagePop I want to find out if there exists unnecessary communication in my code.
> My problem is with understanding the number of reductions performed.
> 
> I have a solver whose stages are logged, and in the summary stages output, I get
> 
> Summary of Stages:   ----- Time ------  ----- Flops -----  --- Messages ---  -- Message Lengths --  -- Reductions --
>                                  Avg     %Total     Avg     %Total   counts   %Total     Avg         %Total   counts   %Total 
> 4:          Solver: 6.5625e-04   4.3%  4.2000e+02  59.7%  1.600e+01  23.2%  3.478e+00       14.5%  8.000e+00   5.3% 
> 
> where it seems I have 8 reduction operations performed. But in the details of the stage events, I get:
> 
> --- Event Stage 4: Solver
> 
> Event                Count      Time (sec)     Flops                             --- Global ---  --- Stage ---   Total
>                    Max Ratio  Max     Ratio   Max  Ratio  Mess   Avg len Reduct  %T %f %M %L %R  %T %f %M %L %R Mflop/s
> ------------------------------------------------------------------------------------------------------------------------
> MatMult                      1 1.0 1.2207e-04 1.8 4.10e+01 4.1 8.0e+00 1.5e+01 0.0e+00  1 16 12  7  0  15 27 50 50  0     1
> MatMultTranspose       1 1.0 1.2112e-04 1.0 4.60e+01 3.8 8.0e+00 1.5e+01 2.0e+00  1 18 12  7  1  18 30 50 50 25     1
> VecDot                       3 1.0 2.6989e-04 1.2 2.90e+01 2.6 0.0e+00 0.0e+00 3.0e+00  2 12  0  0  2  36 20  0  0 38     0
> VecSet                       2 1.0 8.1062e-06 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  0  0  0  0     0
> VecAXPY                   2 1.0 3.7909e-05 1.3 2.00e+01 2.0 0.0e+00 0.0e+00 0.0e+00  0  9  0  0  0   5 15  0  0  0     2
> VecAYPX                   1 1.0 5.0068e-06 1.2 1.00e+01 1.7 0.0e+00 0.0e+00 0.0e+00  0  5  0  0  0   1  8  0  0  0     6
> VecScatterBegin         2 1.0 7.2956e-05 2.4 0.00e+00 0.0 1.6e+01 1.5e+01 0.0e+00  0  0 23 14  0   6  0100100  0     0
> VecScatterEnd           2 1.0 9.5129e-05 2.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0  10  0  0  0  0     0
> 
> It seems there are only 5 reductions.
> 
> But when I detail my log stages, it shows up VecAXPY/VecAYPX operations require reductions as well (I have two VecAXPY and a single VecAYPX, so 5+3 = 8).
> (Whose logs I have not included here).
> 
> Normally these two operations should not require any reductions at all, as opposed to VecDot.
> 
> Do VecAXPY/VecAYPX require reductions? Is it because PETSc is compiled with the debugging option so that it performs additional checks that perform reductions?
> 
> Which is the correct number of reductions in above statistics, 5 or 8?
> 
> Moreover, why does MatMult require no reduction whereas MatMultTranspose requires two of them?
> 
> Thanks in advance.