Performance of MatMatSolve

[Hong Zhang]

David,

Yes, MatMatSolve dominates. Can you also send us the output of
'-log_summary' from superlu_dist?

MUMPS only suppports centralized rhs vector b.
Thus, we must scatter petsc distributed b into a seqential 
rhs vector (stored in root proc) in the petsc interface, which 
explains why the root proc takes longer time.
I see that the numerical factorization and MatMatSolve are called
30 times.
Do you iterate with the sequence similar to
  for i=0,1, ...
    B_i = X_(i-1)
    Solve A_i * X_i = B_i

i.e., the rhs B is based on previously computed X?
If this is the case, we should take sequential output X (mumps has this 
option)
and feed it into next iteration
without mpi scattering.

Hong

On Sat, 14 Mar 2009, David Fuentes wrote:

> Thanks a lot Hong,
>
> The switch definitely seemed to balance the load during the SuperLU 
> matmatsolve.
> Although I'm not completely sure what I'm seeing. Changing the #dof
> also seemed to affect the load balance of the Mumps MatMatSolve.
> I need to investigate a bit more.
>
> Looking in the profile. The majority of the time is spent in the
> MatSolve called by the MatMatSolve.
>
>
> ------------------------------------------------------------------------------------------------------------------------
> Event                Count      Time (sec)     Flops --- Global ---  --- 
> Stage ---   Total
>                   Max Ratio  Max     Ratio   Max  Ratio  Mess   Avg len 
> Reduct  %T %F %M %L %R  %T %F %M %L %R Mflop/s
> ------------------------------------------------------------------------------------------------------------------------
>
> VecCopy           135030 1.0 6.3319e-01 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 
> 0.0e+00  0   0  0  0  0   0  0  0  0  0     0
> VecWAXPY              30 1.0 1.6069e-04 1.9 4.32e+03 1.7 0.0e+00 0.0e+00 
> 0.0e+00  0   0  0  0  0   0  0  0  0  0   840
> VecScatterBegin       30 1.0 7.6072e-03 1.5 0.00e+00 0.0 4.7e+04 9.0e+02 
> 0.0e+00  0   0 15  0  0   0  0 50  0  0     0
> VecScatterEnd         30 1.0 9.1272e-02 6.8 0.00e+00 0.0 0.0e+00 0.0e+00 
> 0.0e+00  0   0  0  0  0   0  0  0  0  0     0
> MatMultAdd            30 1.0 3.3028e-01 1.4 3.89e+07 1.7 4.7e+04 9.0e+02 
> 0.0e+00  0   0 15  0  0   0  0 50  0  0  3679
> MatSolve          135030 1.0 3.0340e+03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 
> 0.0e+00 78   0  0  0  0  81  0  0  0  0     0
> MatLUFactorSym        30 1.0 2.2563e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 
> 0.0e+00  0   0  0  0  0   0  0  0  0  0     0
> MatLUFactorNum        30 1.0 2.7990e+02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 
> 0.0e+00  7   0  0  0  0   7  0  0  0  0     0
> MatConvert           150 1.0 2.9276e+00 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 
> 1.8e+02  0   0  0  0  4   0  0  0  0 30     0
> MatScale              60 1.0 2.7492e-01 1.9 1.94e+07 1.7 0.0e+00 0.0e+00 
> 0.0e+00  0   0  0  0  0   0  0  0  0  0  2210
> MatAssemblyBegin     180 1.0 1.1748e+02236.9 0.00e+00 0.0 0.0e+00 0.0e+00 
> 2.4e+02 2   0  0  0  5   2  0  0  0 40     0
> MatAssemblyEnd       180 1.0 1.9992e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 
> 2.4e+02  0   0  0  0  5   0  0  0  0 40     0
> MatGetRow           4320 1.7 2.2634e-01 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 
> 0.0e+00  0   0  0  0  0   0  0  0  0  0     0
> MatMatMult            30 1.0 4.2578e+02 1.0 1.75e+11 1.7 4.7e+04 4.0e+06 
> 2.4e+02 11 100 15 97  5  11100 50100 40 12841
> MatMatSolve           30 1.0 3.0256e+03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 
> 6.0e+01 77   0  0  0  1  81  0  0  0 10     0
>
>
>
> df
>
>
>
> On Fri, 13 Mar 2009, Hong Zhang wrote:
>
>> David,
>> 
>> You may run with option '-log_summary <log_file>' and
>> check which function dominates the time.
>> I suspect the symbolic factorization, because it is
>> implemented sequentially in mumps.
>> 
>> If this is the case, you may swich to superlu_dist
>> which supports parallel symbolic factorization
>> in the latest release.
>> 
>> Let us know what you get,
>> 
>> Hong
>> 
>> On Fri, 13 Mar 2009, David Fuentes wrote:
>> 
>>> 
>>> The majority of time in my code is spent in the MatMatSolve. I'm running 
>>> MatMatSolve in parallel using Mumps as the factored matrix.
>>> Using top, I've noticed that during the MatMatSolve
>>> the majority of the load seems to be on the root process.
>>> Is this expected? Or do I most likely have a problem with the matrices 
>>> that I'm passing in?
>>> 
>>> 
>>> 
>>> thank you,
>>> David Fuentes
>>> 
>>> 
>> 
>