[petsc-dev] [petsc-users] GAMG setup scalability

[Hong Zhang]

John:

Thanks for reporting your experience with petsc mat-mat operations,
which are consistent with current status of these operations.
Petsc mat-mat operations were written many years ago for users
who wanted these operations for convenience on small size problems, not
intended for huge size and large number of cores.

As Mark started using these operations on 10k+ cores for gamg, I spent
several weeks reworking on some of these operations.
See attached slides on current status of these operations. In summary:

1. C=A*B were rewritten with two versions:
  a) fast but not memory scalable: default
  b) slow but scalable: use option '-matmatmult_scalable'
  I have a rough idea to merge these two version to a single one
  that is fast and scalable, but need time to work on it :-(

2. SPARSE inner-product is more expensive than sparse outer-product. As you
observed, C=A*B^t is worse than C=A*B. Reading your data, I'm surprised
that your C=A*B^t is so much slower than PtAP. How large is your matrices?
Can you run your np=1 case with option
'-matmattrnsmult_color' and send us the output of -log_summary?

3. Current PtAP is a simple cleanup of old version, thus it needs update
- has been on my ToDo for almost a year :-(
According to the analysis (see slides), PtAP should only take 2x of
time(A*B), but it is taking 8x A*B on 14k cores (see slide 8).

Improving this operations is on the top of our ToDo list.

Hong


On Fri, Aug 17, 2012 at 11:27 AM, Mark F. Adams <mark.adams at columbia.edu>
> wrote:
> > John: could you perhaps test MatMatMult and MatMatTransposeMult so we
> could
> > get some concrete numbers?
>
> For MatMatTransposeMult I get this error:
>
> [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [0]PETSC ERROR: No support for this operation for this object type!
> [0]PETSC ERROR: MatMatTransposeMult not supported for A of type mpiaij!
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Development HG revision:
> f9c6cac2d69c724a2258d4e0dc2f51b0825aa875  HG Date: Thu Aug 16 08:37:18
> 2012 -0700
>
> For MatMatMult, my problem is not symmetric.  It doesn't look like
> MatTranspose() followed by MatMatMult is a viable option.
>
> John
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20120820/27f78e12/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: SIAMLA2012_matops.pdf
Type: application/pdf
Size: 950742 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20120820/27f78e12/attachment.pdf>