Performance Issues on ccNuma-System

[Matthew Knepley]

On Mon, Oct 13, 2008 at 10:08 AM, Christoph Statz
<christoph.statz at ifn.et.tu-dresden.de> wrote:
> Hello Matt and PETSc-users,
>
> 1) With any performance question, please send the output of -log_summary
>
> You'll find the output attached
> (But there is _really_ not much to see).

I will look at it.

> 2) I think it is unlikely that cache misses are responsible for this
> performance. It is
>    much more likely that bandwidth limitations are responsible.
>
> As far as I can see, there are neither bandwidth limitations nor latency
> problems (since there is an infiniband-interconnect).
> MPI-Performance (Vampirtrace + Scalasca) looks good (late senders/receivers,
> barriers etcpp.).
> PAPI-Instrumentation says: cache misses.

That stuff is rarely worth running. Without a decent model of the
performance, the
data is no help. I am not talking about network bandwidth, but memory bandwidth.
For a sparse matvec that comes from a simple scalar PDE, you need incredible
amounts of bandwidth to drive the tiny amount of flops. The equation
is in the paper.

> Please see the paper
>    by Kaushik and Gropp which models sparse matvec performance (on
> Dinesh's website).
>
> Which Paper on which website. Please send a link.

I believe you want 11 and 17 here http://www.mcs.anl.gov/~kaushik/
under the Publications link.

> 3) You would see better performance using a block method. Sparse matvec
> without
>    blocks will never see good percentages of peak (ditto for backsolve).
>
> How do I use the block methods?
> Since I rely on the "user-level" interfaces kspsolve etcpp., I don't see how
> i could influence this.

You can't unless your system has block structure. If it does, you can
use the BAIJ
matrix types.

  Matt

> You'll find basic source code attached.
> Sincerly,
> Christoph
> --
> Christoph Statz
> Institut für Nachrichtentechnik
> Technische Universität Dresden
> 01062 Dresden
> Email:  christoph.statz at mailbox.tu-dresden.de
> Phone: +49 351 463 32287
>
>
>
>
>
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which
their experiments lead.
-- Norbert Wiener