[MPICH] MPICH2 performance tuning and characterising

[Anthony Chan]

On Fri, 16 Mar 2007, stephen mulcahy wrote:

> stephen mulcahy wrote:
> >
> > I guess I've read the docs on MPE that I could find and I'm still not
> > clear what information it will give me and whether it's what I'm looking
> > for. I guess to best way to is to go ahead and try it.

Yes.  The most up-to-date doc on MPE are 2 files in mpe2 source tree
mpich2-xxx/src/mpe2/{README,INSTALL}.

> > It is also a little unclear as to how to build in "mpe support" - or
> > whether it is included by default.
> >
> > I've just rebuilt mpich2 1.0.5p3 with
> >
> > configure --enable-mpe --with-mpe

MPE in enabled by default in MPICH2 configure, you don't need to add
any configure options.  What you did is fine too.

> > Will that do the trick or is that unneccesary?
>
> I've built and installed mpich2 1.0.5p3 with those options enabled and
>
> find /usr/local -name *mpe*

Your find command does not show if jumpshot(i.e. slog2sdk) has been built.
Jumpshot requires java from SUN or IBM to be able to run smoothly.
Just to be sure, can you send us the configure and make outputs as seen
on the screen so to make sure that you have all pieces built, i.e.

cd <mpich2-build-dir>

for csh like shell:
<mpich2...>/configure .... |& tee c.txt
make |& tee m.txt

for bourne like shell, remove "&" but add "2>&1" before "|".

> >> To verify if the code is fairly optimized, you could enable MPE logging
> >> on your app and see if the logfile makes sense to you.  If you are using
> >> mpich2-1.0.5, you can relink your code with "mpif90 -mpe=mpilog", it will
> >> generate a clog2 file which can be "viewed" by jumpshot (which will
> >> convert it to the native format, slog2).  If your slog2 file is small
> >> enough, you could send it to us when you have questons (We always like to
> >> look at logfile, :))
>
> So I've rebuilt our app with
>
> mpif90 -mpe=mpilog
>
> and restarted it but I can't see any additional log-files generated in
> /tmp or in the directory from which the app was restarted.

For fortran program, the default logfile name is Unknown.clog2 (if you
don't set MPE_LOGFILE_PREFIX)

There are 2 ways to verify if MPE logging has been built correctly.
1) let's try it on some simpler program, e.g. cpilog.c, in
/usr/local/share/example_logging.  Can you compile cpilog by "..../mpicc
-mpe=mpilog" and run it and you should see the following message from
rank 0 in your stdout:

Writing logfile....
Enabling the Default clock synchronization...
Finished writing logfile cpilog.clog2

both mpicc and mpif90 should be from the same mpich2 install directory.

2) You can run "make installcheck" after "make install".  The output of
installcheck will show if various pieces of mpe2 are built/installed
correctly.

> Is there some additional step required or do I also need to add logging
> code to our app before I can see any log-files?

There is user-defined MPE logging you can add to your code to supplement
MPI logging, check mpich.../src/mpe2/README.

A.Chan