[MPICH] Problem when enabling ADIOI_MPE_LOGGING

[Christina Patrick]

Hi Anthony,

Thank you very much for the detailed explanation above.

I went through all the above steps and build mpich2 along with the
logging. I generated the clog and slog files and visualized them using
jumpshot. When I convert the clog to slot, it gives me the message:

timeElapsed between 1 & 2 = 299 msec
timeElapsed between 2 & 3 = 547 msec

When I logged the IDs  ADIOI_MPE_write_a and  ADIOI_MPE_write_b, it
gave me the following:
ADIOI_MPE_write_a     = 604 ADIOI_MPE_write_b     = 605

The file "src/mpi/romio/adio/common/ad_init.c" logs the write events as follows:
MPE_Describe_state( ADIOI_MPE_write_a, ADIOI_MPE_write_b, "write", "blue" );

Inside jumpshot, I can see a "write" event inside a
"MPI_File_write_at_all" (light blue block inside a dark blue block).

My questions are:
1. What are events 1, 2, 3 & 4 (I logged all the eventIDs in ad_init.c
and they range from 600 to 617. I don't have any eventIDs numbered 1,
2, 3 & 4)?
2. Can I explicity get the timing between events 604 & 605?

Thanks and Warm Regards,
Christina.

On Dec 1, 2007 1:11 AM, Anthony Chan <chan at mcs.anl.gov> wrote:
>
> Just to be clear, ADIOI_MPE_LOGGING only enables client side logging.
> Also I would think that a typical MPICH2 build with ADIOI_MPE_LOGGING
> enabled will cause compilation error (so not sure what you did).
>
> In order to get ADIOI_MPE_LOGGING to work correctly, you need to build
> mpich2 *TWICE* with second build dependent on the first one.  The main
> reason is that MPICH2 builds MPE after ROMIO/ADIO.  In order words,
> ADIOI_MPE_LOGGING can't be enabled without a valid set of MPE logging
> header files. So here are the steps that I would do,
>
> 1, untar a new copy of mpich2, say mpich2-1.0.6p1, into directory
>    <mpich2>
>
> 2, build a standard version of mpich2 with all the options that you
>    normally need using VPATH build.  You don't want to do inpath build
>    here because we need clean mpich2 source tree for the 2nd build, i.e.
>
>    mkdir <build_std>
>    cd <build_std>
>    <mpich2>/configure CC=... CXX=... F77=... F90=... \
>                       --prefix=<install_std>
>    make
>    make install
>
>    Notice we have the 1st installation of MPICH2 in <install_std>.
>
> 3, build another version of mpich2 with an extra CFLAGS that enables
>    ADIOI_MPE_LOGGING and points to a valid set of MPE logging include
>    files, i.e. -I<install_std>/include.
>
>    mkdir <build_adiolog>
>    cd <build_adiolog>
>    <mpich2>/configure CC=... CXX=... F77=... F90=...   \
>           --prefix=<install_adiolog>                   \
>           --with-file-system=pvfs2+pvfs+ufs+nfs        \
>           --with-pvfs2=<pvfs-install-dir>              \
>           CFLAGS="-DADIOI_MPE_LOGGING -I<install_std>/include"
>    make
>    make install
>
>    The 2nd installation of MPICH2 will be in <install_adiolog>.
>
> 4, now you can compile your MPIO+PVFS application with
>
>    <install_adiolog>/bin/mpicc -mpe=mpilog -c foo.c
>
>    and link as
>
>    <install_adiolog>/bin/mpicc -mpe=mpilog -o <exe> *.o
>
>    mpiXX wrappers from <install_adiolog> must be used with -mpe=mpilog
>    for linking, because libmpich.a in <install_adiolog> has references
>    to MPE symbols.
>
> 5, Now run the <exe> as usual.  This will generate clog2 file with
>    {read,write,seek,...} nested inside MPIO states.  Then invoke
>    jumpshot on the clog2, and follows the GUI.
>
> ADIOI_MPE_LOGGING isn't meant for general users, so no big effort
> has been spent to make this easy. Let me know if this works for you
> though.
>
> A.Chan
>
>
> On Fri, 30 Nov 2007, Christina Patrick wrote:
>
> > Hi Everybody,
> >
> > I want to monitor the time it takes to complete collective I/O in my
> > application. However, I also want the breakup of the times the
> > collective call spends in calling the PVFS functions as well as the
> > communication. I saw some ready made logging present in the files
> > ad_pvfs2_write.c. The calls PVFS_sys_write were logged by MPE_Log
> > event() as below (one excerpt of the code):
> >
> > -------------------------------
> > #ifdef ADIOI_MPE_LOGGING
> >                    MPE_Log_event( ADIOI_MPE_write_a, 0, NULL );
> > #endif
> >                    err_flag = PVFS_sys_write(pvfs_fs->object_ref, file_req,
> >                                              file_offsets, PVFS_BOTTOM,
> >                                              mem_req,
> >                                              &(pvfs_fs->credentials),
> >                                              &resp_io);
> > #ifdef ADIOI_MPE_LOGGING
> >                    MPE_Log_event( ADIOI_MPE_write_b, 0, NULL );
> > #endif
> > -------------------------------
> >
> > I modified the makefiles to add the debug flag ADIOI_MPE_LOGGING and
> > enabled the logging. However, when I run my application and generate
> > the clog and slog files, I cannot see these events using jumpshot. I
> > logged the event ID's when they are created in ad_init.c and they are
> > as below from a sample run:
> >
> > ADIOI_MPE_write_a     = 604 ADIOI_MPE_write_b     = 605
> >
> > What am I missing? Is there something that I need to do additionally?
> > How exactly should I profile my application and which tool should I
> > use to get the breakup of these times.
> >
> > I have never profiled MPI before. I would really appreciate any help
> > and suggestions that you could offer.
> >
> > Thanks and Regards,
> > Christina.
> >
> >
>