Tracing MPI_File_write_all from pnetcdf-1.1.1 --> mpich2 --> pvfs2

[Rob Latham]

On Wed, Feb 24, 2010 at 03:05:35PM -0500, Shawn Kim wrote:
> Hi,
> 
> I am now trying to trace from the Flash IO benchmark to the PVFS2. To do so,
> I modified pnetcdf, mpich2, and pvfs2.

> I believe that Parallel netCDF runs as a library between Flash IO
> benchmark and PVFS2. So after processing pnetCDF requests from Flash
> IO, it passes the parallel IO requests down to MPI-IO library, and
> finally PVFS2 servers receive the MPI-IO requests from PVFS2 clients
> and perform IO.

Sounds like you've got the right idea, yup.

> I've found that the function call sequence is that:
> 
> 1. ncmpi_put_vara_all (parallel-netcdf-1.1.1/src/lib/mpinetcdf.c)
> 2. MPI_File_write_all (mpich2-1.0.8/src/mpi/romio/mpi-io/write_all.c)
> 3. MPIOI_File_write_all (mpich2-1.0.8/src/mpi/romio/mpi-io/write_all.c)
> 4. ADIO_WriteStridedColl
> 5. ADIOI_GEN_WriteStridedColl (mpich2-1.0.8/src/mpi/romio/adio/common/ad_write_coll.c)
> 6. ADIO_WriteContig (???)
> 7. ADIOI_PVFS2_WriteContig
> Finally. PVFS_sys_write()


> In ADIOI_GEN_WriteStridedColl, the second ADIO_WriteContig is called as I
> marked above when MPI_FIle_write_all() is issued. Finally, PVFS_sys_write in
> ADIOI_PVFS2_WriteContig is issued and then PVFS2 client is able to know this
> call is what I want to trace.
> 
> Unfortunately, I can't see that MPI_File_write_all in ncmpi_put_vara_all,
> parallel-netcdf-1.1.1 actually calls ADIO_WriteContig. Because call id can't
> be passed from MPI library to PVFS2, I can't trace this call.
> To make sure that this is the case, I ran the simple program which uses the
> same MPI_File_write_all and can see what I want, but I can't see it when I
> ran Flash IO benchmark.
> 
> Is there anybody who help me or has the same situation?

I'm not sure what you are asking here.  You understand the call stack.
Are you having trouble seeing the call stack in a debugger?  Is it
possible that you've built libraries with optimization such that
functions have been inlined?

You could instrument parallel-netcdf with MPE calls and use a tool
like jumpshot to display the interaction between pnetcdf calls and
MPI-IO calls.  

==rob

-- 
Rob Latham
Mathematics and Computer Science Division
Argonne National Lab, IL USA