[Darshan-users] Darshan & EPCC benchio different behaviour

[Carns, Philip H.]

Sure.  Here is a quick explanation of each of these:

  *   MPIIO_F_WRITE_TIME: cumulative amount of time consumed by all MPIIO write functions
  *   MPIIO_F_MAX_WRITE_TIME: the amount of time consumed by the one slowest MPIIO write call
  *   MPIIO_F_WRITE_START_TIMESTAMP: the timestamp (relative to when MPI_Init() was called) that the very first write issued began
  *   MPIIO_F_WRITE_END_TIMESTAMP: the timestamp (relative to when MPI_Init() was called) of the very last write completing

In your example these counters are probably representing all processes (you can tell if the output shows the rank as -1).   That's why, for example, the cumulative time (the first counter) is so much later than the interval between the start/end timestamps.  The former is being summed across all ranks.

thanks,
-Phil

________________________________
From: Piero LANUCARA <p.lanucara at cineca.it>
Sent: Thursday, February 13, 2020 10:21 AM
To: Carns, Philip H. <carns at mcs.anl.gov>; Snyder, Shane <ssnyder at mcs.anl.gov>; Harms, Kevin <harms at alcf.anl.gov>
Cc: darshan-users at lists.mcs.anl.gov <darshan-users at lists.mcs.anl.gov>
Subject: Re: [Darshan-users] Darshan & EPCC benchio different behaviour


Hi Phil

totally agree.

I really appreciated your support very much.

P.S is it possible to undestand at a more detailed level all the parser output?

I still have some problem while trying to relate some of those values. For example, running MPIIO version, the following appears:

1) MPIIO_F_WRITE_TIME      103.772865

2)MPIIO_F_MAX_WRITE_TIME  3.107764

3)MPIIO_F_WRITE_START_TIMESTAMP   22.217328

 4) MPIIO_F_CLOSE_START_TIMESTAMP   25.707373

so, is difficult to relate one with two or one with three, four values.

thanks again

Piero

Il 13/02/2020 14:18, Carns, Philip H. ha scritto:
Thanks Piero.  I agree, but from a practical point of view, I don't see many options for improving Darshan's handling of this particular scenario.  If something in the Fortran runtime is impacting perceived I/O performance, then the only way to observe it would be to wrap/instrument at the Fortran level rather than at the system library (libc) level.  Otherwise Darshan can't tell the difference between the Fortran calls being slow or the Fortran calls being fast with the app doing something else in between calls.

We've been reluctant to pursue that approach (we've hit cases before where instrumenting the Fortran level would have been helpful) because of the development/maintenance cost, in part because there is so much variety in the Fortran compiler world, and in part because our team simply doesn't have a lot of Fortran expertise.

That said, we would entertain a contribution along these lines ��  Something like that could be enabled as a compile-time option.

The good news (as you've seen with the MPI tests from benchio) is that what you are describing isn't really a problem for the MPI-IO interface.  Most of the existing MPI-IO Fortran bindings map almost directly to the underlying C MPI-IO bindings, meaning that what we measure at that level should be a pretty accurate indication of what's going on at the Fortran level.  The scenario you have hit is an issue because the Fortran I/O calls likely have more logic that actually resides in the Fortran runtime itself.

thanks,
-Phil
________________________________
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/darshan-users/attachments/20200213/9bf1f853/attachment-0001.html>