[Darshan-users] Darshan pre-release 2.2.4-pre4

[Phil Carns]

On 11/20/2012 12:05 PM, Rob Latham wrote:
> On Tue, Nov 20, 2012 at 11:55:52AM -0500, Phil Carns wrote:
>> darshan-2.2.4-pre4
>> =============
>> * Disable extra stat() of newly opened files by default in order
>>    to improve performance on shared files.  Reported by Yushu Yao.
>>    - prevents Darshan from reporting SIZE_AT_OPEN, FILE_ALIGNENT, and
>>      DARSHAN_FILE_NOT_ALIGNED values
>>    - features can still be enabled via --enable-stat-at-open option
> Hi Phil.  FILE_ALIGNENT is maybe one of the most important metrics for
> understanding poor GPFS write performance.
>
> ==rob
>

Hi Rob,

Rob Ross was concerned about this too.  So on GPFS systems we should be 
able to keep using stat() like we always have and collect the file 
alignment information (I'll be doing some tests next week to confirm).  
On Lustre (and other file systems) I'm not sure what to do.  stat() is 
the only sure-fire way to detect the file alignment, but it ends up 
being very costly because to answer a stat() the kernel also has to 
calculate the file size, which in turn requires contacting all servers 
and therefore generates quite a bit of overhead if every process does it 
simultaneously.  The xstat() system call would probably solve the 
problem, but it isn't widely available.

I know of at least two possible workarounds, though neither one is great:

a) If we detect that a file is being opened collectively at the MPI-IO 
(or higher) level, then rank 0 could issue a stat() on behalf of all 
processes and broadcast the result.  We would still have to skip the 
stat in other cases though, because without seeing a collective call we 
don't have any way to know how many ranks might open the file.  Its also 
unfortunate that we would have to broadcast the stat result, since as a 
rule we've generally avoided instigating communication within Darshan 
while the application is running.  If we defer aggregating the alignment 
until shutdown then the other N-1 processes wouldn't be able to count 
the number of unaligned accesses.

b) We could use stat() information collected from the mount point when 
Darshan starts up.  The problem with that approach is that the caller 
could always change the striping parameters for a file or directory so 
that they no longer match the mount point settings.  I would be nervous 
about mis-reporting an alignment problem if the application were using 
custom file settings.  If we went this route I would at least want to 
somehow note in the log whether the alignment data is reliable or if it 
is an estimate.

-Phil