[Darshan-users] Files listed twice in darshan output

[Phil Carns]

Hi Ed,

I was thrown off by the extra column from nl at first, but you aren't 
supposed to get two records with identical ranks (-1 in this case, 
meaning the stats are shared by all ranks) and file name hashes 
(14116...).  As you observed in your test program, they should have 
merged into one record.

For completeness, it is definitely legal to see the same file show up 
twice for *different* ranks.  For example, if MPI ranks 0 and 5 both 
opened atomselect.tcl , but no one else did, then you would see two 
entries.  One for rank 0 and one for rank 5.  Darshan only collapses 
shared files into an aggregate record when everyone opened the same 
file.  If a subset of ranks opened it then they remain separate records.

In summary it looks like that log is broken for some reason, but I would 
suggest logging that it happened if you are loading a lot of logs into a 
data base at once to make sure it's not a widespread problem.

thanks,
-Phil

On 03/27/2018 12:58 PM, Ed Karrels wrote:
> Is it normal for a file to be listed more than once in darshan output 
> for the same process rank?
>
> I've got a logfile (generated by runtime version 2.3.0) for which 
> darshan-parser (version 2.3.1) outputs stats for a few of the accessed 
> files twice.
>
> $ darshan-parser 
> stonej_vmd_BLUEWATERS_id1340701_1-19-39124-18216319424980466532_1.darshan.gz 
> | nl | egrep '41554.CP_F_(OPEN|CLOSE)_TIMESTAMP'
>    388  -1      14116071308113141554 CP_F_OPEN_TIMESTAMP     0.314585 
> .../atomselect.tcl      /mnt/a  lustre
>    391  -1      14116071308113141554 CP_F_CLOSE_TIMESTAMP    0.363727 
> .../atomselect.tcl      /mnt/a  lustre
>  30034  -1      14116071308113141554 CP_F_OPEN_TIMESTAMP     0.398742 
> .../atomselect.tcl      /mnt/a  lustre
>  30037  -1      14116071308113141554 CP_F_CLOSE_TIMESTAMP    0.399672 
> .../atomselect.tcl      /mnt/a  lustre
>
> It looks like the file was opened and closed twice, but when I tried 
> this in a test program with darshan runtime and util version 2.3.1, 
> the accesses were combined into one set of records in Darshan. I'm 
> wondering if this is normal and I should deal with it when loading 
> darshan data into a database, or if this file is flawed and I can 
> ignore it.
>
> thanks,
> Ed
>
>
>
> _______________________________________________
> Darshan-users mailing list
> Darshan-users at lists.mcs.anl.gov
> https://lists.mcs.anl.gov/mailman/listinfo/darshan-users


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/darshan-users/attachments/20180329/82c64f90/attachment.html>