<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="font-family: 'Segoe UI',Frutiger,'Frutiger Linotype','Dejavu Sans','Helvetica Neue',Arial,sans-serif; font-size: 14px;">
<div class="hiri-body-wrapper" contenteditable="true">
<div>I just glanced through the latest IOR revision in git and realized that it has some conditional logic for beegfs that will use a beegfs-specific library for some calls.<br>
<br>
I'm not sure if that is being triggered in your case or not, but that's a possibility. Darshan doesn't have wrappers for that API.<br>
<br>
Could you check to see if HAVE_BEEGFS_BEEGFS_H is set in your src/config.h file?<br>
<br>
thanks,<br>
-Phil<br>
<br>
</div>
</div>
<div class="hiri-extra" contenteditable="false">
<p>On 2018-11-26 15:50:10-05:00 Darshan-users wrote:</p>
<blockquote style="padding-left:10px; border-left:1px solid #ccc; margin:0">
<div>
<div class="hiri-body-wrapper" contenteditable="true">
<div>Thanks Cormac. I don't see any obvious problems from your output. We'll see if we can reproduce here.<br>
<br>
In the mean time if you want to debug further, you could try running IOR as a single process (no mpirun or mpiexec) through strace to see if there is anything unusual in the file open path.<br>
<br>
thanks,<br>
-Phil<br>
</div>
</div>
<div class="hiri-extra" contenteditable="false">
<p>On 2018-11-21 17:37:50-05:00 Cormac Garvey wrote:</p>
<blockquote style="padding-left:10px; border-left:1px solid #ccc; margin:0">
<div>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">Hi Phil,
<div>Thanks for looking into this for me.</div>
<div></div>
<div>Here is the information you requested.</div>
<div></div>
<div>ldd ./ior</div>
<div>
<div> linux-vdso.so.1 => (0x00007fff0237c000)</div>
<div> libm.so.6 => /lib64/libm.so.6 (0x00007fce3e88f000)</div>
<div> libmpi.so.12 => /opt/intel/compilers_and_libraries_2016.3.223/linux/mpi/intel64/lib/libmpi.so.12 (0x00007fce3e0bf000)</div>
<div> libc.so.6 => /lib64/libc.so.6 (0x00007fce3dcfc000)</div>
<div> /lib64/ld-linux-x86-64.so.2 (0x000055de86c95000)</div>
<div> librt.so.1 => /lib64/librt.so.1 (0x00007fce3daf4000)</div>
<div> libdl.so.2 => /lib64/libdl.so.2 (0x00007fce3d8ef000)</div>
<div> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fce3d6d9000)</div>
<div> libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fce3d4bd000)</div>
<div>[testusera@ip-0A021004 bin]$</div>
</div>
<div></div>
<div>Here is the info on testfile.0 and testfile.1 before executing the IOR read test.</div>
<div></div>
<div>
<div>[testusera@ip-0A021004 testing]$ pwd</div>
<div>/mnt/beegfs/testing</div>
<div>[testusera@ip-0A021004 testing]$ ls -lt</div>
<div>total 4194304</div>
<div>-rw-r--r--. 1 testusera domain users 2147483648 Nov 21 22:18 testfile.1</div>
<div>-rw-r--r--. 1 testusera domain users 2147483648 Nov 21 22:17 testfile.0</div>
<div>[testusera@ip-0A021004 testing]$</div>
</div>
<div>
<div>[testusera@ip-0A021004 testing]$ stat testfile.0</div>
<div> File: ‘testfile.0’</div>
<div> Size: 2147483648 Blocks: 4194304 IO Block: 524288 regular file</div>
<div>Device: 29h/41d Inode: 2501562826468697566 Links: 1</div>
<div>Access: (0644/-rw-r--r--) Uid: (1264001104/testusera) Gid: (1264000513/domain users)</div>
<div>Context: system_u:object_r:tmp_t:s0</div>
<div>Access: 2018-11-21 22:17:41.000000000 +0000</div>
<div>Modify: 2018-11-21 22:17:56.000000000 +0000</div>
<div>Change: 2018-11-21 22:17:56.000000000 +0000</div>
<div> Birth: -</div>
</div>
<div></div>
<div>Result from running the IOR read test.</div>
<div></div>
<div>
<div>IOR-3.2.0: MPI Coordinated Test of Parallel I/O</div>
<div>Began : Wed Nov 21 22:22:52 2018</div>
<div>Command line : /avere/apps/ior/bin/ior -a POSIX -B -r -k -z -v -o /mnt/beegfs/testing/testfile -i 2 -m -t 32m -b 256M -d 1</div>
<div>Machine : Linux ip-0A021005</div>
<div>Start time skew across all tasks: 0.00 sec</div>
<div>TestID : 0</div>
<div>StartTime : Wed Nov 21 22:22:52 2018</div>
<div>Path : /mnt/beegfs/testing</div>
<div>FS : 11.8 TiB Used FS: 0.0% Inodes: 0.0 Mi Used Inodes: -nan%</div>
<div>Participating tasks: 8</div>
<div></div>
<div>Options:</div>
<div>api : POSIX</div>
<div>apiVersion :</div>
<div>test filename : /mnt/beegfs/testing/testfile</div>
<div>access : single-shared-file</div>
<div>type : independent</div>
<div>segments : 1</div>
<div>ordering in a file : random</div>
<div>ordering inter file : no tasks offsets</div>
<div>tasks : 8</div>
<div>clients per node : 8</div>
<div>repetitions : 2</div>
<div>xfersize : 32 MiB</div>
<div>blocksize : 256 MiB</div>
<div>aggregate filesize : 2 GiB</div>
<div></div>
<div>Results:</div>
<div></div>
<div>access bw(MiB/s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) total(s) iter</div>
<div>------ --------- ---------- --------- -------- -------- -------- -------- ----</div>
<div>delaying 1 seconds . . .</div>
<div>Commencing read performance test: Wed Nov 21 22:22:53 2018</div>
<div>read 481.88 262144 32768 0.020854 4.23 2.97 4.25 0</div>
<div>delaying 1 seconds . . .</div>
<div>Commencing read performance test: Wed Nov 21 22:22:58 2018</div>
<div>read 609.04 262144 32768 0.020336 3.34 2.06 3.36 1</div>
<div>Max Read: 609.04 MiB/sec (638.62 MB/sec)</div>
<div></div>
<div>Summary of all tests:</div>
<div>Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Max(OPs) Min(OPs) Mean(OPs) StdDev Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggs(MiB) API RefNum</div>
<div>read 609.04 481.88 545.46 63.58 19.03 15.06 17.05 1.99 3.80633 0 8 8 2 0 0 1 0 0 1 268435456 33554432 2048.0 POSIX 0</div>
<div>Finished : Wed Nov 21 22:23:02 2018</div>
</div>
<div></div>
<div>I also ran a test without the -B option and got the same result (no reads recorded)?</div>
<div></div>
<div>Thanks,</div>
<div>Cormac.</div>
<div></div>
<div></div>
</div>
</div>
</div>
</div>
<div class="gmail_quote">
<div dir="ltr">On Wed, Nov 21, 2018 at 1:46 PM Carns, Philip H. via Darshan-users <<a href="mailto:darshan-users@lists.mcs.anl.gov">darshan-users@lists.mcs.anl.gov</a>> wrote:</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="font-family:'Segoe UI',Frutiger,'Frutiger Linotype','Dejavu Sans','Helvetica Neue',Arial,sans-serif;font-size:14px">
<div class="m_2543661479316708924hiri-body-wrapper">
<div>Hi Cormac,<br>
<br>
That's strange. The -i 2 and -m options are making IOR attempt to read two files: testfile.0 and testfile.1. Darshan is definitely aware of those files, but not only are the read counters missing but also the open counters. It only shows a stat from each
rank for the files in question.<br>
<br>
Can you confirm that both of the files exist and are big enough to accommodate the requested read volume? I wonder if IOR might stat the files up front and exit early if they aren't the correct size?<br>
<br>
Could you also share the output of "ldd ior"? I'm curious to make sure there isn't anything unusual about the libraries linked in, but usually if that were a problem you wouldn't get a log at all.<br>
<br>
Also one last idea, does the behavior change if you remove the -B (O_DIRECT) option? That shouldn't matter from Darshan's perspective, but it might not hurt to check.<br>
<br>
thanks,<br>
-Phil<br>
</div>
</div>
<div class="m_2543661479316708924hiri-extra">
<p>On 2018-11-20 19:59:55-05:00 Darshan-users wrote:</p>
<blockquote style="padding-left:10px;border-left:1px solid #ccc;margin:0">
<div>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">Hello,
<div>I recently installed Darshan 3.1.6 on Microsoft Azure VM running Centos 7.4.</div>
<div></div>
<div>I got an unexpected result using Darshan to characterize the I/O for an IOR benchmark</div>
<div>experiment. </div>
<div></div>
<div>export LD_PRELOAD=/avere/apps/spack/linux-centos7-x86_64/gcc-4.8.5/darshan-runtime-3.1.6-daohky4yevagxajjl33lwk472lcgn6g4/lib/libdarshan.so</div>
<div>
<div>BLOCKSIZE=256M</div>
<div>mpirun -np 8 ior -a POSIX -B -r -k -z -v -o $FILEPATH -i 2 -m -t 32m -b 256M -d 1 </div>
</div>
<div></div>
<div>After the job completed, a Darshan log file was created, the resulting text report (darshan-parser_ior_read_shared.out, its attached) was generated using the following command</div>
<div></div>
<div>darshan-parser --all testuser_ior_id22-20-80229-14589323501426222819_1.darshan >& darshan-parser_ior_read_shared.out</div>
<div></div>
<div>The above IOR benchmark is a read only benchmark to a shared file system, but the resulting darshan report indicates there are no READ operations?</div>
<div></div>
<div>Any ideas why the resulting Darshan resport has no read operations? (Note if I add a -w option to the above IOR benchmark (i.e do a write and read to a shared filesystem, darshan only</div>
<div>reports the writes and not reads?)</div>
<div></div>
<div>Any help would be appreciated,</div>
<div>Thanks for your support,</div>
<div>Cormac.</div>
<div></div>
<div></div>
<div></div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
_______________________________________________<br>
Darshan-users mailing list<br>
<a target="_blank" href="mailto:Darshan-users@lists.mcs.anl.gov">Darshan-users@lists.mcs.anl.gov</a><br>
<a rel="noreferrer" target="_blank" href="https://lists.mcs.anl.gov/mailman/listinfo/darshan-users">https://lists.mcs.anl.gov/mailman/listinfo/darshan-users</a></blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</body>
</html>