<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Hi Phil,<div>Thanks for looking into this for me.</div><div><br></div><div>Here is the information you requested.</div><div><br></div><div>ldd ./ior</div><div><div> linux-vdso.so.1 => (0x00007fff0237c000)</div><div> libm.so.6 => /lib64/libm.so.6 (0x00007fce3e88f000)</div><div> libmpi.so.12 => /opt/intel/compilers_and_libraries_2016.3.223/linux/mpi/intel64/lib/libmpi.so.12 (0x00007fce3e0bf000)</div><div> libc.so.6 => /lib64/libc.so.6 (0x00007fce3dcfc000)</div><div> /lib64/ld-linux-x86-64.so.2 (0x000055de86c95000)</div><div> librt.so.1 => /lib64/librt.so.1 (0x00007fce3daf4000)</div><div> libdl.so.2 => /lib64/libdl.so.2 (0x00007fce3d8ef000)</div><div> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fce3d6d9000)</div><div> libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fce3d4bd000)</div><div>[testusera@ip-0A021004 bin]$</div></div><div><br></div><div>Here is the info on testfile.0 and testfile.1 before executing the IOR read test.</div><div><br></div><div><div>[testusera@ip-0A021004 testing]$ pwd</div><div>/mnt/beegfs/testing</div><div>[testusera@ip-0A021004 testing]$ ls -lt</div><div>total 4194304</div><div>-rw-r--r--. 1 testusera domain users 2147483648 Nov 21 22:18 testfile.1</div><div>-rw-r--r--. 1 testusera domain users 2147483648 Nov 21 22:17 testfile.0</div><div>[testusera@ip-0A021004 testing]$</div></div><div><div>[testusera@ip-0A021004 testing]$ stat testfile.0</div><div> File: ‘testfile.0’</div><div> Size: 2147483648 Blocks: 4194304 IO Block: 524288 regular file</div><div>Device: 29h/41d Inode: 2501562826468697566 Links: 1</div><div>Access: (0644/-rw-r--r--) Uid: (1264001104/testusera) Gid: (1264000513/domain users)</div><div>Context: system_u:object_r:tmp_t:s0</div><div>Access: 2018-11-21 22:17:41.000000000 +0000</div><div>Modify: 2018-11-21 22:17:56.000000000 +0000</div><div>Change: 2018-11-21 22:17:56.000000000 +0000</div><div> Birth: -</div></div><div><br></div><div>Result from running the IOR read test.</div><div><br></div><div><div>IOR-3.2.0: MPI Coordinated Test of Parallel I/O</div><div>Began : Wed Nov 21 22:22:52 2018</div><div>Command line : /avere/apps/ior/bin/ior -a POSIX -B -r -k -z -v -o /mnt/beegfs/testing/testfile -i 2 -m -t 32m -b 256M -d 1</div><div>Machine : Linux ip-0A021005</div><div>Start time skew across all tasks: 0.00 sec</div><div>TestID : 0</div><div>StartTime : Wed Nov 21 22:22:52 2018</div><div>Path : /mnt/beegfs/testing</div><div>FS : 11.8 TiB Used FS: 0.0% Inodes: 0.0 Mi Used Inodes: -nan%</div><div>Participating tasks: 8</div><div><br></div><div>Options:</div><div>api : POSIX</div><div>apiVersion :</div><div>test filename : /mnt/beegfs/testing/testfile</div><div>access : single-shared-file</div><div>type : independent</div><div>segments : 1</div><div>ordering in a file : random</div><div>ordering inter file : no tasks offsets</div><div>tasks : 8</div><div>clients per node : 8</div><div>repetitions : 2</div><div>xfersize : 32 MiB</div><div>blocksize : 256 MiB</div><div>aggregate filesize : 2 GiB</div><div><br></div><div>Results:</div><div><br></div><div>access bw(MiB/s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) total(s) iter</div><div>------ --------- ---------- --------- -------- -------- -------- -------- ----</div><div>delaying 1 seconds . . .</div><div>Commencing read performance test: Wed Nov 21 22:22:53 2018</div><div>read 481.88 262144 32768 0.020854 4.23 2.97 4.25 0</div><div>delaying 1 seconds . . .</div><div>Commencing read performance test: Wed Nov 21 22:22:58 2018</div><div>read 609.04 262144 32768 0.020336 3.34 2.06 3.36 1</div><div>Max Read: 609.04 MiB/sec (638.62 MB/sec)</div><div><br></div><div>Summary of all tests:</div><div>Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Max(OPs) Min(OPs) Mean(OPs) StdDev Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggs(MiB) API RefNum</div><div>read 609.04 481.88 545.46 63.58 19.03 15.06 17.05 1.99 3.80633 0 8 8 2 0 0 1 0 0 1 268435456 33554432 2048.0 POSIX 0</div><div>Finished : Wed Nov 21 22:23:02 2018</div></div><div><br></div><div>I also ran a test without the -B option and got the same result (no reads recorded)?</div><div><br></div><div>Thanks,</div><div>Cormac.</div><div><br></div><div><br></div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr">On Wed, Nov 21, 2018 at 1:46 PM Carns, Philip H. via Darshan-users <<a href="mailto:darshan-users@lists.mcs.anl.gov">darshan-users@lists.mcs.anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="font-family:'Segoe UI',Frutiger,'Frutiger Linotype','Dejavu Sans','Helvetica Neue',Arial,sans-serif;font-size:14px">
<div class="m_2543661479316708924hiri-body-wrapper">
<div>Hi Cormac,<br>
<br>
That's strange. The -i 2 and -m options are making IOR attempt to read two files: testfile.0 and testfile.1. Darshan is definitely aware of those files, but not only are the read counters missing but also the open counters. It only shows a stat from each
rank for the files in question.<br>
<br>
Can you confirm that both of the files exist and are big enough to accommodate the requested read volume? I wonder if IOR might stat the files up front and exit early if they aren't the correct size?<br>
<br>
Could you also share the output of "ldd ior"? I'm curious to make sure there isn't anything unusual about the libraries linked in, but usually if that were a problem you wouldn't get a log at all.<br>
<br>
Also one last idea, does the behavior change if you remove the -B (O_DIRECT) option? That shouldn't matter from Darshan's perspective, but it might not hurt to check.<br>
<br>
thanks,<br>
-Phil<br>
</div>
</div>
<div class="m_2543661479316708924hiri-extra">
<p>On 2018-11-20 19:59:55-05:00 Darshan-users wrote:</p>
<blockquote style="padding-left:10px;border-left:1px solid #ccc;margin:0">
<div>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">Hello,
<div>I recently installed Darshan 3.1.6 on Microsoft Azure VM running Centos 7.4.</div>
<div></div>
<div>I got an unexpected result using Darshan to characterize the I/O for an IOR benchmark</div>
<div>experiment. </div>
<div></div>
<div>export LD_PRELOAD=/avere/apps/spack/linux-centos7-x86_64/gcc-4.8.5/darshan-runtime-3.1.6-daohky4yevagxajjl33lwk472lcgn6g4/lib/libdarshan.so</div>
<div>
<div>BLOCKSIZE=256M</div>
<div>mpirun -np 8 ior -a POSIX -B -r -k -z -v -o $FILEPATH -i 2 -m -t 32m -b 256M -d 1 </div>
</div>
<div></div>
<div>After the job completed, a Darshan log file was created, the resulting text report (darshan-parser_ior_read_shared.out, its attached) was generated using the following command</div>
<div></div>
<div>darshan-parser --all testuser_ior_id22-20-80229-14589323501426222819_1.darshan >& darshan-parser_ior_read_shared.out</div>
<div></div>
<div>The above IOR benchmark is a read only benchmark to a shared file system, but the resulting darshan report indicates there are no READ operations?</div>
<div></div>
<div>Any ideas why the resulting Darshan resport has no read operations? (Note if I add a -w option to the above IOR benchmark (i.e do a write and read to a shared filesystem, darshan only</div>
<div>reports the writes and not reads?)</div>
<div></div>
<div>Any help would be appreciated,</div>
<div>Thanks for your support,</div>
<div>Cormac.</div>
<div></div>
<div></div>
<div></div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
_______________________________________________<br>
Darshan-users mailing list<br>
<a href="mailto:Darshan-users@lists.mcs.anl.gov" target="_blank">Darshan-users@lists.mcs.anl.gov</a><br>
<a href="https://lists.mcs.anl.gov/mailman/listinfo/darshan-users" rel="noreferrer" target="_blank">https://lists.mcs.anl.gov/mailman/listinfo/darshan-users</a><br>
</blockquote></div></div>