<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="font-family: 'Segoe UI',Frutiger,'Frutiger Linotype','Dejavu Sans','Helvetica Neue',Arial,sans-serif; font-size: 14px;">
<div class="hiri-body-wrapper" contenteditable="true">
<div>Here is a PR that fixes the issue and makes the posix read counters show up properly in IOR with Darshan preloaded:<br>
<br>
<a href="https://xgitlab.cels.anl.gov/darshan/darshan/merge_requests/28" _src="https://xgitlab.cels.anl.gov/darshan/darshan/merge_requests/28">https://xgitlab.cels.anl.gov/darshan/darshan/merge_requests/28</a><br>
<br>
It's a little frustrating to hit cases like this where glibc is using a different symbol that we expect, but I'm not sure what else to do about it.<br>
<br>
Thanks for the detailed bug report!<br>
<br>
-Phil<br>
</div>
</div>
<div class="hiri-extra" contenteditable="false">
<p>On 2018-11-27 08:56:07-05:00 Carns, Philip H. wrote:</p>
<blockquote style="padding-left:10px; border-left:1px solid #ccc; margin:0">
<div>
<div class="hiri-body-wrapper" contenteditable="true">
<div><br>
Thanks for being patient :)<br>
<br>
It turns out that I can reproduce this on Ubuntu 18.10 as well. It looks like the problem is that ior is getting to the glibc open call via a path that we aren't wrapping. Here's the stack trace from breaking on open() in ior with darshan preloaded:<br>
<div>Breakpoint 1, __libc_open64 (file=file@entry=0x7fffffffc490 "test.0", </div>
<div> oflag=16386) at ../sysdeps/unix/sysv/linux/open64.c:37</div>
<div>37<span class="Apple-tab-span" style="white-space:pre"> </span>in ../sysdeps/unix/sysv/linux/open64.c</div>
<div>(gdb) where</div>
<div>#0 __libc_open64 (file=file@entry=0x7fffffffc490 "test.0", oflag=16386)</div>
<div> at ../sysdeps/unix/sysv/linux/open64.c:37</div>
<div>#1 0x00007ffff79aac1b in __open_2 (file=file@entry=0x7fffffffc490 "test.0", </div>
<div> oflag=<optimized out>) at open_2.c:28</div>
<div>#2 0x000055555555a5a0 in open (__oflag=<optimized out>, </div>
<div> __path=0x7fffffffc490 "test.0")</div>
<div> at /usr/include/x86_64-linux-gnu/bits/fcntl2.h:57</div>
<div>#3 POSIX_Open (testFileName=0x7fffffffc490 "test.0", param=0x555555574340)</div>
<div> at ../../src/aiori-POSIX.c:386</div>
<div>#4 0x000055555556092e in TestIoSys (test=test@entry=0x555555574340)</div>
<div> at ../../src/ior.c:1459</div>
<div>#5 0x00005555555621b8 in ior_main (argc=<optimized out>, argv=<optimized out>)</div>
<div> at ../../src/ior.c:148</div>
<div>#6 0x00007ffff78c509b in __libc_start_main (main=0x5555555577e0 <main>, </div>
<div> argc=19, argv=0x7fffffffcc08, init=<optimized out>, fini=<optimized out>, </div>
<div> rtld_fini=<optimized out>, stack_end=0x7fffffffcbf8)</div>
<div> at ../csu/libc-start.c:308</div>
<div>#7 0x000055555555781a in _start ()</div>
</div>
<div><br>
Notice that the open call is defined in a header (fcntl2.h) and is actually calling __open_2() instead in the transition to the glibc shared library.<br>
<br>
Here are the relevant lines from fcntl2.h on my machine:<br>
<div>__fortify_function int</div>
<div>open (const char *__path, int __oflag, ...)</div>
<div>{</div>
<div> if (__va_arg_pack_len () > 1)</div>
<div> __open_too_many_args ();</div>
<div></div>
<div> if (__builtin_constant_p (__oflag))</div>
<div> {</div>
<div> if (__OPEN_NEEDS_MODE (__oflag) && __va_arg_pack_len () < 1)</div>
<div> {</div>
<div> __open_missing_mode ();</div>
<div> return __open_2 (__path, __oflag);</div>
<div> }</div>
<div> return __open_alias (__path, __oflag, __va_arg_pack ());</div>
<div> }</div>
<div></div>
<div> if (__va_arg_pack_len () < 1)</div>
<div> return __open_2 (__path, __oflag);</div>
<div></div>
<div> return __open_alias (__path, __oflag, __va_arg_pack ());</div>
<div>}</div>
</div>
<div><br>
I haven't tracked down why we are getting here (there are a number of compile time conditionals that can alter the open() path), or what to do about it yet. I just wanted to confirm that it's been reproduced on another system.<br>
<br>
thanks,<br>
-Phil</div>
</div>
<div class="hiri-extra" contenteditable="false">
<p>On 2018-11-26 19:36:18-05:00 Cormac Garvey wrote:</p>
<blockquote style="padding-left:10px; border-left:1px solid #ccc; margin:0">
<div>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div>Hi Phil,</div>
<div>Here is the information you requested. (See attached files)</div>
<div></div>
<div>strace_ior_write.log - Running strace on a single ior process (no mpirun) (write file) </div>
<div>strace_ior_read.log - Running strace on a single ior process (no mpirun) (read files generated in write step)</div>
<div></div>
<div>I do not have the original build environment for the ior binary I am using, but I am sure beegfs was not enabled in the build, I did not explicitly enable it and I recall during the build process it could not find the necessary beegfs header files. I have
attached the output from running nm on the ior executable (nm ior >& nm_ior.log) instead.</div>
<div></div>
<div>Also </div>
<div>ldd /avere/apps/ior/bin/ior</div>
<div>
<div> linux-vdso.so.1 => (0x00007ffd26b7e000)</div>
<div> libm.so.6 => /lib64/libm.so.6 (0x00007f981c0a7000)</div>
<div> libmpi.so.12 => /opt/intel/compilers_and_libraries_2016.3.223/linux/mpi/intel64/lib/libmpi.so.12 (0x00007f981b8d7000)</div>
<div> libc.so.6 => /lib64/libc.so.6 (0x00007f981b514000)</div>
<div> /lib64/ld-linux-x86-64.so.2 (0x000055a13c2f4000)</div>
<div> librt.so.1 => /lib64/librt.so.1 (0x00007f981b30c000)</div>
<div> libdl.so.2 => /lib64/libdl.so.2 (0x00007f981b107000)</div>
<div> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f981aef1000)</div>
<div> libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f981acd5000)</div>
</div>
<div></div>
<div>Thanks for your support,</div>
<div>Cormac.</div>
</div>
</div>
</div>
<div class="gmail_quote">
<div dir="ltr">On Mon, Nov 26, 2018 at 2:16 PM Carns, Philip H. <<a href="mailto:carns@mcs.anl.gov">carns@mcs.anl.gov</a>> wrote:</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="font-family:'Segoe UI',Frutiger,'Frutiger Linotype','Dejavu Sans','Helvetica Neue',Arial,sans-serif;font-size:14px">
<div class="m_-4332855151699852336hiri-body-wrapper">
<div>I just glanced through the latest IOR revision in git and realized that it has some conditional logic for beegfs that will use a beegfs-specific library for some calls.<br>
<br>
I'm not sure if that is being triggered in your case or not, but that's a possibility. Darshan doesn't have wrappers for that API.<br>
<br>
Could you check to see if HAVE_BEEGFS_BEEGFS_H is set in your src/config.h file?<br>
<br>
thanks,<br>
-Phil<br>
<br>
</div>
</div>
<div class="m_-4332855151699852336hiri-extra">
<p>On 2018-11-26 15:50:10-05:00 Darshan-users wrote:</p>
<blockquote style="padding-left:10px;border-left:1px solid #ccc;margin:0">
<div>
<div class="m_-4332855151699852336hiri-body-wrapper">
<div>Thanks Cormac. I don't see any obvious problems from your output. We'll see if we can reproduce here.<br>
<br>
In the mean time if you want to debug further, you could try running IOR as a single process (no mpirun or mpiexec) through strace to see if there is anything unusual in the file open path.<br>
<br>
thanks,<br>
-Phil<br>
</div>
</div>
<div class="m_-4332855151699852336hiri-extra">
<p>On 2018-11-21 17:37:50-05:00 Cormac Garvey wrote:</p>
<blockquote style="padding-left:10px;border-left:1px solid #ccc;margin:0">
<div>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">Hi Phil,
<div>Thanks for looking into this for me.</div>
<div></div>
<div>Here is the information you requested.</div>
<div></div>
<div>ldd ./ior</div>
<div>
<div> linux-vdso.so.1 => (0x00007fff0237c000)</div>
<div> libm.so.6 => /lib64/libm.so.6 (0x00007fce3e88f000)</div>
<div> libmpi.so.12 => /opt/intel/compilers_and_libraries_2016.3.223/linux/mpi/intel64/lib/libmpi.so.12 (0x00007fce3e0bf000)</div>
<div> libc.so.6 => /lib64/libc.so.6 (0x00007fce3dcfc000)</div>
<div> /lib64/ld-linux-x86-64.so.2 (0x000055de86c95000)</div>
<div> librt.so.1 => /lib64/librt.so.1 (0x00007fce3daf4000)</div>
<div> libdl.so.2 => /lib64/libdl.so.2 (0x00007fce3d8ef000)</div>
<div> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fce3d6d9000)</div>
<div> libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fce3d4bd000)</div>
<div>[testusera@ip-0A021004 bin]$</div>
</div>
<div></div>
<div>Here is the info on testfile.0 and testfile.1 before executing the IOR read test.</div>
<div></div>
<div>
<div>[testusera@ip-0A021004 testing]$ pwd</div>
<div>/mnt/beegfs/testing</div>
<div>[testusera@ip-0A021004 testing]$ ls -lt</div>
<div>total 4194304</div>
<div>-rw-r--r--. 1 testusera domain users 2147483648 Nov 21 22:18 testfile.1</div>
<div>-rw-r--r--. 1 testusera domain users 2147483648 Nov 21 22:17 testfile.0</div>
<div>[testusera@ip-0A021004 testing]$</div>
</div>
<div>
<div>[testusera@ip-0A021004 testing]$ stat testfile.0</div>
<div> File: ‘testfile.0’</div>
<div> Size: 2147483648 Blocks: 4194304 IO Block: 524288 regular file</div>
<div>Device: 29h/41d Inode: 2501562826468697566 Links: 1</div>
<div>Access: (0644/-rw-r--r--) Uid: (1264001104/testusera) Gid: (1264000513/domain users)</div>
<div>Context: system_u:object_r:tmp_t:s0</div>
<div>Access: 2018-11-21 22:17:41.000000000 +0000</div>
<div>Modify: 2018-11-21 22:17:56.000000000 +0000</div>
<div>Change: 2018-11-21 22:17:56.000000000 +0000</div>
<div> Birth: -</div>
</div>
<div></div>
<div>Result from running the IOR read test.</div>
<div></div>
<div>
<div>IOR-3.2.0: MPI Coordinated Test of Parallel I/O</div>
<div>Began : Wed Nov 21 22:22:52 2018</div>
<div>Command line : /avere/apps/ior/bin/ior -a POSIX -B -r -k -z -v -o /mnt/beegfs/testing/testfile -i 2 -m -t 32m -b 256M -d 1</div>
<div>Machine : Linux ip-0A021005</div>
<div>Start time skew across all tasks: 0.00 sec</div>
<div>TestID : 0</div>
<div>StartTime : Wed Nov 21 22:22:52 2018</div>
<div>Path : /mnt/beegfs/testing</div>
<div>FS : 11.8 TiB Used FS: 0.0% Inodes: 0.0 Mi Used Inodes: -nan%</div>
<div>Participating tasks: 8</div>
<div></div>
<div>Options:</div>
<div>api : POSIX</div>
<div>apiVersion :</div>
<div>test filename : /mnt/beegfs/testing/testfile</div>
<div>access : single-shared-file</div>
<div>type : independent</div>
<div>segments : 1</div>
<div>ordering in a file : random</div>
<div>ordering inter file : no tasks offsets</div>
<div>tasks : 8</div>
<div>clients per node : 8</div>
<div>repetitions : 2</div>
<div>xfersize : 32 MiB</div>
<div>blocksize : 256 MiB</div>
<div>aggregate filesize : 2 GiB</div>
<div></div>
<div>Results:</div>
<div></div>
<div>access bw(MiB/s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) total(s) iter</div>
<div>------ --------- ---------- --------- -------- -------- -------- -------- ----</div>
<div>delaying 1 seconds . . .</div>
<div>Commencing read performance test: Wed Nov 21 22:22:53 2018</div>
<div>read 481.88 262144 32768 0.020854 4.23 2.97 4.25 0</div>
<div>delaying 1 seconds . . .</div>
<div>Commencing read performance test: Wed Nov 21 22:22:58 2018</div>
<div>read 609.04 262144 32768 0.020336 3.34 2.06 3.36 1</div>
<div>Max Read: 609.04 MiB/sec (638.62 MB/sec)</div>
<div></div>
<div>Summary of all tests:</div>
<div>Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Max(OPs) Min(OPs) Mean(OPs) StdDev Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggs(MiB) API RefNum</div>
<div>read 609.04 481.88 545.46 63.58 19.03 15.06 17.05 1.99 3.80633 0 8 8 2 0 0 1 0 0 1 268435456 33554432 2048.0 POSIX 0</div>
<div>Finished : Wed Nov 21 22:23:02 2018</div>
</div>
<div></div>
<div>I also ran a test without the -B option and got the same result (no reads recorded)?</div>
<div></div>
<div>Thanks,</div>
<div>Cormac.</div>
<div></div>
<div></div>
</div>
</div>
</div>
</div>
<div class="gmail_quote">
<div dir="ltr">On Wed, Nov 21, 2018 at 1:46 PM Carns, Philip H. via Darshan-users <<a target="_blank" href="mailto:darshan-users@lists.mcs.anl.gov">darshan-users@lists.mcs.anl.gov</a>> wrote:</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="font-family:'Segoe UI',Frutiger,'Frutiger Linotype','Dejavu Sans','Helvetica Neue',Arial,sans-serif;font-size:14px">
<div class="m_-4332855151699852336m_2543661479316708924hiri-body-wrapper">
<div>Hi Cormac,<br>
<br>
That's strange. The -i 2 and -m options are making IOR attempt to read two files: testfile.0 and testfile.1. Darshan is definitely aware of those files, but not only are the read counters missing but also the open counters. It only shows a stat from each
rank for the files in question.<br>
<br>
Can you confirm that both of the files exist and are big enough to accommodate the requested read volume? I wonder if IOR might stat the files up front and exit early if they aren't the correct size?<br>
<br>
Could you also share the output of "ldd ior"? I'm curious to make sure there isn't anything unusual about the libraries linked in, but usually if that were a problem you wouldn't get a log at all.<br>
<br>
Also one last idea, does the behavior change if you remove the -B (O_DIRECT) option? That shouldn't matter from Darshan's perspective, but it might not hurt to check.<br>
<br>
thanks,<br>
-Phil<br>
</div>
</div>
<div class="m_-4332855151699852336m_2543661479316708924hiri-extra">
<p>On 2018-11-20 19:59:55-05:00 Darshan-users wrote:</p>
<blockquote style="padding-left:10px;border-left:1px solid #ccc;margin:0">
<div>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">Hello,
<div>I recently installed Darshan 3.1.6 on Microsoft Azure VM running Centos 7.4.</div>
<div></div>
<div>I got an unexpected result using Darshan to characterize the I/O for an IOR benchmark</div>
<div>experiment. </div>
<div></div>
<div>export LD_PRELOAD=/avere/apps/spack/linux-centos7-x86_64/gcc-4.8.5/darshan-runtime-3.1.6-daohky4yevagxajjl33lwk472lcgn6g4/lib/libdarshan.so</div>
<div>
<div>BLOCKSIZE=256M</div>
<div>mpirun -np 8 ior -a POSIX -B -r -k -z -v -o $FILEPATH -i 2 -m -t 32m -b 256M -d 1 </div>
</div>
<div></div>
<div>After the job completed, a Darshan log file was created, the resulting text report (darshan-parser_ior_read_shared.out, its attached) was generated using the following command</div>
<div></div>
<div>darshan-parser --all testuser_ior_id22-20-80229-14589323501426222819_1.darshan >& darshan-parser_ior_read_shared.out</div>
<div></div>
<div>The above IOR benchmark is a read only benchmark to a shared file system, but the resulting darshan report indicates there are no READ operations?</div>
<div></div>
<div>Any ideas why the resulting Darshan resport has no read operations? (Note if I add a -w option to the above IOR benchmark (i.e do a write and read to a shared filesystem, darshan only</div>
<div>reports the writes and not reads?)</div>
<div></div>
<div>Any help would be appreciated,</div>
<div>Thanks for your support,</div>
<div>Cormac.</div>
<div></div>
<div></div>
<div></div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
_______________________________________________<br>
Darshan-users mailing list<br>
<a target="_blank" href="mailto:Darshan-users@lists.mcs.anl.gov">Darshan-users@lists.mcs.anl.gov</a><br>
<a rel="noreferrer" target="_blank" href="https://lists.mcs.anl.gov/mailman/listinfo/darshan-users">https://lists.mcs.anl.gov/mailman/listinfo/darshan-users</a></blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</body>
</html>