<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi Cormac,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thanks for reporting this bug.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
This is the first I've seen an error like this and, to be honest, I'm not sure what could ultimately cause it. It seems Darshan is failing to write its header, which is the last thing that is written to the log file -- presumably, Darshan was able to write
other portions of the log file with no problem, which is the odd part.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I'll have to try to see if I can reproduce the problem and work out what's going wrong. I'll keep you posted.<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
--Shane<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Darshan-users <darshan-users-bounces@lists.mcs.anl.gov> on behalf of Cormac Garvey <cormac.t.garvey@gmail.com><br>
<b>Sent:</b> Friday, September 18, 2020 1:31 PM<br>
<b>To:</b> darshan-users@lists.mcs.anl.gov <darshan-users@lists.mcs.anl.gov><br>
<b>Subject:</b> [Darshan-users] Darshan v3.2.1 hangs with mvapich2 2.3.3 (IOR)</font>
<div> </div>
</div>
<div>
<div dir="ltr"><br>
<div>Hi All,</div>
<div>I am unable to get darshan v3.2.1 to work with IOR v3.2.1 built with mvapich2 v2.3.3.</div>
<div><br>
</div>
<div>I am using CentOS 7.7</div>
<div><br>
</div>
<div>module load mpi/mvapich2<br>
</div>
<div>spack load darshan-runtime^mvapich2<br>
</div>
<div>export DARSHAN_LOG_DIR_PATH=/share/home/hpcuser/darshan_logs<br>
</div>
<div>export LD_PRELOAD=${DARSHAN_RUNTIME_DIR}/lib/libdarshan.so<br>
</div>
<div><br>
</div>
<div>TRANSFER_SIZE=32m<br>
SIZE=2G<br>
IO_API=POSIX<br>
IO_API_ARG="-F"<br>
TYPE_IO="direct_io"<br>
TYPE_IO_ARG="-B"<br>
</div>
<div><br>
</div>
<div>MPI_OPTS=""-np 4 --hostfile $PBS_NODEFILE"</div>
<div><br>
</div>
<div>mpirun -genvlist LD_PRELOAD,DARSHAN_LOG_DIR_PATH -bind-to hwthread $MPI_OPTS $IOR_BIN/ior -a $IO_API -v -i 1 $TYPE_IO_ARG -m -d 1 $IO_API_ARG -w -t $TRANSFER_SIZE -b $SIZE -o ${FILESYSTEM}/test -O summaryFormat=$SUMMARY_FORMAT -O summaryFile=ior_${IO_API}_${TYPE_IO}_${TRANSFER_SIZE}_${SIZE}_${HOST}_${NUMPROCS}.out_$$<br>
</div>
<div><br>
</div>
<div>The IOR job hangs at the end if I export the darshan LD_PRELOAD (Runs correctly if I remove the LD_PRELOAD)</div>
<div><br>
</div>
<div>When I kill the job, i get the following information in PBS stderr file.</div>
<div><br>
</div>
<div>"darshan_library_warning: unable to write header to file /share/home/hpcuser/darshan_logs/hpcuser_ior_id26_9-18-65156-2174583250033636718.darshan_partial."</div>
<div><br>
</div>
<div>The IOR benchmark completes (i.e can see the I/O stats), but does not appear to exit correctly (remains in a hung state until I kill it)</div>
<div><br>
</div>
<div><br>
</div>
<div>Running a similar job with IOR+mpich or IOR+OpenMPI works fine with darshan.</div>
<div><br>
</div>
<div>Any ideas what I am missing?</div>
<div><br>
</div>
<div>Thanks for your support.</div>
<div><br>
</div>
<div>Regards,</div>
<div>Cormac.</div>
</div>
</div>
</body>
</html>