<div dir="ltr"><div dir="ltr"><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">Here is my run script on perlmutter:</div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)"><br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">#!/usr/bin/env python<br>#<br>#SBATCH -A mp9<br>#SBATCH -C cpu<br>#SBATCH --qos=regular<br>#SBATCH --time=15<br>#SBATCH --nodes=16<br>#SBATCH --ntasks-per-node=128<br><br>import os<br>import glob<br><br>with open("<a href="http://pioperf.nl">pioperf.nl</a>","w") as fd:<br> fd.write("&pioperf\n")<br> fd.write(" decompfile='ROUNDROBIN'\n")<br># for filename in decompfiles:<br># fd.write(" '"+filename+"',\n")<br> fd.write(" varsize=18560\n");<br> fd.write(" pio_typenames = 'pnetcdf','pnetcdf'\n");<br> fd.write(" rearrangers = 2\n");<br> fd.write(" nframes = 1\n");<br> fd.write(" nvars = 64\n");<br> fd.write(" niotasks = 16\n");<br> fd.write(" /\n")<br><br>os.system("srun -n 2048 ~/parallelio/bld/tests/performance/pioperf ")</div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)"><br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)"><br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">Module environment:</div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">Currently Loaded Modules:<br> 1) craype-x86-milan 6) cpe/23.03 11) craype-accel-nvidia80 16) craype/2.7.20 21) cmake/3.24.3<br> 2) libfabric/<a href="http://1.15.2.0">1.15.2.0</a> 7) xalt/2.10.2 12) gpu/1.0 17) cray-dsmml/0.2.2 22) cray-parallel-netcdf/<a href="http://1.12.3.3">1.12.3.3</a><br> 3) craype-network-ofi 8) Nsight-Compute/2022.1.1 13) evp-patch 18) cray-mpich/8.1.25 23) cray-hdf5/<a href="http://1.12.2.3">1.12.2.3</a><br> 4) xpmem/2.5.2-2.4_3.49__gd0f7936.shasta 9) Nsight-Systems/2022.2.1 14) python/3.9-anaconda-2021.11 19) cray-libsci/<a href="http://23.02.1.1">23.02.1.1</a> 24) cray-netcdf/<a href="http://4.9.0.3">4.9.0.3</a><br> 5) perftools-base/23.03.0 10) cudatoolkit/11.7 15) intel/2023.1.0 20) PrgEnv-intel/8.3.3<br><br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">cmake command:<br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)"> CC=mpicc FC=mpifort cmake -DPNETCDF_DIR=$CRAY_PARALLEL_NETCDF_DIR/intel/19.0 -DNETCDF_DIR=$CRAY_NETCDF_PREFIX -DHAVE_PAR_FILTERS=OFF ../</div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)"><br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">There are a couple of issues with the build that can be fixed by editing file config.h (created in the bld directory by cmake)</div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)"><br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">Add the following to config.h:<br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)"><br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">#undef NC_HAS_ZSTD<br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">#undef NC_HAS_BZ2</div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)"><br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">then:</div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">make pioperf <br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)"><br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">once it's built run the submit script from $SCRATCH<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Aug 11, 2023 at 11:13 AM Wei-Keng Liao <<a href="mailto:wkliao@northwestern.edu">wkliao@northwestern.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
OK. I will test it myself on Perlmutter.
<div>Do you have a small test program to reproduce or is it still pioperf?</div>
<div>If pioperf, are the build instructions on Perlmutter the same?</div>
<div><br>
</div>
<div>Please let me know how you run on Perlmutter, i.e. no. process, nodes,</div>
<div>Lustre striping, problem size, etc.</div>
<div><br>
</div>
<div>Does "1 16 64" in your results mean 16 I/O tasks and 64 variables,</div></div></blockquote><div><div style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)" class="gmail_default">yes this is correct<br></div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
<div>and only 16 MPI processes out of total ? processes call PnetCDF APIs?</div>
<div> <br></div></div></blockquote><div><div style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)" class="gmail_default">yes this is also correct. <br></div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div>
<div>Wei-keng </div>
<div><br>
<blockquote type="cite">
<div>On Aug 11, 2023, at 9:35 AM, Jim Edwards <<a href="mailto:jedwards@ucar.edu" target="_blank">jedwards@ucar.edu</a>> wrote:</div>
<br>
<div>
<div dir="ltr">
<div style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
I tried on perlmutter and am seeing the same issue only maybe even worse:</div>
<div style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
<br>
</div>
<div style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
RESULT: write SUBSET 1 16 64 1261.0737058071 14.7176171500</div>
<div style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
RESULT: write SUBSET 1 16 64 90.3736534450 205.3695882870</div>
<div style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, Aug 11, 2023 at 8:17 AM Jim Edwards <<a href="mailto:jedwards@ucar.edu" target="_blank">jedwards@ucar.edu</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
Hi Wei-Keng,</div>
<div style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
<br>
</div>
<div style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
I released that the numbers in this table are all showing the slow performing file and the fast file
<br>
</div>
<div style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
(the one without the scalar variable) are not represented - I will rerun and present these numbers again.</div>
<div style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
<br>
</div>
<div style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
Here are corrected numbers for a few cases:<br>
</div>
<div style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
<div style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
GPFS (/glade/work on derecho):</div>
<div style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
RESULT: write SUBSET 1 16 64 4570.2078677815 4.0610844270</div>
<div style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
RESULT: write SUBSET 1 16 64 4470.3231494386 4.1518251320</div>
<div style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
<br>
</div>
<div style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
Lustre, default PFL's:</div>
<div style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
RESULT: write SUBSET 1 16 64 2808.6570137094 6.6081404420</div>
<div style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
RESULT: write SUBSET 1 16 64 1025.1671656858 18.1043644600</div>
<div style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
<br>
<div style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
<span>LUSTRE, no PFL's and very wide stripe:<br>
</span> RESULT: write SUBSET 1 16 64 4687.6852437580 3.9593102000</div>
<div style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
RESULT: write SUBSET 1 16 64 3001.4741125579 6.1836282120</div>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu, Aug 10, 2023 at 11:34 AM Jim Edwards <<a href="mailto:jedwards@ucar.edu" target="_blank">jedwards@ucar.edu</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
the stripe settings</div>
<div style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
lfs setstripe -c 96 -S 128M <br>
<pre style="box-sizing:inherit;margin:4px 0px;padding:8px;font-size:12px;font-variant-ligatures:none;line-height:1.50001;white-space:pre-wrap;word-break:normal;font-family:Monaco,Menlo,Consolas,"Courier New",monospace;border-radius:4px;overflow-y:hidden;color:rgb(29,28,29);font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:left;text-indent:0px;text-transform:none;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial">logs/c96_S128M/</pre>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr"><br>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote></div><br clear="all"><br><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div><div>Jim Edwards<br><br></div><font size="1">CESM Software Engineer<br></font></div><font size="1">National Center for Atmospheric Research<br></font></div><font size="1">Boulder, CO</font> <br></div></div></div>