<div dir="ltr"><div dir="ltr"><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">Hi Wei-keng,,</div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)"><br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">For this case I'm using a RoundRobin distribution as shown here.<br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)"><br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)"> if(doftype .eq. 'ROUNDROBIN') then<br> do i=1,varsize<br> compmap(i) = (i-1)*npe+mype+1<br> enddo</div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)"><br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)"><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Aug 11, 2023 at 4:54 PM Wei-Keng Liao <<a href="mailto:wkliao@northwestern.edu">wkliao@northwestern.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>Hi, Jim</div>
<div><br>
</div>
<div>Can you please describe the data partitioning pattern used in pioperf?</div>
<br>
<div>Wei-keng </div>
<div><br>
<blockquote type="cite">
<div>On Aug 11, 2023, at 5:46 PM, Jim Edwards <<a href="mailto:jedwards@ucar.edu" target="_blank">jedwards@ucar.edu</a>> wrote:</div>
<br>
<div>
<div dir="ltr">
<div style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
Yes that line is called by all processes, but it in turn calls into pio_getput_int.c line 1159</div>
<div style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
<br>
</div>
<div style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
ierr = ncmpi_bput_vars_text(file->fh, varid, start, count, fake_stride, buf, request);</div>
<div style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
<br>
</div>
<div style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
which is called only by MPI_ROOT (line 1088 of the same file). <br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, Aug 11, 2023 at 4:41 PM Wei-Keng Liao <<a href="mailto:wkliao@northwestern.edu" target="_blank">wkliao@northwestern.edu</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>I can see line 344 of pioperformance.F90 is called by all processes.
<div><br>
</div>
<div>
<div> nvarmult= pio_put_var(File, rundate, date//' '//time(1:4))</div>
<div><br>
</div>
<div>How do I change it, so it is called by rank 0 only?</div>
<div><br>
</div>
<div><br>
</div>
<div>Wei-keng </div>
<div><br>
<blockquote type="cite">
<div>On Aug 11, 2023, at 4:25 PM, Jim Edwards <<a href="mailto:jedwards@ucar.edu" target="_blank">jedwards@ucar.edu</a>> wrote:</div>
<br>
<div>
<div dir="ltr">
<div style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
Yes - src/clib/pio_darray_int.c and src/clib/pio_getput_int.c <br>
</div>
<div style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
<br>
</div>
<div style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
I've also attached a couple of darshan dxt profiles. Both use the same</div>
<div style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
lustre file parameters, the first (dxt1.out) is the fast one without the scalar write.</div>
<div style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
dxt2.out is the slow one. It seems like adding the scalar is causing all of the other writes to get broken up into smaller bits.</div>
<div style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
I've also tried moving around where the scalar variable is defined and written with respect to the record variables - that doesn't seem to make any difference.
<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, Aug 11, 2023 at 3:21 PM Wei-Keng Liao <<a href="mailto:wkliao@northwestern.edu" target="_blank">wkliao@northwestern.edu</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>Yes, I have.
<div><br>
</div>
<div>Can you let me know the source codes files that make the PnetCDF API calls?</div>
<div><br>
</div>
<div><br>
<div>Wei-keng </div>
<div><br>
<blockquote type="cite">
<div>On Aug 11, 2023, at 4:10 PM, Jim Edwards <<a href="mailto:jedwards@ucar.edu" target="_blank">jedwards@ucar.edu</a>> wrote:</div>
<br>
<div>
<div dir="ltr">
<div style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
Hi Wei-Keng,</div>
<div style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
<br>
</div>
<div style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
Sorry about the miscommunication earlier today - I just wanted to confirm that you've been able to reproduce the issue now?<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, Aug 11, 2023 at 1:01 PM Jim Edwards <<a href="mailto:jedwards@ucar.edu" target="_blank">jedwards@ucar.edu</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
I'm sorry - I thought that I had provided that, but I guess not.</div>
<div style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
repo: git@github.com:jedwards4b/ParallelIO.git </div>
<div style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
branch: bugtest/lustre</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, Aug 11, 2023 at 12:46 PM Wei-Keng Liao <<a href="mailto:wkliao@northwestern.edu" target="_blank">wkliao@northwestern.edu</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Any particular github branch I should use?<br>
<br>
I got an error during make.<br>
/global/homes/w/wkliao/PIO/Github/ParallelIO/src/clib/pio_nc4.c:1481:18: error: call to undeclared function 'nc_inq_var_filter_ids'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]<br>
ierr = nc_inq_var_filter_ids(file->fh, varid, nfiltersp, ids);<br>
^<br>
<br>
<br>
Setting these 2 does not help.<br>
#undef NC_HAS_ZSTD<br>
#undef NC_HAS_BZ2<br>
<br>
<br>
Wei-keng<br>
<br>
> On Aug 11, 2023, at 12:44 PM, Jim Edwards <<a href="mailto:jedwards@ucar.edu" target="_blank">jedwards@ucar.edu</a>> wrote:<br>
> <br>
> I see I missed answering one question - total 2048 tasks. (16 nodes)<br>
> <br>
> On Fri, Aug 11, 2023 at 11:35 AM Jim Edwards <<a href="mailto:jedwards@ucar.edu" target="_blank">jedwards@ucar.edu</a>> wrote:<br>
> Here is my run script on perlmutter:<br>
> <br>
> #!/usr/bin/env python<br>
> #<br>
> #SBATCH -A mp9<br>
> #SBATCH -C cpu<br>
> #SBATCH --qos=regular<br>
> #SBATCH --time=15<br>
> #SBATCH --nodes=16<br>
> #SBATCH --ntasks-per-node=128<br>
> <br>
> import os<br>
> import glob<br>
> <br>
> with open("<a href="https://urldefense.com/v3/__http://pioperf.nl__;!!Dq0X2DkFhyF93HkjWTBQKhk!Wb6cQ1sLURpN2Ny6hale9LwE7W4NZTmSs0o72VGzPYA0zKG52eGy3PPukBWcmzrlC-J0mML7UGxclZF0OT7T_80$" rel="noreferrer" target="_blank">pioperf.nl</a>","w") as fd:<br>
> fd.write("&pioperf\n")<br>
> fd.write(" decompfile='ROUNDROBIN'\n")<br>
> # for filename in decompfiles:<br>
> # fd.write(" '"+filename+"',\n")<br>
> fd.write(" varsize=18560\n");<br>
> fd.write(" pio_typenames = 'pnetcdf','pnetcdf'\n");<br>
> fd.write(" rearrangers = 2\n");<br>
> fd.write(" nframes = 1\n");<br>
> fd.write(" nvars = 64\n");<br>
> fd.write(" niotasks = 16\n");<br>
> fd.write(" /\n")<br>
> <br>
> os.system("srun -n 2048 ~/parallelio/bld/tests/performance/pioperf ")<br>
> <br>
> <br>
> Module environment:<br>
> Currently Loaded Modules:<br>
> 1) craype-x86-milan 6) cpe/23.03 11) craype-accel-nvidia80 16) craype/2.7.20 21) cmake/3.24.3<br>
> 2) libfabric/<a href="https://urldefense.com/v3/__http://1.15.2.0__;!!Dq0X2DkFhyF93HkjWTBQKhk!Wb6cQ1sLURpN2Ny6hale9LwE7W4NZTmSs0o72VGzPYA0zKG52eGy3PPukBWcmzrlC-J0mML7UGxclZF09lbxLQc$" rel="noreferrer" target="_blank">1.15.2.0</a> 7)
xalt/2.10.2 12) gpu/1.0 17) cray-dsmml/0.2.2 22) cray-parallel-netcdf/<a href="https://urldefense.com/v3/__http://1.12.3.3__;!!Dq0X2DkFhyF93HkjWTBQKhk!Wb6cQ1sLURpN2Ny6hale9LwE7W4NZTmSs0o72VGzPYA0zKG52eGy3PPukBWcmzrlC-J0mML7UGxclZF0srcAuMk$" rel="noreferrer" target="_blank">1.12.3.3</a><br>
> 3) craype-network-ofi 8) Nsight-Compute/2022.1.1 13) evp-patch 18) cray-mpich/8.1.25 23) cray-hdf5/<a href="https://urldefense.com/v3/__http://1.12.2.3__;!!Dq0X2DkFhyF93HkjWTBQKhk!Wb6cQ1sLURpN2Ny6hale9LwE7W4NZTmSs0o72VGzPYA0zKG52eGy3PPukBWcmzrlC-J0mML7UGxclZF0dMhur8g$" rel="noreferrer" target="_blank">1.12.2.3</a><br>
> 4) xpmem/2.5.2-2.4_3.49__gd0f7936.shasta 9) Nsight-Systems/2022.2.1 14) python/3.9-anaconda-2021.11 19) cray-libsci/<a href="https://urldefense.com/v3/__http://23.02.1.1__;!!Dq0X2DkFhyF93HkjWTBQKhk!Wb6cQ1sLURpN2Ny6hale9LwE7W4NZTmSs0o72VGzPYA0zKG52eGy3PPukBWcmzrlC-J0mML7UGxclZF0iCkdumc$" rel="noreferrer" target="_blank">23.02.1.1</a>
24) cray-netcdf/<a href="https://urldefense.com/v3/__http://4.9.0.3__;!!Dq0X2DkFhyF93HkjWTBQKhk!Wb6cQ1sLURpN2Ny6hale9LwE7W4NZTmSs0o72VGzPYA0zKG52eGy3PPukBWcmzrlC-J0mML7UGxclZF0K5aEvpc$" rel="noreferrer" target="_blank">4.9.0.3</a><br>
> 5) perftools-base/23.03.0 10) cudatoolkit/11.7 15) intel/2023.1.0 20) PrgEnv-intel/8.3.3<br>
> <br>
> cmake command:<br>
> CC=mpicc FC=mpifort cmake -DPNETCDF_DIR=$CRAY_PARALLEL_NETCDF_DIR/intel/19.0 -DNETCDF_DIR=$CRAY_NETCDF_PREFIX -DHAVE_PAR_FILTERS=OFF ../<br>
> <br>
> There are a couple of issues with the build that can be fixed by editing file config.h (created in the bld directory by cmake)<br>
> <br>
> Add the following to config.h:<br>
> <br>
> #undef NC_HAS_ZSTD<br>
> #undef NC_HAS_BZ2<br>
> <br>
> then:<br>
> make pioperf <br>
> <br>
> once it's built run the submit script from $SCRATCH<br>
> <br>
> On Fri, Aug 11, 2023 at 11:13 AM Wei-Keng Liao <<a href="mailto:wkliao@northwestern.edu" target="_blank">wkliao@northwestern.edu</a>> wrote:<br>
> OK. I will test it myself on Perlmutter. <br>
> Do you have a small test program to reproduce or is it still pioperf?<br>
> If pioperf, are the build instructions on Perlmutter the same?<br>
> <br>
> Please let me know how you run on Perlmutter, i.e. no. process, nodes,<br>
> Lustre striping, problem size, etc.<br>
> <br>
> Does "1 16 64" in your results mean 16 I/O tasks and 64 variables,<br>
> yes this is correct<br>
> <br>
> and only 16 MPI processes out of total ? processes call PnetCDF APIs?<br>
> <br>
> yes this is also correct. <br>
> <br>
> Wei-keng <br>
> <br>
>> On Aug 11, 2023, at 9:35 AM, Jim Edwards <<a href="mailto:jedwards@ucar.edu" target="_blank">jedwards@ucar.edu</a>> wrote:<br>
>> <br>
>> I tried on perlmutter and am seeing the same issue only maybe even worse:<br>
>> <br>
>> RESULT: write SUBSET 1 16 64 1261.0737058071 14.7176171500<br>
>> RESULT: write SUBSET 1 16 64 90.3736534450 205.3695882870<br>
>> <br>
>> <br>
>> On Fri, Aug 11, 2023 at 8:17 AM Jim Edwards <<a href="mailto:jedwards@ucar.edu" target="_blank">jedwards@ucar.edu</a>> wrote:<br>
>> Hi Wei-Keng,<br>
>> <br>
>> I released that the numbers in this table are all showing the slow performing file and the fast file
<br>
>> (the one without the scalar variable) are not represented - I will rerun and present these numbers again.<br>
>> <br>
>> Here are corrected numbers for a few cases:<br>
>> GPFS (/glade/work on derecho):<br>
>> RESULT: write SUBSET 1 16 64 4570.2078677815 4.0610844270<br>
>> RESULT: write SUBSET 1 16 64 4470.3231494386 4.1518251320<br>
>> <br>
>> Lustre, default PFL's:<br>
>> RESULT: write SUBSET 1 16 64 2808.6570137094 6.6081404420<br>
>> RESULT: write SUBSET 1 16 64 1025.1671656858 18.1043644600<br>
>> <br>
>> LUSTRE, no PFL's and very wide stripe:<br>
>> RESULT: write SUBSET 1 16 64 4687.6852437580 3.9593102000<br>
>> RESULT: write SUBSET 1 16 64 3001.4741125579 6.1836282120<br>
>> <br>
>> On Thu, Aug 10, 2023 at 11:34 AM Jim Edwards <<a href="mailto:jedwards@ucar.edu" target="_blank">jedwards@ucar.edu</a>> wrote:<br>
>> the stripe settings<br>
>> lfs setstripe -c 96 -S 128M <br>
>> logs/c96_S128M/<br>
>> <br>
>> <br>
> <br>
> <br>
> <br>
> -- <br>
> Jim Edwards<br>
> <br>
> CESM Software Engineer<br>
> National Center for Atmospheric Research<br>
> Boulder, CO <br>
> <br>
> <br>
> -- <br>
> Jim Edwards<br>
> <br>
> CESM Software Engineer<br>
> National Center for Atmospheric Research<br>
> Boulder, CO <br>
<br>
</blockquote>
</div>
<br clear="all">
<br>
<span class="gmail_signature_prefix">-- </span><br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div>
<div>Jim Edwards<br>
<br>
</div>
<font size="1">CESM Software Engineer<br>
</font></div>
<font size="1">National Center for Atmospheric Research<br>
</font></div>
<font size="1">Boulder, CO</font> <br>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<br>
<span class="gmail_signature_prefix">-- </span><br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div>
<div>Jim Edwards<br>
<br>
</div>
<font size="1">CESM Software Engineer<br>
</font></div>
<font size="1">National Center for Atmospheric Research<br>
</font></div>
<font size="1">Boulder, CO</font> <br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<br>
<span class="gmail_signature_prefix">-- </span><br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div>
<div>Jim Edwards<br>
<br>
</div>
<font size="1">CESM Software Engineer<br>
</font></div>
<font size="1">National Center for Atmospheric Research<br>
</font></div>
<font size="1">Boulder, CO</font> <br>
</div>
</div>
<span id="m_-5595126119828247558m_7585114653106705547cid:f_ll73imsk1"><dxt2.out></span><span id="m_-5595126119828247558m_7585114653106705547cid:f_ll73imsf0"><dxt1.out></span></div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<br>
<span class="gmail_signature_prefix">-- </span><br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div>
<div>Jim Edwards<br>
<br>
</div>
<font size="1">CESM Software Engineer<br>
</font></div>
<font size="1">National Center for Atmospheric Research<br>
</font></div>
<font size="1">Boulder, CO</font> <br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote></div><br clear="all"><br><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div><div>Jim Edwards<br><br></div><font size="1">CESM Software Engineer<br></font></div><font size="1">National Center for Atmospheric Research<br></font></div><font size="1">Boulder, CO</font> <br></div></div></div>