<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
OK. I will test it myself on Perlmutter.
<div>Do you have a small test program to reproduce or is it still pioperf?</div>
<div>If pioperf, are the build instructions on Perlmutter the same?</div>
<div><br>
</div>
<div>Please let me know how you run on Perlmutter, i.e. no. process, nodes,</div>
<div>Lustre striping, problem size, etc.</div>
<div><br>
</div>
<div>Does "1 16 64" in your results mean 16 I/O tasks and 64 variables,</div>
<div>and only 16 MPI processes out of total ? processes call PnetCDF APIs?</div>
<div> <br>
<div>Wei-keng </div>
<div><br>
<blockquote type="cite">
<div>On Aug 11, 2023, at 9:35 AM, Jim Edwards <jedwards@ucar.edu> wrote:</div>
<br class="Apple-interchange-newline">
<div>
<div dir="ltr">
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">
I tried on perlmutter and am seeing the same issue only maybe even worse:</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">
<br>
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">
RESULT: write SUBSET 1 16 64 1261.0737058071 14.7176171500</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">
RESULT: write SUBSET 1 16 64 90.3736534450 205.3695882870</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">
<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, Aug 11, 2023 at 8:17 AM Jim Edwards <<a href="mailto:jedwards@ucar.edu">jedwards@ucar.edu</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
Hi Wei-Keng,</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
<br>
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
I released that the numbers in this table are all showing the slow performing file and the fast file
<br>
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
(the one without the scalar variable) are not represented - I will rerun and present these numbers again.</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
<br>
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
Here are corrected numbers for a few cases:<br>
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
GPFS (/glade/work on derecho):</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
RESULT: write SUBSET 1 16 64 4570.2078677815 4.0610844270</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
RESULT: write SUBSET 1 16 64 4470.3231494386 4.1518251320</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
<br>
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
Lustre, default PFL's:</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
RESULT: write SUBSET 1 16 64 2808.6570137094 6.6081404420</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
RESULT: write SUBSET 1 16 64 1025.1671656858 18.1043644600</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
<br>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
<span>LUSTRE, no PFL's and very wide stripe:<br>
</span> RESULT: write SUBSET 1 16 64 4687.6852437580 3.9593102000</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
RESULT: write SUBSET 1 16 64 3001.4741125579 6.1836282120</div>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu, Aug 10, 2023 at 11:34 AM Jim Edwards <<a href="mailto:jedwards@ucar.edu" target="_blank">jedwards@ucar.edu</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
the stripe settings</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
lfs setstripe -c 96 -S 128M <br>
<pre style="box-sizing:inherit;margin:4px 0px;padding:8px;font-size:12px;font-variant-ligatures:none;line-height:1.50001;white-space:pre-wrap;word-break:normal;font-family:Monaco,Menlo,Consolas,"Courier New",monospace;border-radius:4px;overflow-y:hidden;color:rgb(29,28,29);font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:left;text-indent:0px;text-transform:none;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial">logs/c96_S128M/</pre>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr"><br>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</body>
</html>