<div dir="ltr"><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">I am using the new ncar system, <a href="https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&cad=rja&uact=8&ved=2ahUKEwigtcytrsOAAxWXHjQIHVoDD6YQFnoECBcQAQ&url=https%3A%2F%2Farc.ucar.edu%2Fknowledge_base%2F74317833&usg=AOvVaw2aXlWuOfLnua7fFmIgvfoV&opi=89978449" target="_blank">derecho</a>, which has a lustre parallel file system.</div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d"><br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">Looking at the difference between the two headers below makes me wonder if the issue is with variable attributes?<br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d"><br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d"><br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d"><br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d"><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">Here are the steps to reproduce the standalone pioperf:</div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)"><br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">git clone <a href="https://github.com/NCAR/ParallelIO" target="_blank">https://github.com/NCAR/ParallelIO</a> <br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">cd ParallelIO</div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">mkdir bld</div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">cd bld</div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">module purge<br>module load cesmdev/1.0 <br>module load ncarenv/23.06</div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">module load intel cray-mpich netcdf-mpi parallel-netcdf cmake</div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">CC=cc FC=ftn cmake ../</div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">make pioperf</div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)"><br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">cd $SCRATCH</div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">cp -r /glade/scratch/jedwards/piodecomps .</div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">cd piodecomps/run</div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">[ edit file run32768.py to change pioperfpaths and account #]<br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">qsub run32768.py</div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)"><br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)"><br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">Here are the steps to reproduce the cesm case:</div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">git clone <a href="https://github.com/ESCOMP/CESM.git" target="_blank">https://github.com/ESCOMP/CESM.git</a></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">cd CESM</div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">./manage_externals/checkout_externals</div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">cd cime/scripts</div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">./create_test SMS_Ln9.mpasa7p5_mpasa7p5_mg17.QPC6.derecho_intel.cam-outfrq9s --no-build</div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">cd $SCRATCH/SMS_Ln9.mpasa7p5_mpasa7p5_mg17.QPC6.derecho_intel.cam-outfrq9s.*</div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">./xmlchange CAM_CONFIG_OPTS=" -phys cam6 -aquaplanet -analytic_ic -nlev=58 "</div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">./xmlchange PIO_NETCDF_FORMAT=64bit_data <br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">./xmlchange STOP_N=2 --file env_test.xml <br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">cp /glade/derecho/scratch/jedwards/SMS_D_Ln9.mpasa7p5_mpasa7p5_mg17.QPC6.derecho_intel.cam-outfrq9s.20230726_094231_iz24v6/user_nl_cam.64 user_nl_cam</div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">qcmd -- ./case.build</div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">./case.submit</div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)"><br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">once the job has completed:</div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">grep wrapup timing/cesm.ESMF*</div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)"><br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">Note
that in this process we found some bugs in the esmf 8.5.0 release that
don't allow the profiler tools to work. If you want to use profiling
let me know and I can provide further instructions.<font color="#888888"><br></font></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)"><font color="#888888"><br></font></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)"><font color="#888888">Here is the header dump of the pioperf file:</font></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)"><font color="#888888"><br></font></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)"><font color="#888888">netcdf pioperf.2-0256-1 {<br>dimensions:<br> dim000001 = 10485762 ;<br> dim000002 = 58 ;<br> time = UNLIMITED ; // (1 currently)<br>variables:<br> double vard0001(time, dim000002, dim000001) ;<br> vard0001:_FillValue = 9.96920996838687e+36 ;<br> double vard0002(time, dim000002, dim000001) ;<br> vard0002:_FillValue = 9.96920996838687e+36 ;<br> double vard0003(time, dim000002, dim000001) ;<br> vard0003:_FillValue = 9.96920996838687e+36 ;<br> double vard0004(time, dim000002, dim000001) ;<br> vard0004:_FillValue = 9.96920996838687e+36 ;<br> double vard0005(time, dim000002, dim000001) ;<br> vard0005:_FillValue = 9.96920996838687e+36 ;<br> double vard0006(time, dim000002, dim000001) ;<br> vard0006:_FillValue = 9.96920996838687e+36 ;<br> double vard0007(time, dim000002, dim000001) ;<br> vard0007:_FillValue = 9.96920996838687e+36 ;<br> double vard0008(time, dim000002, dim000001) ;<br> vard0008:_FillValue = 9.96920996838687e+36 ;<br> double vard0009(time, dim000002, dim000001) ;<br> vard0009:_FillValue = 9.96920996838687e+36 ;<br> double vard0010(time, dim000002, dim000001) ;<br> vard0010:_FillValue = 9.96920996838687e+36 ;<br> double vard0011(time, dim000002, dim000001) ;<br> vard0011:_FillValue = 9.96920996838687e+36 ;<br> double vard0012(time, dim000002, dim000001) ;<br> vard0012:_FillValue = 9.96920996838687e+36 ;<br> double vard0013(time, dim000002, dim000001) ;<br> vard0013:_FillValue = 9.96920996838687e+36 ;<br> double vard0014(time, dim000002, dim000001) ;<br> vard0014:_FillValue = 9.96920996838687e+36 ;<br> double vard0015(time, dim000002, dim000001) ;<br> vard0015:_FillValue = 9.96920996838687e+36 ;<br> double vard0016(time, dim000002, dim000001) ;<br> vard0016:_FillValue = 9.96920996838687e+36 ;<br> double vard0017(time, dim000002, dim000001) ;<br> vard0017:_FillValue = 9.96920996838687e+36 ;<br> double vard0018(time, dim000002, dim000001) ;<br> vard0018:_FillValue = 9.96920996838687e+36 ;<br> double vard0019(time, dim000002, dim000001) ;<br> vard0019:_FillValue = 9.96920996838687e+36 ;<br> double vard0020(time, dim000002, dim000001) ;<br> vard0020:_FillValue = 9.96920996838687e+36 ;<br> double vard0021(time, dim000002, dim000001) ;<br> vard0021:_FillValue = 9.96920996838687e+36 ;<br> double vard0022(time, dim000002, dim000001) ;<br> vard0022:_FillValue = 9.96920996838687e+36 ;<br> double vard0023(time, dim000002, dim000001) ;<br> vard0023:_FillValue = 9.96920996838687e+36 ;<br> double vard0024(time, dim000002, dim000001) ;<br> vard0024:_FillValue = 9.96920996838687e+36 ;<br> double vard0025(time, dim000002, dim000001) ;<br> vard0025:_FillValue = 9.96920996838687e+36 ;<br> double vard0026(time, dim000002, dim000001) ;<br> vard0026:_FillValue = 9.96920996838687e+36 ;<br> double vard0027(time, dim000002, dim000001) ;<br> vard0027:_FillValue = 9.96920996838687e+36 ;<br> double vard0028(time, dim000002, dim000001) ;<br> vard0028:_FillValue = 9.96920996838687e+36 ;<br> double vard0029(time, dim000002, dim000001) ;<br> vard0029:_FillValue = 9.96920996838687e+36 ;<br> double vard0030(time, dim000002, dim000001) ;<br> vard0030:_FillValue = 9.96920996838687e+36 ;<br> double vard0031(time, dim000002, dim000001) ;<br> vard0031:_FillValue = 9.96920996838687e+36 ;<br> double vard0032(time, dim000002, dim000001) ;<br> vard0032:_FillValue = 9.96920996838687e+36 ;<br> double vard0033(time, dim000002, dim000001) ;<br> vard0033:_FillValue = 9.96920996838687e+36 ;<br> double vard0034(time, dim000002, dim000001) ;<br> vard0034:_FillValue = 9.96920996838687e+36 ;<br> double vard0035(time, dim000002, dim000001) ;<br> vard0035:_FillValue = 9.96920996838687e+36 ;<br> double vard0036(time, dim000002, dim000001) ;<br> vard0036:_FillValue = 9.96920996838687e+36 ;<br> double vard0037(time, dim000002, dim000001) ;<br> vard0037:_FillValue = 9.96920996838687e+36 ;<br> double vard0038(time, dim000002, dim000001) ;<br> vard0038:_FillValue = 9.96920996838687e+36 ;<br> double vard0039(time, dim000002, dim000001) ;<br> vard0039:_FillValue = 9.96920996838687e+36 ;<br> double vard0040(time, dim000002, dim000001) ;<br> vard0040:_FillValue = 9.96920996838687e+36 ;<br> double vard0041(time, dim000002, dim000001) ;<br> vard0041:_FillValue = 9.96920996838687e+36 ;<br> double vard0042(time, dim000002, dim000001) ;<br> vard0042:_FillValue = 9.96920996838687e+36 ;<br> double vard0043(time, dim000002, dim000001) ;<br> vard0043:_FillValue = 9.96920996838687e+36 ;<br> double vard0044(time, dim000002, dim000001) ;<br> vard0044:_FillValue = 9.96920996838687e+36 ;<br> double vard0045(time, dim000002, dim000001) ;<br> vard0045:_FillValue = 9.96920996838687e+36 ;<br> double vard0046(time, dim000002, dim000001) ;<br> vard0046:_FillValue = 9.96920996838687e+36 ;<br> double vard0047(time, dim000002, dim000001) ;<br> vard0047:_FillValue = 9.96920996838687e+36 ;<br> double vard0048(time, dim000002, dim000001) ;<br> vard0048:_FillValue = 9.96920996838687e+36 ;<br> double vard0049(time, dim000002, dim000001) ;<br> vard0049:_FillValue = 9.96920996838687e+36 ;<br> double vard0050(time, dim000002, dim000001) ;<br> vard0050:_FillValue = 9.96920996838687e+36 ;<br> double vard0051(time, dim000002, dim000001) ;<br> vard0051:_FillValue = 9.96920996838687e+36 ;<br> double vard0052(time, dim000002, dim000001) ;<br> vard0052:_FillValue = 9.96920996838687e+36 ;<br> double vard0053(time, dim000002, dim000001) ;<br> vard0053:_FillValue = 9.96920996838687e+36 ;<br> double vard0054(time, dim000002, dim000001) ;<br> vard0054:_FillValue = 9.96920996838687e+36 ;<br> double vard0055(time, dim000002, dim000001) ;<br> vard0055:_FillValue = 9.96920996838687e+36 ;<br> double vard0056(time, dim000002, dim000001) ;<br> vard0056:_FillValue = 9.96920996838687e+36 ;<br> double vard0057(time, dim000002, dim000001) ;<br> vard0057:_FillValue = 9.96920996838687e+36 ;<br> double vard0058(time, dim000002, dim000001) ;<br> vard0058:_FillValue = 9.96920996838687e+36 ;<br> double vard0059(time, dim000002, dim000001) ;<br> vard0059:_FillValue = 9.96920996838687e+36 ;<br> double vard0060(time, dim000002, dim000001) ;<br> vard0060:_FillValue = 9.96920996838687e+36 ;<br> double vard0061(time, dim000002, dim000001) ;<br> vard0061:_FillValue = 9.96920996838687e+36 ;<br> double vard0062(time, dim000002, dim000001) ;<br> vard0062:_FillValue = 9.96920996838687e+36 ;<br> double vard0063(time, dim000002, dim000001) ;<br> vard0063:_FillValue = 9.96920996838687e+36 ;<br> double vard0064(time, dim000002, dim000001) ;<br> vard0064:_FillValue = 9.96920996838687e+36 ;<br>}<br></font></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)"><font color="#888888"><br></font></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)"><font color="#888888"><br></font></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)"><font color="#888888"><br></font></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)"><font color="#888888"><br></font></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)"><font color="#888888">Here is the header dump of the cesm file:</font></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)"><font color="#888888">netcdf SMS_D_Ln9.mpasa7p5_mpasa7p5_mg17.QPC6.derecho_intel.cam-outfrq9s.20230726_094231_iz24v6.cam.h0.0001-01-01-00000 {<br>dimensions:<br> ncol = 10485762 ;<br> time = UNLIMITED ; // (1 currently)<br> nbnd = 2 ;<br> chars = 8 ;<br> lev = 58 ;<br> ilev = 59 ;<br>variables:<br> double lat(ncol) ;<br> lat:long_name = "latitude" ;<br> lat:units = "degrees_north" ;<br> double lon(ncol) ;<br> lon:long_name = "longitude" ;<br> lon:units = "degrees_east" ;<br> double areawt(ncol) ;<br> areawt:long_name = "physics column area weight" ;<br> double area(ncol) ;<br> area:long_name = "physics column areas" ;<br> double lev(lev) ;<br> lev:long_name = "zeta level at vertical midpoints" ;<br> lev:units = "m" ;<br> double ilev(ilev) ;<br> ilev:long_name = "zeta level at vertical interfaces" ;<br> ilev:units = "m" ;<br> double time(time) ;<br> time:long_name = "time" ;<br> time:units = "days since 0001-01-01 00:00:00" ;<br> time:calendar = "noleap" ;<br> time:bounds = "time_bnds" ;<br> int date(time) ;<br> date:long_name = "current date (YYYYMMDD)" ;<br> int datesec(time) ;<br> datesec:long_name = "current seconds of current date" ;<br> double time_bnds(time, nbnd) ;<br> time_bnds:long_name = "time interval endpoints" ;<br> char date_written(time, chars) ;<br> char time_written(time, chars) ;<br> int ndbase ;<br> ndbase:long_name = "base day" ;<br> int nsbase ;<br> nsbase:long_name = "seconds of base day" ;<br> int nbdate ;<br> nbdate:long_name = "base date (YYYYMMDD)" ;<br> int nbsec ;<br> nbsec:long_name = "seconds of base date" ;<br> int mdt ;<br> mdt:long_name = "timestep" ;<br> mdt:units = "s" ;<br> int ndcur(time) ;<br> ndcur:long_name = "current day (from base day)" ;<br> int nscur(time) ;<br> nscur:long_name = "current seconds of current day" ;<br> double co2vmr(time) ;<br> co2vmr:long_name = "co2 volume mixing ratio" ;<br> double ch4vmr(time) ;<br> ch4vmr:long_name = "ch4 volume mixing ratio" ;<br> double n2ovmr(time) ;<br> n2ovmr:long_name = "n2o volume mixing ratio" ;<br> double f11vmr(time) ;<br> f11vmr:long_name = "f11 volume mixing ratio" ;<br> double f12vmr(time) ;<br> f12vmr:long_name = "f12 volume mixing ratio" ;<br> double sol_tsi(time) ;<br> sol_tsi:long_name = "total solar irradiance" ;<br> sol_tsi:units = "W/m2" ;<br> int nsteph(time) ;<br> nsteph:long_name = "current timestep" ;<br> double ADRAIN(time, lev, ncol) ;<br> ADRAIN:mdims = 1 ;<br> ADRAIN:units = "m" ;<br> ADRAIN:long_name = "Average rain effective Diameter" ;<br> ADRAIN:cell_methods = "time: mean" ;<br> double ADSNOW(time, lev, ncol) ;<br> ADSNOW:mdims = 1 ;<br> ADSNOW:units = "m" ;<br> ADSNOW:long_name = "Average snow effective Diameter" ;<br> ADSNOW:cell_methods = "time: mean" ;<br> double ANRAIN(time, lev, ncol) ;<br> ANRAIN:mdims = 1 ;<br> ANRAIN:units = "m-3" ;<br> ANRAIN:long_name = "Average rain number conc" ;<br> ANRAIN:cell_methods = "time: mean" ;<br> double ANSNOW(time, lev, ncol) ;<br> ANSNOW:mdims = 1 ;<br> ANSNOW:units = "m-3" ;<br> ANSNOW:long_name = "Average snow number conc" ;<br> ANSNOW:cell_methods = "time: mean" ;<br> double AQRAIN(time, lev, ncol) ;<br> AQRAIN:mdims = 1 ;<br> AQRAIN:units = "kg/kg" ;<br> AQRAIN:long_name = "Average rain mixing ratio" ;<br> AQRAIN:cell_methods = "time: mean" ;<br> double AQSNOW(time, lev, ncol) ;<br> AQSNOW:mdims = 1 ;<br> AQSNOW:units = "kg/kg" ;<br> AQSNOW:long_name = "Average snow mixing ratio" ;<br> AQSNOW:cell_methods = "time: mean" ;<br> double AREI(time, lev, ncol) ;<br> AREI:mdims = 1 ;<br> AREI:units = "Micron" ;<br> AREI:long_name = "Average ice effective radius" ;<br> AREI:cell_methods = "time: mean" ;<br> double AREL(time, lev, ncol) ;<br> AREL:mdims = 1 ;<br> AREL:units = "Micron" ;<br> AREL:long_name = "Average droplet effective radius" ;<br> AREL:cell_methods = "time: mean" ;<br> double AWNC(time, lev, ncol) ;<br> AWNC:mdims = 1 ;<br> AWNC:units = "m-3" ;<br> AWNC:long_name = "Average cloud water number conc" ;<br> AWNC:cell_methods = "time: mean" ;<br> double AWNI(time, lev, ncol) ;<br> AWNI:mdims = 1 ;<br> AWNI:units = "m-3" ;<br> AWNI:long_name = "Average cloud ice number conc" ;<br> AWNI:cell_methods = "time: mean" ;<br> double CCN3(time, lev, ncol) ;<br> CCN3:mdims = 1 ;<br> CCN3:units = "#/cm3" ;<br> CCN3:long_name = "CCN concentration at S=0.1%" ;<br> CCN3:cell_methods = "time: mean" ;<br> double CLDICE(time, lev, ncol) ;<br> CLDICE:mdims = 1 ;<br> CLDICE:units = "kg/kg" ;<br> CLDICE:mixing_ratio = "wet" ;<br> CLDICE:long_name = "Grid box averaged cloud ice amount" ;<br> CLDICE:cell_methods = "time: mean" ;<br> double CLDLIQ(time, lev, ncol) ;<br> CLDLIQ:mdims = 1 ;<br> CLDLIQ:units = "kg/kg" ;<br> CLDLIQ:mixing_ratio = "wet" ;<br> CLDLIQ:long_name = "Grid box averaged cloud liquid amount" ;<br> CLDLIQ:cell_methods = "time: mean" ;<br> double CLOUD(time, lev, ncol) ;<br> CLOUD:mdims = 1 ;<br> CLOUD:units = "fraction" ;<br> CLOUD:long_name = "Cloud fraction" ;<br> CLOUD:cell_methods = "time: mean" ;<br> double CLOUDCOVER_CLUBB(time, lev, ncol) ;<br> CLOUDCOVER_CLUBB:mdims = 1 ;<br> CLOUDCOVER_CLUBB:units = "fraction" ;<br> CLOUDCOVER_CLUBB:long_name = "Cloud Cover" ;<br> CLOUDCOVER_CLUBB:cell_methods = "time: mean" ;<br> double CLOUDFRAC_CLUBB(time, lev, ncol) ;<br> CLOUDFRAC_CLUBB:mdims = 1 ;<br> CLOUDFRAC_CLUBB:units = "fraction" ;<br> CLOUDFRAC_CLUBB:long_name = "Cloud Fraction" ;<br> CLOUDFRAC_CLUBB:cell_methods = "time: mean" ;<br> double CONCLD(time, lev, ncol) ;<br> CONCLD:mdims = 1 ;<br> CONCLD:units = "fraction" ;<br> CONCLD:long_name = "Convective cloud cover" ;<br> CONCLD:cell_methods = "time: mean" ;<br> double DCQ(time, lev, ncol) ;<br> DCQ:mdims = 1 ;<br> DCQ:units = "kg/kg/s" ;<br> DCQ:long_name = "Q tendency due to moist processes" ;<br> DCQ:cell_methods = "time: mean" ;<br> double DMS(time, lev, ncol) ;<br> DMS:mdims = 1 ;<br> DMS:units = "mol/mol" ;<br> DMS:mixing_ratio = "dry" ;<br> DMS:long_name = "DMS concentration" ;<br> DMS:cell_methods = "time: mean" ;<br> double DTCOND(time, lev, ncol) ;<br> DTCOND:mdims = 1 ;<br> DTCOND:units = "K/s" ;<br> DTCOND:long_name = "T tendency - moist processes" ;<br> DTCOND:cell_methods = "time: mean" ;<br> double DTV(time, lev, ncol) ;<br> DTV:mdims = 1 ;<br> DTV:units = "K/s" ;<br> DTV:long_name = "T vertical diffusion" ;<br> DTV:cell_methods = "time: mean" ;<br> double DTWR_DMS(time, lev, ncol) ;<br> DTWR_DMS:mdims = 1 ;<br> DTWR_DMS:units = "kg/kg/s" ;<br> DTWR_DMS:long_name = "wet removal Neu scheme tendency" ;<br> DTWR_DMS:cell_methods = "time: mean" ;<br> double DTWR_H2O2(time, lev, ncol) ;<br> DTWR_H2O2:mdims = 1 ;<br> DTWR_H2O2:units = "kg/kg/s" ;<br> DTWR_H2O2:long_name = "wet removal Neu scheme tendency" ;<br> DTWR_H2O2:cell_methods = "time: mean" ;<br> double DTWR_H2SO4(time, lev, ncol) ;<br> DTWR_H2SO4:mdims = 1 ;<br> DTWR_H2SO4:units = "kg/kg/s" ;<br> DTWR_H2SO4:long_name = "wet removal Neu scheme tendency" ;<br> DTWR_H2SO4:cell_methods = "time: mean" ;<br> double DTWR_SO2(time, lev, ncol) ;<br> DTWR_SO2:mdims = 1 ;<br> DTWR_SO2:units = "kg/kg/s" ;<br> DTWR_SO2:long_name = "wet removal Neu scheme tendency" ;<br> DTWR_SO2:cell_methods = "time: mean" ;<br> double DTWR_SOAG(time, lev, ncol) ;<br> DTWR_SOAG:mdims = 1 ;<br> DTWR_SOAG:units = "kg/kg/s" ;<br> DTWR_SOAG:long_name = "wet removal Neu scheme tendency" ;<br> DTWR_SOAG:cell_methods = "time: mean" ;<br> double FICE(time, lev, ncol) ;<br> FICE:mdims = 1 ;<br> FICE:units = "fraction" ;<br> FICE:long_name = "Fractional ice content within cloud" ;<br> FICE:cell_methods = "time: mean" ;<br> double FREQI(time, lev, ncol) ;<br> FREQI:mdims = 1 ;<br> FREQI:units = "fraction" ;<br> FREQI:long_name = "Fractional occurrence of ice" ;<br> FREQI:cell_methods = "time: mean" ;<br> double FREQL(time, lev, ncol) ;<br> FREQL:mdims = 1 ;<br> FREQL:units = "fraction" ;<br> FREQL:long_name = "Fractional occurrence of liquid" ;<br> FREQL:cell_methods = "time: mean" ;<br> double FREQR(time, lev, ncol) ;<br> FREQR:mdims = 1 ;<br> FREQR:units = "fraction" ;<br> FREQR:long_name = "Fractional occurrence of rain" ;<br> FREQR:cell_methods = "time: mean" ;<br> double FREQS(time, lev, ncol) ;<br> FREQS:mdims = 1 ;<br> FREQS:units = "fraction" ;<br> FREQS:long_name = "Fractional occurrence of snow" ;<br> FREQS:cell_methods = "time: mean" ;<br> double H2O(time, lev, ncol) ;<br> H2O:mdims = 1 ;<br> H2O:units = "mol/mol" ;<br> H2O:long_name = "water vapor concentration" ;<br> H2O:cell_methods = "time: mean" ;<br> double H2O2(time, lev, ncol) ;<br> H2O2:mdims = 1 ;<br> H2O2:units = "mol/mol" ;<br> H2O2:mixing_ratio = "dry" ;<br> H2O2:long_name = "H2O2 concentration" ;<br> H2O2:cell_methods = "time: mean" ;<br> double H2SO4(time, lev, ncol) ;<br> H2SO4:mdims = 1 ;<br> H2SO4:units = "mol/mol" ;<br> H2SO4:mixing_ratio = "dry" ;<br> H2SO4:long_name = "H2SO4 concentration" ;<br> H2SO4:cell_methods = "time: mean" ;<br> double ICIMR(time, lev, ncol) ;<br> ICIMR:mdims = 1 ;<br> ICIMR:units = "kg/kg" ;<br> ICIMR:long_name = "Prognostic in-cloud ice mixing ratio" ;<br> ICIMR:cell_methods = "time: mean" ;<br> double ICWMR(time, lev, ncol) ;<br> ICWMR:mdims = 1 ;<br> ICWMR:units = "kg/kg" ;<br> ICWMR:long_name = "Prognostic in-cloud water mixing ratio" ;<br> ICWMR:cell_methods = "time: mean" ;<br> double IWC(time, lev, ncol) ;<br> IWC:mdims = 1 ;<br> IWC:units = "kg/m3" ;<br> IWC:long_name = "Grid box average ice water content" ;<br> IWC:cell_methods = "time: mean" ;<br> double NUMICE(time, lev, ncol) ;<br> NUMICE:mdims = 1 ;<br> NUMICE:units = "1/kg" ;<br> NUMICE:mixing_ratio = "wet" ;<br> NUMICE:long_name = "Grid box averaged cloud ice number" ;<br> NUMICE:cell_methods = "time: mean" ;<br> double NUMLIQ(time, lev, ncol) ;<br> NUMLIQ:mdims = 1 ;<br> NUMLIQ:units = "1/kg" ;<br> NUMLIQ:mixing_ratio = "wet" ;<br> NUMLIQ:long_name = "Grid box averaged cloud liquid number" ;<br> NUMLIQ:cell_methods = "time: mean" ;<br> double NUMRAI(time, lev, ncol) ;<br> NUMRAI:mdims = 1 ;<br> NUMRAI:units = "1/kg" ;<br> NUMRAI:mixing_ratio = "wet" ;<br> NUMRAI:long_name = "Grid box averaged rain number" ;<br> NUMRAI:cell_methods = "time: mean" ;<br> double NUMSNO(time, lev, ncol) ;<br> NUMSNO:mdims = 1 ;<br> NUMSNO:units = "1/kg" ;<br> NUMSNO:mixing_ratio = "wet" ;<br> NUMSNO:long_name = "Grid box averaged snow number" ;<br> NUMSNO:cell_methods = "time: mean" ;<br> double OMEGA(time, lev, ncol) ;<br> OMEGA:mdims = 1 ;<br> OMEGA:units = "Pa/s" ;<br> OMEGA:long_name = "Vertical velocity (pressure)" ;<br> OMEGA:cell_methods = "time: mean" ;<br> double OMEGAT(time, lev, ncol) ;<br> OMEGAT:mdims = 1 ;<br> OMEGAT:units = "K Pa/s" ;<br> OMEGAT:long_name = "Vertical heat flux" ;<br> OMEGAT:cell_methods = "time: mean" ;<br> double PDEL(time, lev, ncol) ;<br> PDEL:mdims = 1 ;<br> PDEL:units = "Pa" ;<br> PDEL:long_name = "Pressure difference between levels" ;<br> PDEL:cell_methods = "time: mean" ;<br> double PMID(time, lev, ncol) ;<br> PMID:mdims = 1 ;<br> PMID:units = "Pa" ;<br> PMID:long_name = "Pressure at layer midpoints" ;<br> PMID:cell_methods = "time: mean" ;<br> double Q(time, lev, ncol) ;<br> Q:mdims = 1 ;<br> Q:units = "kg/kg" ;<br> Q:mixing_ratio = "wet" ;<br> Q:long_name = "Specific humidity" ;<br> Q:cell_methods = "time: mean" ;<br> double QRL(time, lev, ncol) ;<br> QRL:mdims = 1 ;<br> QRL:Sampling_Sequence = "rad_lwsw" ;<br> QRL:units = "K/s" ;<br> QRL:long_name = "Longwave heating rate" ;<br> QRL:cell_methods = "time: mean" ;<br> double QRS(time, lev, ncol) ;<br> QRS:mdims = 1 ;<br> QRS:Sampling_Sequence = "rad_lwsw" ;<br> QRS:units = "K/s" ;<br> QRS:long_name = "Solar heating rate" ;<br> QRS:cell_methods = "time: mean" ;<br> double RAINQM(time, lev, ncol) ;<br> RAINQM:mdims = 1 ;<br> RAINQM:units = "kg/kg" ;<br> RAINQM:mixing_ratio = "wet" ;<br> RAINQM:long_name = "Grid box averaged rain amount" ;<br> RAINQM:cell_methods = "time: mean" ;<br> double RCMINLAYER_CLUBB(time, lev, ncol) ;<br> RCMINLAYER_CLUBB:mdims = 1 ;<br> RCMINLAYER_CLUBB:units = "kg/kg" ;<br> RCMINLAYER_CLUBB:long_name = "Cloud Water in Layer" ;<br> RCMINLAYER_CLUBB:cell_methods = "time: mean" ;<br> double RCMTEND_CLUBB(time, lev, ncol) ;<br> RCMTEND_CLUBB:mdims = 1 ;<br> RCMTEND_CLUBB:units = "kg/kg /s" ;<br> RCMTEND_CLUBB:long_name = "Cloud Liquid Water Tendency" ;<br> RCMTEND_CLUBB:cell_methods = "time: mean" ;<br> double RCM_CLUBB(time, lev, ncol) ;<br> RCM_CLUBB:mdims = 1 ;<br> RCM_CLUBB:units = "kg/kg" ;<br> RCM_CLUBB:long_name = "Cloud Water Mixing Ratio" ;<br> RCM_CLUBB:cell_methods = "time: mean" ;<br> double RELHUM(time, lev, ncol) ;<br> RELHUM:mdims = 1 ;<br> RELHUM:units = "percent" ;<br> RELHUM:long_name = "Relative humidity" ;<br> RELHUM:cell_methods = "time: mean" ;<br> double RELVAR(time, lev, ncol) ;<br> RELVAR:mdims = 1 ;<br> RELVAR:units = "-" ;<br> RELVAR:long_name = "Relative cloud water variance" ;<br> RELVAR:cell_methods = "time: mean" ;<br> double RHO_CLUBB(time, lev, ncol) ;<br> RHO_CLUBB:mdims = 1 ;<br> RHO_CLUBB:units = "kg/m3" ;<br> RHO_CLUBB:long_name = "Air Density" ;<br> RHO_CLUBB:cell_methods = "time: mean" ;<br> double RIMTEND_CLUBB(time, lev, ncol) ;<br> RIMTEND_CLUBB:mdims = 1 ;<br> RIMTEND_CLUBB:units = "kg/kg /s" ;<br> RIMTEND_CLUBB:long_name = "Cloud Ice Tendency" ;<br> RIMTEND_CLUBB:cell_methods = "time: mean" ;<br> double RTM_CLUBB(time, lev, ncol) ;<br> RTM_CLUBB:mdims = 1 ;<br> RTM_CLUBB:units = "kg/kg" ;<br> RTM_CLUBB:long_name = "Total Water Mixing Ratio" ;<br> RTM_CLUBB:cell_methods = "time: mean" ;<br> double RTP2_ZT_CLUBB(time, lev, ncol) ;<br> RTP2_ZT_CLUBB:mdims = 1 ;<br> RTP2_ZT_CLUBB:units = "kg^2/kg^2" ;<br> RTP2_ZT_CLUBB:long_name = "Moisture Variance on zt grid" ;<br> RTP2_ZT_CLUBB:cell_methods = "time: mean" ;<br> double RVMTEND_CLUBB(time, lev, ncol) ;<br> RVMTEND_CLUBB:mdims = 1 ;<br> RVMTEND_CLUBB:units = "kg/kg /s" ;<br> RVMTEND_CLUBB:long_name = "Water vapor tendency" ;<br> RVMTEND_CLUBB:cell_methods = "time: mean" ;<br> double SNOWQM(time, lev, ncol) ;<br> SNOWQM:mdims = 1 ;<br> SNOWQM:units = "kg/kg" ;<br> SNOWQM:mixing_ratio = "wet" ;<br> SNOWQM:long_name = "Grid box averaged snow amount" ;<br> SNOWQM:cell_methods = "time: mean" ;<br> double SO2(time, lev, ncol) ;<br> SO2:mdims = 1 ;<br> SO2:units = "mol/mol" ;<br> SO2:mixing_ratio = "dry" ;<br> SO2:long_name = "SO2 concentration" ;<br> SO2:cell_methods = "time: mean" ;<br> double SOAE(time, lev, ncol) ;<br> SOAE:mdims = 1 ;<br> SOAE:units = "mol/mol" ;<br> SOAE:mixing_ratio = "dry" ;<br> SOAE:long_name = "SOAE concentration" ;<br> SOAE:cell_methods = "time: mean" ;<br> double SOAG(time, lev, ncol) ;<br> SOAG:mdims = 1 ;<br> SOAG:units = "mol/mol" ;<br> SOAG:mixing_ratio = "dry" ;<br> SOAG:long_name = "SOAG concentration" ;<br> SOAG:cell_methods = "time: mean" ;<br> double STEND_CLUBB(time, lev, ncol) ;<br> STEND_CLUBB:mdims = 1 ;<br> STEND_CLUBB:units = "J/(kg s)" ;<br> STEND_CLUBB:long_name = "Static energy tendency" ;<br> STEND_CLUBB:cell_methods = "time: mean" ;<br><br>// global attributes:<br> :Conventions = "CF-1.0" ;<br> :source = "CAM" ;<br> :case = "SMS_D_Ln9.mpasa7p5_mpasa7p5_mg17.QPC6.derecho_intel.cam-outfrq9s.20230726_094231_iz24v6" ;<br> :logname = "jedwards" ;<br> :host = "derecho4" ;<br> :initial_file = "/glade/work/bdobbins/derecho/MPAS/source/MPASv8_CPU/init_7.5km_L58.cam/<a href="http://x1.10485762.init_fake_donotuse.nc" target="_blank">x1.10485762.init_fake_donotuse.nc</a>" ;<br> :topography_file = "bnd_topo" ;<br> :model_doi_url = "not_set" ;<br> :time_period_freq = "hour_1" ;<br>}<br></font></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Aug 4, 2023 at 9:39 AM Wei-Keng Liao <<a href="mailto:wkliao@northwestern.edu" target="_blank">wkliao@northwestern.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
Hi, Jim
<div><br>
</div>
<div>Can your provide the test program and the file header dumped by "ncdump -h", if that is available?</div>
<div>Also, what machine was used in the tests and its the parallel file system configuration is?</div>
<div>These can help diagnose.</div>
<div><br>
</div>
<div>
<div>Wei-keng </div>
<div><br>
<blockquote type="cite">
<div>On Aug 4, 2023, at 8:49 AM, Jim Edwards <<a href="mailto:jedwards@ucar.edu" target="_blank">jedwards@ucar.edu</a>> wrote:</div>
<br>
<div>
<div dir="ltr">
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
I am using ncmpi_iput_varn and ncmpi_wait_all to write output from my model. I have a test program that does nothing but test the
<br>
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
performance of the write operation. Attached is a plot of performance in the model and in the standalone application. I'm looking for
<br>
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
clues as to why the model performance is scaling so badly with the number of variables but the standalone program performance is fine.
<br>
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
<br>
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
<br clear="all">
</div>
<br>
<span class="gmail_signature_prefix">-- </span><br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div>
<div>Jim Edwards<br>
<br>
</div>
<font size="1">CESM Software Engineer<br>
</font></div>
<font size="1">National Center for Atmospheric Research<br>
</font></div>
<font size="1">Boulder, CO</font> <br>
</div>
</div>
</div>
<span id="m_2599353008599558804m_3752102915513421155m_3416875553179895269m_1419731810550872517m_-4661512969452605678cid:f_lkwn4kse0"><Screenshot 2023-07-27 at 11.49.03 AM.png></span></div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote></div><br clear="all"><br><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div><div>Jim Edwards<br><br></div><font size="1">CESM Software Engineer<br></font></div><font size="1">National Center for Atmospheric Research<br></font></div><font size="1">Boulder, CO</font> <br></div></div>