performance issue
Jim Edwards
jedwards at ucar.edu
Fri Aug 4 11:05:36 CDT 2023
I am using the new ncar system, derecho
<https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&cad=rja&uact=8&ved=2ahUKEwigtcytrsOAAxWXHjQIHVoDD6YQFnoECBcQAQ&url=https%3A%2F%2Farc.ucar.edu%2Fknowledge_base%2F74317833&usg=AOvVaw2aXlWuOfLnua7fFmIgvfoV&opi=89978449>,
which has a lustre parallel file system.
Looking at the difference between the two headers below makes me wonder if
the issue is with variable attributes?
Here are the steps to reproduce the standalone pioperf:
git clone https://github.com/NCAR/ParallelIO
cd ParallelIO
mkdir bld
cd bld
module purge
module load cesmdev/1.0
module load ncarenv/23.06
module load intel cray-mpich netcdf-mpi parallel-netcdf cmake
CC=cc FC=ftn cmake ../
make pioperf
cd $SCRATCH
cp -r /glade/scratch/jedwards/piodecomps .
cd piodecomps/run
[ edit file run32768.py to change pioperfpaths and account #]
qsub run32768.py
Here are the steps to reproduce the cesm case:
git clone https://github.com/ESCOMP/CESM.git
cd CESM
./manage_externals/checkout_externals
cd cime/scripts
./create_test
SMS_Ln9.mpasa7p5_mpasa7p5_mg17.QPC6.derecho_intel.cam-outfrq9s --no-build
cd $SCRATCH/SMS_Ln9.mpasa7p5_mpasa7p5_mg17.QPC6.derecho_intel.cam-outfrq9s.*
./xmlchange CAM_CONFIG_OPTS=" -phys cam6 -aquaplanet -analytic_ic
-nlev=58 "
./xmlchange PIO_NETCDF_FORMAT=64bit_data
./xmlchange STOP_N=2 --file env_test.xml
cp
/glade/derecho/scratch/jedwards/SMS_D_Ln9.mpasa7p5_mpasa7p5_mg17.QPC6.derecho_intel.cam-outfrq9s.20230726_094231_iz24v6/user_nl_cam.64
user_nl_cam
qcmd -- ./case.build
./case.submit
once the job has completed:
grep wrapup timing/cesm.ESMF*
Note that in this process we found some bugs in the esmf 8.5.0 release that
don't allow the profiler tools to work. If you want to use profiling let
me know and I can provide further instructions.
Here is the header dump of the pioperf file:
netcdf pioperf.2-0256-1 {
dimensions:
dim000001 = 10485762 ;
dim000002 = 58 ;
time = UNLIMITED ; // (1 currently)
variables:
double vard0001(time, dim000002, dim000001) ;
vard0001:_FillValue = 9.96920996838687e+36 ;
double vard0002(time, dim000002, dim000001) ;
vard0002:_FillValue = 9.96920996838687e+36 ;
double vard0003(time, dim000002, dim000001) ;
vard0003:_FillValue = 9.96920996838687e+36 ;
double vard0004(time, dim000002, dim000001) ;
vard0004:_FillValue = 9.96920996838687e+36 ;
double vard0005(time, dim000002, dim000001) ;
vard0005:_FillValue = 9.96920996838687e+36 ;
double vard0006(time, dim000002, dim000001) ;
vard0006:_FillValue = 9.96920996838687e+36 ;
double vard0007(time, dim000002, dim000001) ;
vard0007:_FillValue = 9.96920996838687e+36 ;
double vard0008(time, dim000002, dim000001) ;
vard0008:_FillValue = 9.96920996838687e+36 ;
double vard0009(time, dim000002, dim000001) ;
vard0009:_FillValue = 9.96920996838687e+36 ;
double vard0010(time, dim000002, dim000001) ;
vard0010:_FillValue = 9.96920996838687e+36 ;
double vard0011(time, dim000002, dim000001) ;
vard0011:_FillValue = 9.96920996838687e+36 ;
double vard0012(time, dim000002, dim000001) ;
vard0012:_FillValue = 9.96920996838687e+36 ;
double vard0013(time, dim000002, dim000001) ;
vard0013:_FillValue = 9.96920996838687e+36 ;
double vard0014(time, dim000002, dim000001) ;
vard0014:_FillValue = 9.96920996838687e+36 ;
double vard0015(time, dim000002, dim000001) ;
vard0015:_FillValue = 9.96920996838687e+36 ;
double vard0016(time, dim000002, dim000001) ;
vard0016:_FillValue = 9.96920996838687e+36 ;
double vard0017(time, dim000002, dim000001) ;
vard0017:_FillValue = 9.96920996838687e+36 ;
double vard0018(time, dim000002, dim000001) ;
vard0018:_FillValue = 9.96920996838687e+36 ;
double vard0019(time, dim000002, dim000001) ;
vard0019:_FillValue = 9.96920996838687e+36 ;
double vard0020(time, dim000002, dim000001) ;
vard0020:_FillValue = 9.96920996838687e+36 ;
double vard0021(time, dim000002, dim000001) ;
vard0021:_FillValue = 9.96920996838687e+36 ;
double vard0022(time, dim000002, dim000001) ;
vard0022:_FillValue = 9.96920996838687e+36 ;
double vard0023(time, dim000002, dim000001) ;
vard0023:_FillValue = 9.96920996838687e+36 ;
double vard0024(time, dim000002, dim000001) ;
vard0024:_FillValue = 9.96920996838687e+36 ;
double vard0025(time, dim000002, dim000001) ;
vard0025:_FillValue = 9.96920996838687e+36 ;
double vard0026(time, dim000002, dim000001) ;
vard0026:_FillValue = 9.96920996838687e+36 ;
double vard0027(time, dim000002, dim000001) ;
vard0027:_FillValue = 9.96920996838687e+36 ;
double vard0028(time, dim000002, dim000001) ;
vard0028:_FillValue = 9.96920996838687e+36 ;
double vard0029(time, dim000002, dim000001) ;
vard0029:_FillValue = 9.96920996838687e+36 ;
double vard0030(time, dim000002, dim000001) ;
vard0030:_FillValue = 9.96920996838687e+36 ;
double vard0031(time, dim000002, dim000001) ;
vard0031:_FillValue = 9.96920996838687e+36 ;
double vard0032(time, dim000002, dim000001) ;
vard0032:_FillValue = 9.96920996838687e+36 ;
double vard0033(time, dim000002, dim000001) ;
vard0033:_FillValue = 9.96920996838687e+36 ;
double vard0034(time, dim000002, dim000001) ;
vard0034:_FillValue = 9.96920996838687e+36 ;
double vard0035(time, dim000002, dim000001) ;
vard0035:_FillValue = 9.96920996838687e+36 ;
double vard0036(time, dim000002, dim000001) ;
vard0036:_FillValue = 9.96920996838687e+36 ;
double vard0037(time, dim000002, dim000001) ;
vard0037:_FillValue = 9.96920996838687e+36 ;
double vard0038(time, dim000002, dim000001) ;
vard0038:_FillValue = 9.96920996838687e+36 ;
double vard0039(time, dim000002, dim000001) ;
vard0039:_FillValue = 9.96920996838687e+36 ;
double vard0040(time, dim000002, dim000001) ;
vard0040:_FillValue = 9.96920996838687e+36 ;
double vard0041(time, dim000002, dim000001) ;
vard0041:_FillValue = 9.96920996838687e+36 ;
double vard0042(time, dim000002, dim000001) ;
vard0042:_FillValue = 9.96920996838687e+36 ;
double vard0043(time, dim000002, dim000001) ;
vard0043:_FillValue = 9.96920996838687e+36 ;
double vard0044(time, dim000002, dim000001) ;
vard0044:_FillValue = 9.96920996838687e+36 ;
double vard0045(time, dim000002, dim000001) ;
vard0045:_FillValue = 9.96920996838687e+36 ;
double vard0046(time, dim000002, dim000001) ;
vard0046:_FillValue = 9.96920996838687e+36 ;
double vard0047(time, dim000002, dim000001) ;
vard0047:_FillValue = 9.96920996838687e+36 ;
double vard0048(time, dim000002, dim000001) ;
vard0048:_FillValue = 9.96920996838687e+36 ;
double vard0049(time, dim000002, dim000001) ;
vard0049:_FillValue = 9.96920996838687e+36 ;
double vard0050(time, dim000002, dim000001) ;
vard0050:_FillValue = 9.96920996838687e+36 ;
double vard0051(time, dim000002, dim000001) ;
vard0051:_FillValue = 9.96920996838687e+36 ;
double vard0052(time, dim000002, dim000001) ;
vard0052:_FillValue = 9.96920996838687e+36 ;
double vard0053(time, dim000002, dim000001) ;
vard0053:_FillValue = 9.96920996838687e+36 ;
double vard0054(time, dim000002, dim000001) ;
vard0054:_FillValue = 9.96920996838687e+36 ;
double vard0055(time, dim000002, dim000001) ;
vard0055:_FillValue = 9.96920996838687e+36 ;
double vard0056(time, dim000002, dim000001) ;
vard0056:_FillValue = 9.96920996838687e+36 ;
double vard0057(time, dim000002, dim000001) ;
vard0057:_FillValue = 9.96920996838687e+36 ;
double vard0058(time, dim000002, dim000001) ;
vard0058:_FillValue = 9.96920996838687e+36 ;
double vard0059(time, dim000002, dim000001) ;
vard0059:_FillValue = 9.96920996838687e+36 ;
double vard0060(time, dim000002, dim000001) ;
vard0060:_FillValue = 9.96920996838687e+36 ;
double vard0061(time, dim000002, dim000001) ;
vard0061:_FillValue = 9.96920996838687e+36 ;
double vard0062(time, dim000002, dim000001) ;
vard0062:_FillValue = 9.96920996838687e+36 ;
double vard0063(time, dim000002, dim000001) ;
vard0063:_FillValue = 9.96920996838687e+36 ;
double vard0064(time, dim000002, dim000001) ;
vard0064:_FillValue = 9.96920996838687e+36 ;
}
Here is the header dump of the cesm file:
netcdf
SMS_D_Ln9.mpasa7p5_mpasa7p5_mg17.QPC6.derecho_intel.cam-outfrq9s.20230726_094231_iz24v6.cam.h0.0001-01-01-00000
{
dimensions:
ncol = 10485762 ;
time = UNLIMITED ; // (1 currently)
nbnd = 2 ;
chars = 8 ;
lev = 58 ;
ilev = 59 ;
variables:
double lat(ncol) ;
lat:long_name = "latitude" ;
lat:units = "degrees_north" ;
double lon(ncol) ;
lon:long_name = "longitude" ;
lon:units = "degrees_east" ;
double areawt(ncol) ;
areawt:long_name = "physics column area weight" ;
double area(ncol) ;
area:long_name = "physics column areas" ;
double lev(lev) ;
lev:long_name = "zeta level at vertical midpoints" ;
lev:units = "m" ;
double ilev(ilev) ;
ilev:long_name = "zeta level at vertical interfaces" ;
ilev:units = "m" ;
double time(time) ;
time:long_name = "time" ;
time:units = "days since 0001-01-01 00:00:00" ;
time:calendar = "noleap" ;
time:bounds = "time_bnds" ;
int date(time) ;
date:long_name = "current date (YYYYMMDD)" ;
int datesec(time) ;
datesec:long_name = "current seconds of current date" ;
double time_bnds(time, nbnd) ;
time_bnds:long_name = "time interval endpoints" ;
char date_written(time, chars) ;
char time_written(time, chars) ;
int ndbase ;
ndbase:long_name = "base day" ;
int nsbase ;
nsbase:long_name = "seconds of base day" ;
int nbdate ;
nbdate:long_name = "base date (YYYYMMDD)" ;
int nbsec ;
nbsec:long_name = "seconds of base date" ;
int mdt ;
mdt:long_name = "timestep" ;
mdt:units = "s" ;
int ndcur(time) ;
ndcur:long_name = "current day (from base day)" ;
int nscur(time) ;
nscur:long_name = "current seconds of current day" ;
double co2vmr(time) ;
co2vmr:long_name = "co2 volume mixing ratio" ;
double ch4vmr(time) ;
ch4vmr:long_name = "ch4 volume mixing ratio" ;
double n2ovmr(time) ;
n2ovmr:long_name = "n2o volume mixing ratio" ;
double f11vmr(time) ;
f11vmr:long_name = "f11 volume mixing ratio" ;
double f12vmr(time) ;
f12vmr:long_name = "f12 volume mixing ratio" ;
double sol_tsi(time) ;
sol_tsi:long_name = "total solar irradiance" ;
sol_tsi:units = "W/m2" ;
int nsteph(time) ;
nsteph:long_name = "current timestep" ;
double ADRAIN(time, lev, ncol) ;
ADRAIN:mdims = 1 ;
ADRAIN:units = "m" ;
ADRAIN:long_name = "Average rain effective Diameter" ;
ADRAIN:cell_methods = "time: mean" ;
double ADSNOW(time, lev, ncol) ;
ADSNOW:mdims = 1 ;
ADSNOW:units = "m" ;
ADSNOW:long_name = "Average snow effective Diameter" ;
ADSNOW:cell_methods = "time: mean" ;
double ANRAIN(time, lev, ncol) ;
ANRAIN:mdims = 1 ;
ANRAIN:units = "m-3" ;
ANRAIN:long_name = "Average rain number conc" ;
ANRAIN:cell_methods = "time: mean" ;
double ANSNOW(time, lev, ncol) ;
ANSNOW:mdims = 1 ;
ANSNOW:units = "m-3" ;
ANSNOW:long_name = "Average snow number conc" ;
ANSNOW:cell_methods = "time: mean" ;
double AQRAIN(time, lev, ncol) ;
AQRAIN:mdims = 1 ;
AQRAIN:units = "kg/kg" ;
AQRAIN:long_name = "Average rain mixing ratio" ;
AQRAIN:cell_methods = "time: mean" ;
double AQSNOW(time, lev, ncol) ;
AQSNOW:mdims = 1 ;
AQSNOW:units = "kg/kg" ;
AQSNOW:long_name = "Average snow mixing ratio" ;
AQSNOW:cell_methods = "time: mean" ;
double AREI(time, lev, ncol) ;
AREI:mdims = 1 ;
AREI:units = "Micron" ;
AREI:long_name = "Average ice effective radius" ;
AREI:cell_methods = "time: mean" ;
double AREL(time, lev, ncol) ;
AREL:mdims = 1 ;
AREL:units = "Micron" ;
AREL:long_name = "Average droplet effective radius" ;
AREL:cell_methods = "time: mean" ;
double AWNC(time, lev, ncol) ;
AWNC:mdims = 1 ;
AWNC:units = "m-3" ;
AWNC:long_name = "Average cloud water number conc" ;
AWNC:cell_methods = "time: mean" ;
double AWNI(time, lev, ncol) ;
AWNI:mdims = 1 ;
AWNI:units = "m-3" ;
AWNI:long_name = "Average cloud ice number conc" ;
AWNI:cell_methods = "time: mean" ;
double CCN3(time, lev, ncol) ;
CCN3:mdims = 1 ;
CCN3:units = "#/cm3" ;
CCN3:long_name = "CCN concentration at S=0.1%" ;
CCN3:cell_methods = "time: mean" ;
double CLDICE(time, lev, ncol) ;
CLDICE:mdims = 1 ;
CLDICE:units = "kg/kg" ;
CLDICE:mixing_ratio = "wet" ;
CLDICE:long_name = "Grid box averaged cloud ice amount" ;
CLDICE:cell_methods = "time: mean" ;
double CLDLIQ(time, lev, ncol) ;
CLDLIQ:mdims = 1 ;
CLDLIQ:units = "kg/kg" ;
CLDLIQ:mixing_ratio = "wet" ;
CLDLIQ:long_name = "Grid box averaged cloud liquid amount" ;
CLDLIQ:cell_methods = "time: mean" ;
double CLOUD(time, lev, ncol) ;
CLOUD:mdims = 1 ;
CLOUD:units = "fraction" ;
CLOUD:long_name = "Cloud fraction" ;
CLOUD:cell_methods = "time: mean" ;
double CLOUDCOVER_CLUBB(time, lev, ncol) ;
CLOUDCOVER_CLUBB:mdims = 1 ;
CLOUDCOVER_CLUBB:units = "fraction" ;
CLOUDCOVER_CLUBB:long_name = "Cloud Cover" ;
CLOUDCOVER_CLUBB:cell_methods = "time: mean" ;
double CLOUDFRAC_CLUBB(time, lev, ncol) ;
CLOUDFRAC_CLUBB:mdims = 1 ;
CLOUDFRAC_CLUBB:units = "fraction" ;
CLOUDFRAC_CLUBB:long_name = "Cloud Fraction" ;
CLOUDFRAC_CLUBB:cell_methods = "time: mean" ;
double CONCLD(time, lev, ncol) ;
CONCLD:mdims = 1 ;
CONCLD:units = "fraction" ;
CONCLD:long_name = "Convective cloud cover" ;
CONCLD:cell_methods = "time: mean" ;
double DCQ(time, lev, ncol) ;
DCQ:mdims = 1 ;
DCQ:units = "kg/kg/s" ;
DCQ:long_name = "Q tendency due to moist processes" ;
DCQ:cell_methods = "time: mean" ;
double DMS(time, lev, ncol) ;
DMS:mdims = 1 ;
DMS:units = "mol/mol" ;
DMS:mixing_ratio = "dry" ;
DMS:long_name = "DMS concentration" ;
DMS:cell_methods = "time: mean" ;
double DTCOND(time, lev, ncol) ;
DTCOND:mdims = 1 ;
DTCOND:units = "K/s" ;
DTCOND:long_name = "T tendency - moist processes" ;
DTCOND:cell_methods = "time: mean" ;
double DTV(time, lev, ncol) ;
DTV:mdims = 1 ;
DTV:units = "K/s" ;
DTV:long_name = "T vertical diffusion" ;
DTV:cell_methods = "time: mean" ;
double DTWR_DMS(time, lev, ncol) ;
DTWR_DMS:mdims = 1 ;
DTWR_DMS:units = "kg/kg/s" ;
DTWR_DMS:long_name = "wet removal Neu scheme tendency" ;
DTWR_DMS:cell_methods = "time: mean" ;
double DTWR_H2O2(time, lev, ncol) ;
DTWR_H2O2:mdims = 1 ;
DTWR_H2O2:units = "kg/kg/s" ;
DTWR_H2O2:long_name = "wet removal Neu scheme tendency" ;
DTWR_H2O2:cell_methods = "time: mean" ;
double DTWR_H2SO4(time, lev, ncol) ;
DTWR_H2SO4:mdims = 1 ;
DTWR_H2SO4:units = "kg/kg/s" ;
DTWR_H2SO4:long_name = "wet removal Neu scheme tendency" ;
DTWR_H2SO4:cell_methods = "time: mean" ;
double DTWR_SO2(time, lev, ncol) ;
DTWR_SO2:mdims = 1 ;
DTWR_SO2:units = "kg/kg/s" ;
DTWR_SO2:long_name = "wet removal Neu scheme tendency" ;
DTWR_SO2:cell_methods = "time: mean" ;
double DTWR_SOAG(time, lev, ncol) ;
DTWR_SOAG:mdims = 1 ;
DTWR_SOAG:units = "kg/kg/s" ;
DTWR_SOAG:long_name = "wet removal Neu scheme tendency" ;
DTWR_SOAG:cell_methods = "time: mean" ;
double FICE(time, lev, ncol) ;
FICE:mdims = 1 ;
FICE:units = "fraction" ;
FICE:long_name = "Fractional ice content within cloud" ;
FICE:cell_methods = "time: mean" ;
double FREQI(time, lev, ncol) ;
FREQI:mdims = 1 ;
FREQI:units = "fraction" ;
FREQI:long_name = "Fractional occurrence of ice" ;
FREQI:cell_methods = "time: mean" ;
double FREQL(time, lev, ncol) ;
FREQL:mdims = 1 ;
FREQL:units = "fraction" ;
FREQL:long_name = "Fractional occurrence of liquid" ;
FREQL:cell_methods = "time: mean" ;
double FREQR(time, lev, ncol) ;
FREQR:mdims = 1 ;
FREQR:units = "fraction" ;
FREQR:long_name = "Fractional occurrence of rain" ;
FREQR:cell_methods = "time: mean" ;
double FREQS(time, lev, ncol) ;
FREQS:mdims = 1 ;
FREQS:units = "fraction" ;
FREQS:long_name = "Fractional occurrence of snow" ;
FREQS:cell_methods = "time: mean" ;
double H2O(time, lev, ncol) ;
H2O:mdims = 1 ;
H2O:units = "mol/mol" ;
H2O:long_name = "water vapor concentration" ;
H2O:cell_methods = "time: mean" ;
double H2O2(time, lev, ncol) ;
H2O2:mdims = 1 ;
H2O2:units = "mol/mol" ;
H2O2:mixing_ratio = "dry" ;
H2O2:long_name = "H2O2 concentration" ;
H2O2:cell_methods = "time: mean" ;
double H2SO4(time, lev, ncol) ;
H2SO4:mdims = 1 ;
H2SO4:units = "mol/mol" ;
H2SO4:mixing_ratio = "dry" ;
H2SO4:long_name = "H2SO4 concentration" ;
H2SO4:cell_methods = "time: mean" ;
double ICIMR(time, lev, ncol) ;
ICIMR:mdims = 1 ;
ICIMR:units = "kg/kg" ;
ICIMR:long_name = "Prognostic in-cloud ice mixing ratio" ;
ICIMR:cell_methods = "time: mean" ;
double ICWMR(time, lev, ncol) ;
ICWMR:mdims = 1 ;
ICWMR:units = "kg/kg" ;
ICWMR:long_name = "Prognostic in-cloud water mixing ratio" ;
ICWMR:cell_methods = "time: mean" ;
double IWC(time, lev, ncol) ;
IWC:mdims = 1 ;
IWC:units = "kg/m3" ;
IWC:long_name = "Grid box average ice water content" ;
IWC:cell_methods = "time: mean" ;
double NUMICE(time, lev, ncol) ;
NUMICE:mdims = 1 ;
NUMICE:units = "1/kg" ;
NUMICE:mixing_ratio = "wet" ;
NUMICE:long_name = "Grid box averaged cloud ice number" ;
NUMICE:cell_methods = "time: mean" ;
double NUMLIQ(time, lev, ncol) ;
NUMLIQ:mdims = 1 ;
NUMLIQ:units = "1/kg" ;
NUMLIQ:mixing_ratio = "wet" ;
NUMLIQ:long_name = "Grid box averaged cloud liquid number" ;
NUMLIQ:cell_methods = "time: mean" ;
double NUMRAI(time, lev, ncol) ;
NUMRAI:mdims = 1 ;
NUMRAI:units = "1/kg" ;
NUMRAI:mixing_ratio = "wet" ;
NUMRAI:long_name = "Grid box averaged rain number" ;
NUMRAI:cell_methods = "time: mean" ;
double NUMSNO(time, lev, ncol) ;
NUMSNO:mdims = 1 ;
NUMSNO:units = "1/kg" ;
NUMSNO:mixing_ratio = "wet" ;
NUMSNO:long_name = "Grid box averaged snow number" ;
NUMSNO:cell_methods = "time: mean" ;
double OMEGA(time, lev, ncol) ;
OMEGA:mdims = 1 ;
OMEGA:units = "Pa/s" ;
OMEGA:long_name = "Vertical velocity (pressure)" ;
OMEGA:cell_methods = "time: mean" ;
double OMEGAT(time, lev, ncol) ;
OMEGAT:mdims = 1 ;
OMEGAT:units = "K Pa/s" ;
OMEGAT:long_name = "Vertical heat flux" ;
OMEGAT:cell_methods = "time: mean" ;
double PDEL(time, lev, ncol) ;
PDEL:mdims = 1 ;
PDEL:units = "Pa" ;
PDEL:long_name = "Pressure difference between levels" ;
PDEL:cell_methods = "time: mean" ;
double PMID(time, lev, ncol) ;
PMID:mdims = 1 ;
PMID:units = "Pa" ;
PMID:long_name = "Pressure at layer midpoints" ;
PMID:cell_methods = "time: mean" ;
double Q(time, lev, ncol) ;
Q:mdims = 1 ;
Q:units = "kg/kg" ;
Q:mixing_ratio = "wet" ;
Q:long_name = "Specific humidity" ;
Q:cell_methods = "time: mean" ;
double QRL(time, lev, ncol) ;
QRL:mdims = 1 ;
QRL:Sampling_Sequence = "rad_lwsw" ;
QRL:units = "K/s" ;
QRL:long_name = "Longwave heating rate" ;
QRL:cell_methods = "time: mean" ;
double QRS(time, lev, ncol) ;
QRS:mdims = 1 ;
QRS:Sampling_Sequence = "rad_lwsw" ;
QRS:units = "K/s" ;
QRS:long_name = "Solar heating rate" ;
QRS:cell_methods = "time: mean" ;
double RAINQM(time, lev, ncol) ;
RAINQM:mdims = 1 ;
RAINQM:units = "kg/kg" ;
RAINQM:mixing_ratio = "wet" ;
RAINQM:long_name = "Grid box averaged rain amount" ;
RAINQM:cell_methods = "time: mean" ;
double RCMINLAYER_CLUBB(time, lev, ncol) ;
RCMINLAYER_CLUBB:mdims = 1 ;
RCMINLAYER_CLUBB:units = "kg/kg" ;
RCMINLAYER_CLUBB:long_name = "Cloud Water in Layer" ;
RCMINLAYER_CLUBB:cell_methods = "time: mean" ;
double RCMTEND_CLUBB(time, lev, ncol) ;
RCMTEND_CLUBB:mdims = 1 ;
RCMTEND_CLUBB:units = "kg/kg /s" ;
RCMTEND_CLUBB:long_name = "Cloud Liquid Water Tendency" ;
RCMTEND_CLUBB:cell_methods = "time: mean" ;
double RCM_CLUBB(time, lev, ncol) ;
RCM_CLUBB:mdims = 1 ;
RCM_CLUBB:units = "kg/kg" ;
RCM_CLUBB:long_name = "Cloud Water Mixing Ratio" ;
RCM_CLUBB:cell_methods = "time: mean" ;
double RELHUM(time, lev, ncol) ;
RELHUM:mdims = 1 ;
RELHUM:units = "percent" ;
RELHUM:long_name = "Relative humidity" ;
RELHUM:cell_methods = "time: mean" ;
double RELVAR(time, lev, ncol) ;
RELVAR:mdims = 1 ;
RELVAR:units = "-" ;
RELVAR:long_name = "Relative cloud water variance" ;
RELVAR:cell_methods = "time: mean" ;
double RHO_CLUBB(time, lev, ncol) ;
RHO_CLUBB:mdims = 1 ;
RHO_CLUBB:units = "kg/m3" ;
RHO_CLUBB:long_name = "Air Density" ;
RHO_CLUBB:cell_methods = "time: mean" ;
double RIMTEND_CLUBB(time, lev, ncol) ;
RIMTEND_CLUBB:mdims = 1 ;
RIMTEND_CLUBB:units = "kg/kg /s" ;
RIMTEND_CLUBB:long_name = "Cloud Ice Tendency" ;
RIMTEND_CLUBB:cell_methods = "time: mean" ;
double RTM_CLUBB(time, lev, ncol) ;
RTM_CLUBB:mdims = 1 ;
RTM_CLUBB:units = "kg/kg" ;
RTM_CLUBB:long_name = "Total Water Mixing Ratio" ;
RTM_CLUBB:cell_methods = "time: mean" ;
double RTP2_ZT_CLUBB(time, lev, ncol) ;
RTP2_ZT_CLUBB:mdims = 1 ;
RTP2_ZT_CLUBB:units = "kg^2/kg^2" ;
RTP2_ZT_CLUBB:long_name = "Moisture Variance on zt grid" ;
RTP2_ZT_CLUBB:cell_methods = "time: mean" ;
double RVMTEND_CLUBB(time, lev, ncol) ;
RVMTEND_CLUBB:mdims = 1 ;
RVMTEND_CLUBB:units = "kg/kg /s" ;
RVMTEND_CLUBB:long_name = "Water vapor tendency" ;
RVMTEND_CLUBB:cell_methods = "time: mean" ;
double SNOWQM(time, lev, ncol) ;
SNOWQM:mdims = 1 ;
SNOWQM:units = "kg/kg" ;
SNOWQM:mixing_ratio = "wet" ;
SNOWQM:long_name = "Grid box averaged snow amount" ;
SNOWQM:cell_methods = "time: mean" ;
double SO2(time, lev, ncol) ;
SO2:mdims = 1 ;
SO2:units = "mol/mol" ;
SO2:mixing_ratio = "dry" ;
SO2:long_name = "SO2 concentration" ;
SO2:cell_methods = "time: mean" ;
double SOAE(time, lev, ncol) ;
SOAE:mdims = 1 ;
SOAE:units = "mol/mol" ;
SOAE:mixing_ratio = "dry" ;
SOAE:long_name = "SOAE concentration" ;
SOAE:cell_methods = "time: mean" ;
double SOAG(time, lev, ncol) ;
SOAG:mdims = 1 ;
SOAG:units = "mol/mol" ;
SOAG:mixing_ratio = "dry" ;
SOAG:long_name = "SOAG concentration" ;
SOAG:cell_methods = "time: mean" ;
double STEND_CLUBB(time, lev, ncol) ;
STEND_CLUBB:mdims = 1 ;
STEND_CLUBB:units = "J/(kg s)" ;
STEND_CLUBB:long_name = "Static energy tendency" ;
STEND_CLUBB:cell_methods = "time: mean" ;
// global attributes:
:Conventions = "CF-1.0" ;
:source = "CAM" ;
:case =
"SMS_D_Ln9.mpasa7p5_mpasa7p5_mg17.QPC6.derecho_intel.cam-outfrq9s.20230726_094231_iz24v6"
;
:logname = "jedwards" ;
:host = "derecho4" ;
:initial_file =
"/glade/work/bdobbins/derecho/MPAS/source/MPASv8_CPU/init_7.5km_L58.cam/
x1.10485762.init_fake_donotuse.nc" ;
:topography_file = "bnd_topo" ;
:model_doi_url = "not_set" ;
:time_period_freq = "hour_1" ;
}
On Fri, Aug 4, 2023 at 9:39 AM Wei-Keng Liao <wkliao at northwestern.edu>
wrote:
> Hi, Jim
>
> Can your provide the test program and the file header dumped by "ncdump
> -h", if that is available?
> Also, what machine was used in the tests and its the parallel file system
> configuration is?
> These can help diagnose.
>
> Wei-keng
>
> On Aug 4, 2023, at 8:49 AM, Jim Edwards <jedwards at ucar.edu> wrote:
>
> I am using ncmpi_iput_varn and ncmpi_wait_all to write output from my
> model. I have a test program that does nothing but test the
> performance of the write operation. Attached is a plot of performance in
> the model and in the standalone application. I'm looking for
> clues as to why the model performance is scaling so badly with the number
> of variables but the standalone program performance is fine.
>
>
>
> --
> Jim Edwards
>
> CESM Software Engineer
> National Center for Atmospheric Research
> Boulder, CO
> <Screenshot 2023-07-27 at 11.49.03 AM.png>
>
>
>
--
Jim Edwards
CESM Software Engineer
National Center for Atmospheric Research
Boulder, CO
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/parallel-netcdf/attachments/20230804/9cd0a7d5/attachment-0001.html>
More information about the parallel-netcdf
mailing list