<div dir="ltr"><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">Yes, _enddef is called only once. <br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d"><br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d"><a href="https://github.com/NCAR/ParallelIO/blob/main/src/clib/pio_getput_int.c#L128" target="_blank">Here</a> is the code that writes attributes. <a href="https://github.com/NCAR/ParallelIO/blob/main/src/clib/pio_darray_int.c#L661" target="_blank"> Here</a> is where variables are written. <br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d"><br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">ncoffsets -sg <a href="http://pioperf.2-0256-1.nc" target="_blank">pioperf.2-0256-1.nc</a><br>netcdf <a href="http://pioperf.2-0256-1.nc" target="_blank">pioperf.2-0256-1.nc</a> {<br>// file format: CDF-5<br><br>file header:<br> size = 7804 bytes<br> extent = 8192 bytes<br><br>dimensions:<br> dim000001 = 10485762<br> dim000002 = 58<br> time = UNLIMITED // (1 currently)<br><br>record variables:<br> double vard0001(time, dim000002, dim000001):<br> start file offset = 8192 (0th record)<br> end file offset = 4865401760 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 388<br> double vard0002(time, dim000002, dim000001):<br> start file offset = 4865401760 (0th record)<br> end file offset = 9730795328 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0003(time, dim000002, dim000001):<br> start file offset = 9730795328 (0th record)<br> end file offset = 14596188896 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0004(time, dim000002, dim000001):<br> start file offset = 14596188896 (0th record)<br> end file offset = 19461582464 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0005(time, dim000002, dim000001):<br> start file offset = 19461582464 (0th record)<br> end file offset = 24326976032 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0006(time, dim000002, dim000001):<br> start file offset = 24326976032 (0th record)<br> end file offset = 29192369600 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0007(time, dim000002, dim000001):<br> start file offset = 29192369600 (0th record)<br> end file offset = 34057763168 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0008(time, dim000002, dim000001):<br> start file offset = 34057763168 (0th record)<br> end file offset = 38923156736 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0009(time, dim000002, dim000001):<br> start file offset = 38923156736 (0th record)<br> end file offset = 43788550304 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0010(time, dim000002, dim000001):<br> start file offset = 43788550304 (0th record)<br> end file offset = 48653943872 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0011(time, dim000002, dim000001):<br> start file offset = 48653943872 (0th record)<br> end file offset = 53519337440 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0012(time, dim000002, dim000001):<br> start file offset = 53519337440 (0th record)<br> end file offset = 58384731008 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0013(time, dim000002, dim000001):<br> start file offset = 58384731008 (0th record)<br> end file offset = 63250124576 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0014(time, dim000002, dim000001):<br> start file offset = 63250124576 (0th record)<br> end file offset = 68115518144 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0015(time, dim000002, dim000001):<br> start file offset = 68115518144 (0th record)<br> end file offset = 72980911712 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0016(time, dim000002, dim000001):<br> start file offset = 72980911712 (0th record)<br> end file offset = 77846305280 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0017(time, dim000002, dim000001):<br> start file offset = 77846305280 (0th record)<br> end file offset = 82711698848 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0018(time, dim000002, dim000001):<br> start file offset = 82711698848 (0th record)<br> end file offset = 87577092416 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0019(time, dim000002, dim000001):<br> start file offset = 87577092416 (0th record)<br> end file offset = 92442485984 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0020(time, dim000002, dim000001):<br> start file offset = 92442485984 (0th record)<br> end file offset = 97307879552 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0021(time, dim000002, dim000001):<br> start file offset = 97307879552 (0th record)<br> end file offset =102173273120 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0022(time, dim000002, dim000001):<br> start file offset =102173273120 (0th record)<br> end file offset =107038666688 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0023(time, dim000002, dim000001):<br> start file offset =107038666688 (0th record)<br> end file offset =111904060256 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0024(time, dim000002, dim000001):<br> start file offset =111904060256 (0th record)<br> end file offset =116769453824 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0025(time, dim000002, dim000001):<br> start file offset =116769453824 (0th record)<br> end file offset =121634847392 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0026(time, dim000002, dim000001):<br> start file offset =121634847392 (0th record)<br> end file offset =126500240960 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0027(time, dim000002, dim000001):<br> start file offset =126500240960 (0th record)<br> end file offset =131365634528 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0028(time, dim000002, dim000001):<br> start file offset =131365634528 (0th record)<br> end file offset =136231028096 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0029(time, dim000002, dim000001):<br> start file offset =136231028096 (0th record)<br> end file offset =141096421664 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0030(time, dim000002, dim000001):<br> start file offset =141096421664 (0th record)<br> end file offset =145961815232 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0031(time, dim000002, dim000001):<br> start file offset =145961815232 (0th record)<br> end file offset =150827208800 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0032(time, dim000002, dim000001):<br> start file offset =150827208800 (0th record)<br> end file offset =155692602368 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0033(time, dim000002, dim000001):<br> start file offset =155692602368 (0th record)<br> end file offset =160557995936 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0034(time, dim000002, dim000001):<br> start file offset =160557995936 (0th record)<br> end file offset =165423389504 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0035(time, dim000002, dim000001):<br> start file offset =165423389504 (0th record)<br> end file offset =170288783072 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0036(time, dim000002, dim000001):<br> start file offset =170288783072 (0th record)<br> end file offset =175154176640 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0037(time, dim000002, dim000001):<br> start file offset =175154176640 (0th record)<br> end file offset =180019570208 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0038(time, dim000002, dim000001):<br> start file offset =180019570208 (0th record)<br> end file offset =184884963776 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0039(time, dim000002, dim000001):<br> start file offset =184884963776 (0th record)<br> end file offset =189750357344 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0040(time, dim000002, dim000001):<br> start file offset =189750357344 (0th record)<br> end file offset =194615750912 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0041(time, dim000002, dim000001):<br> start file offset =194615750912 (0th record)<br> end file offset =199481144480 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0042(time, dim000002, dim000001):<br> start file offset =199481144480 (0th record)<br> end file offset =204346538048 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0043(time, dim000002, dim000001):<br> start file offset =204346538048 (0th record)<br> end file offset =209211931616 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0044(time, dim000002, dim000001):<br> start file offset =209211931616 (0th record)<br> end file offset =214077325184 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0045(time, dim000002, dim000001):<br> start file offset =214077325184 (0th record)<br> end file offset =218942718752 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0046(time, dim000002, dim000001):<br> start file offset =218942718752 (0th record)<br> end file offset =223808112320 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0047(time, dim000002, dim000001):<br> start file offset =223808112320 (0th record)<br> end file offset =228673505888 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0048(time, dim000002, dim000001):<br> start file offset =228673505888 (0th record)<br> end file offset =233538899456 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0049(time, dim000002, dim000001):<br> start file offset =233538899456 (0th record)<br> end file offset =238404293024 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0050(time, dim000002, dim000001):<br> start file offset =238404293024 (0th record)<br> end file offset =243269686592 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0051(time, dim000002, dim000001):<br> start file offset =243269686592 (0th record)<br> end file offset =248135080160 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0052(time, dim000002, dim000001):<br> start file offset =248135080160 (0th record)<br> end file offset =253000473728 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0053(time, dim000002, dim000001):<br> start file offset =253000473728 (0th record)<br> end file offset =257865867296 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0054(time, dim000002, dim000001):<br> start file offset =257865867296 (0th record)<br> end file offset =262731260864 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0055(time, dim000002, dim000001):<br> start file offset =262731260864 (0th record)<br> end file offset =267596654432 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0056(time, dim000002, dim000001):<br> start file offset =267596654432 (0th record)<br> end file offset =272462048000 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0057(time, dim000002, dim000001):<br> start file offset =272462048000 (0th record)<br> end file offset =277327441568 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0058(time, dim000002, dim000001):<br> start file offset =277327441568 (0th record)<br> end file offset =282192835136 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0059(time, dim000002, dim000001):<br> start file offset =282192835136 (0th record)<br> end file offset =287058228704 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0060(time, dim000002, dim000001):<br> start file offset =287058228704 (0th record)<br> end file offset =291923622272 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0061(time, dim000002, dim000001):<br> start file offset =291923622272 (0th record)<br> end file offset =296789015840 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0062(time, dim000002, dim000001):<br> start file offset =296789015840 (0th record)<br> end file offset =301654409408 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0063(time, dim000002, dim000001):<br> start file offset =301654409408 (0th record)<br> end file offset =306519802976 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double vard0064(time, dim000002, dim000001):<br> start file offset =306519802976 (0th record)<br> end file offset =311385196544 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br>}</div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d"><br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d"> ncoffsets -sg run/<a href="http://SMS_D_Ln9.mpasa7p5_mpasa7p5_mg17.QPC6.derecho_intel.cam-outfrq9s.20230726_094231_iz24v6.cam.h0.0001-01-01-03600.nc" target="_blank">SMS_D_Ln9.mpasa7p5_mpasa7p5_mg17.QPC6.derecho_intel.cam-outfrq9s.20230726_094231_iz24v6.cam.h0.0001-01-01-03600.nc</a><br>netcdf run/<a href="http://SMS_D_Ln9.mpasa7p5_mpasa7p5_mg17.QPC6.derecho_intel.cam-outfrq9s.20230726_094231_iz24v6.cam.h0.0001-01-01-03600.nc" target="_blank">SMS_D_Ln9.mpasa7p5_mpasa7p5_mg17.QPC6.derecho_intel.cam-outfrq9s.20230726_094231_iz24v6.cam.h0.0001-01-01-03600.nc</a> {<br>// file format: CDF-5<br><br>file header:<br> size = 20620 bytes<br> extent = 16777216 bytes<br><br>dimensions:<br> ncol = 10485762<br> time = UNLIMITED // (1 currently)<br> nbnd = 2<br> chars = 8<br> lev = 58<br> ilev = 59<br><br>fixed-size variables:<br> double lat(ncol):<br> start file offset = 16777216<br> end file offset = 100663312<br> size in bytes = 83886096<br> gap from prev var = 16756596<br> double lon(ncol):<br> start file offset = 100663312<br> end file offset = 184549408<br> size in bytes = 83886096<br> gap from prev var = 0<br> double areawt(ncol):<br> start file offset = 184549408<br> end file offset = 268435504<br> size in bytes = 83886096<br> gap from prev var = 0<br> double area(ncol):<br> start file offset = 268435504<br> end file offset = 352321600<br> size in bytes = 83886096<br> gap from prev var = 0<br> double lev(lev):<br> start file offset = 352321600<br> end file offset = 352322064<br> size in bytes = 464<br> gap from prev var = 0<br> double ilev(ilev):<br> start file offset = 352322064<br> end file offset = 352322536<br> size in bytes = 472<br> gap from prev var = 0<br> int ndbase:<br> start file offset = 352322536<br> end file offset = 352322540<br> size in bytes = 4<br> gap from prev var = 0<br> int nsbase:<br> start file offset = 352322540<br> end file offset = 352322544<br> size in bytes = 4<br> gap from prev var = 0<br> int nbdate:<br> start file offset = 352322544<br> end file offset = 352322548<br> size in bytes = 4<br> gap from prev var = 0<br> int nbsec:<br> start file offset = 352322548<br> end file offset = 352322552<br> size in bytes = 4<br> gap from prev var = 0<br> int mdt:<br> start file offset = 352322552<br> end file offset = 352322556<br> size in bytes = 4<br> gap from prev var = 0<br><br>record variables:<br> double time(time):<br> start file offset = 352322556 (0th record)<br> end file offset = 352322564 (0th record)<br> size in bytes = 8 (of one record)<br> gap from prev var = 0<br> int date(time):<br> start file offset = 352322564 (0th record)<br> end file offset = 352322568 (0th record)<br> size in bytes = 4 (of one record)<br> gap from prev var = 0<br> int datesec(time):<br> start file offset = 352322568 (0th record)<br> end file offset = 352322572 (0th record)<br> size in bytes = 4 (of one record)<br> gap from prev var = 0<br> double time_bnds(time, nbnd):<br> start file offset = 352322572 (0th record)<br> end file offset = 352322588 (0th record)<br> size in bytes = 16 (of one record)<br> gap from prev var = 0<br> char date_written(time, chars):<br> start file offset = 352322588 (0th record)<br> end file offset = 352322596 (0th record)<br> size in bytes = 8 (of one record)<br> gap from prev var = 0<br> char time_written(time, chars):<br> start file offset = 352322596 (0th record)<br> end file offset = 352322604 (0th record)<br> size in bytes = 8 (of one record)<br> gap from prev var = 0<br> int ndcur(time):<br> start file offset = 352322604 (0th record)<br> end file offset = 352322608 (0th record)<br> size in bytes = 4 (of one record)<br> gap from prev var = 0<br> int nscur(time):<br> start file offset = 352322608 (0th record)<br> end file offset = 352322612 (0th record)<br> size in bytes = 4 (of one record)<br> gap from prev var = 0<br> double co2vmr(time):<br> start file offset = 352322612 (0th record)<br> end file offset = 352322620 (0th record)<br> size in bytes = 8 (of one record)<br> gap from prev var = 0<br> double ch4vmr(time):<br> start file offset = 352322620 (0th record)<br> end file offset = 352322628 (0th record)<br> size in bytes = 8 (of one record)<br> gap from prev var = 0<br> double n2ovmr(time):<br> start file offset = 352322628 (0th record)<br> end file offset = 352322636 (0th record)<br> size in bytes = 8 (of one record)<br> gap from prev var = 0<br> double f11vmr(time):<br> start file offset = 352322636 (0th record)<br> end file offset = 352322644 (0th record)<br> size in bytes = 8 (of one record)<br> gap from prev var = 0<br> double f12vmr(time):<br> start file offset = 352322644 (0th record)<br> end file offset = 352322652 (0th record)<br> size in bytes = 8 (of one record)<br> gap from prev var = 0<br> double sol_tsi(time):<br> start file offset = 352322652 (0th record)<br> end file offset = 352322660 (0th record)<br> size in bytes = 8 (of one record)<br> gap from prev var = 0<br> int nsteph(time):<br> start file offset = 352322660 (0th record)<br> end file offset = 352322664 (0th record)<br> size in bytes = 4 (of one record)<br> gap from prev var = 0<br> double ADRAIN(time, lev, ncol):<br> start file offset = 352322664 (0th record)<br> end file offset = 5217716232 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double ADSNOW(time, lev, ncol):<br> start file offset = 5217716232 (0th record)<br> end file offset = 10083109800 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double ANRAIN(time, lev, ncol):<br> start file offset = 10083109800 (0th record)<br> end file offset = 14948503368 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double ANSNOW(time, lev, ncol):<br> start file offset = 14948503368 (0th record)<br> end file offset = 19813896936 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double AQRAIN(time, lev, ncol):<br> start file offset = 19813896936 (0th record)<br> end file offset = 24679290504 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double AQSNOW(time, lev, ncol):<br> start file offset = 24679290504 (0th record)<br> end file offset = 29544684072 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double AREI(time, lev, ncol):<br> start file offset = 29544684072 (0th record)<br> end file offset = 34410077640 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double AREL(time, lev, ncol):<br> start file offset = 34410077640 (0th record)<br> end file offset = 39275471208 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double AWNC(time, lev, ncol):<br> start file offset = 39275471208 (0th record)<br> end file offset = 44140864776 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double AWNI(time, lev, ncol):<br> start file offset = 44140864776 (0th record)<br> end file offset = 49006258344 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double CCN3(time, lev, ncol):<br> start file offset = 49006258344 (0th record)<br> end file offset = 53871651912 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double CLDICE(time, lev, ncol):<br> start file offset = 53871651912 (0th record)<br> end file offset = 58737045480 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double CLDLIQ(time, lev, ncol):<br> start file offset = 58737045480 (0th record)<br> end file offset = 63602439048 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double CLOUD(time, lev, ncol):<br> start file offset = 63602439048 (0th record)<br> end file offset = 68467832616 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double CLOUDCOVER_CLUBB(time, lev, ncol):<br> start file offset = 68467832616 (0th record)<br> end file offset = 73333226184 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double CLOUDFRAC_CLUBB(time, lev, ncol):<br> start file offset = 73333226184 (0th record)<br> end file offset = 78198619752 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double CONCLD(time, lev, ncol):<br> start file offset = 78198619752 (0th record)<br> end file offset = 83064013320 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double DCQ(time, lev, ncol):<br> start file offset = 83064013320 (0th record)<br> end file offset = 87929406888 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double DMS(time, lev, ncol):<br> start file offset = 87929406888 (0th record)<br> end file offset = 92794800456 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double DTCOND(time, lev, ncol):<br> start file offset = 92794800456 (0th record)<br> end file offset = 97660194024 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double DTV(time, lev, ncol):<br> start file offset = 97660194024 (0th record)<br> end file offset =102525587592 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double DTWR_DMS(time, lev, ncol):<br> start file offset =102525587592 (0th record)<br> end file offset =107390981160 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double DTWR_H2O2(time, lev, ncol):<br> start file offset =107390981160 (0th record)<br> end file offset =112256374728 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double DTWR_H2SO4(time, lev, ncol):<br> start file offset =112256374728 (0th record)<br> end file offset =117121768296 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double DTWR_SO2(time, lev, ncol):<br> start file offset =117121768296 (0th record)<br> end file offset =121987161864 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double DTWR_SOAG(time, lev, ncol):<br> start file offset =121987161864 (0th record)<br> end file offset =126852555432 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double FICE(time, lev, ncol):<br> start file offset =126852555432 (0th record)<br> end file offset =131717949000 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double FREQI(time, lev, ncol):<br> start file offset =131717949000 (0th record)<br> end file offset =136583342568 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double FREQL(time, lev, ncol):<br> start file offset =136583342568 (0th record)<br> end file offset =141448736136 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double FREQR(time, lev, ncol):<br> start file offset =141448736136 (0th record)<br> end file offset =146314129704 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double FREQS(time, lev, ncol):<br> start file offset =146314129704 (0th record)<br> end file offset =151179523272 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double H2O(time, lev, ncol):<br> start file offset =151179523272 (0th record)<br> end file offset =156044916840 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double H2O2(time, lev, ncol):<br> start file offset =156044916840 (0th record)<br> end file offset =160910310408 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double H2SO4(time, lev, ncol):<br> start file offset =160910310408 (0th record)<br> end file offset =165775703976 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double ICIMR(time, lev, ncol):<br> start file offset =165775703976 (0th record)<br> end file offset =170641097544 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double ICWMR(time, lev, ncol):<br> start file offset =170641097544 (0th record)<br> end file offset =175506491112 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double IWC(time, lev, ncol):<br> start file offset =175506491112 (0th record)<br> end file offset =180371884680 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double NUMICE(time, lev, ncol):<br> start file offset =180371884680 (0th record)<br> end file offset =185237278248 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double NUMLIQ(time, lev, ncol):<br> start file offset =185237278248 (0th record)<br> end file offset =190102671816 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double NUMRAI(time, lev, ncol):<br> start file offset =190102671816 (0th record)<br> end file offset =194968065384 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double NUMSNO(time, lev, ncol):<br> start file offset =194968065384 (0th record)<br> end file offset =199833458952 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double OMEGA(time, lev, ncol):<br> start file offset =199833458952 (0th record)<br> end file offset =204698852520 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double OMEGAT(time, lev, ncol):<br> start file offset =204698852520 (0th record)<br> end file offset =209564246088 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double PDEL(time, lev, ncol):<br> start file offset =209564246088 (0th record)<br> end file offset =214429639656 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double PMID(time, lev, ncol):<br> start file offset =214429639656 (0th record)<br> end file offset =219295033224 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double Q(time, lev, ncol):<br> start file offset =219295033224 (0th record)<br> end file offset =224160426792 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double QRL(time, lev, ncol):<br> start file offset =224160426792 (0th record)<br> end file offset =229025820360 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double QRS(time, lev, ncol):<br> start file offset =229025820360 (0th record)<br> end file offset =233891213928 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double RAINQM(time, lev, ncol):<br> start file offset =233891213928 (0th record)<br> end file offset =238756607496 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double RCMINLAYER_CLUBB(time, lev, ncol):<br> start file offset =238756607496 (0th record)<br> end file offset =243622001064 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double RCMTEND_CLUBB(time, lev, ncol):<br> start file offset =243622001064 (0th record)<br> end file offset =248487394632 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double RCM_CLUBB(time, lev, ncol):<br> start file offset =248487394632 (0th record)<br> end file offset =253352788200 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double RELHUM(time, lev, ncol):<br> start file offset =253352788200 (0th record)<br> end file offset =258218181768 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double RELVAR(time, lev, ncol):<br> start file offset =258218181768 (0th record)<br> end file offset =263083575336 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double RHO_CLUBB(time, lev, ncol):<br> start file offset =263083575336 (0th record)<br> end file offset =267948968904 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double RIMTEND_CLUBB(time, lev, ncol):<br> start file offset =267948968904 (0th record)<br> end file offset =272814362472 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double RTM_CLUBB(time, lev, ncol):<br> start file offset =272814362472 (0th record)<br> end file offset =277679756040 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double RTP2_ZT_CLUBB(time, lev, ncol):<br> start file offset =277679756040 (0th record)<br> end file offset =282545149608 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double RVMTEND_CLUBB(time, lev, ncol):<br> start file offset =282545149608 (0th record)<br> end file offset =287410543176 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double SNOWQM(time, lev, ncol):<br> start file offset =287410543176 (0th record)<br> end file offset =292275936744 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double SO2(time, lev, ncol):<br> start file offset =292275936744 (0th record)<br> end file offset =297141330312 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double SOAE(time, lev, ncol):<br> start file offset =297141330312 (0th record)<br> end file offset =302006723880 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double SOAG(time, lev, ncol):<br> start file offset =302006723880 (0th record)<br> end file offset =306872117448 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br> double STEND_CLUBB(time, lev, ncol):<br> start file offset =306872117448 (0th record)<br> end file offset =311737511016 (0th record)<br> size in bytes = 4865393568 (of one record)<br> gap from prev var = 0<br>}<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Aug 4, 2023 at 10:35 AM Wei-Keng Liao <<a href="mailto:wkliao@northwestern.edu" target="_blank">wkliao@northwestern.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>Can you run command "ncoffsets -sg <a href="http://file.nc" target="_blank">file.nc</a>" that shows the sizes of file header</div>
<div>and all variables? For the cesm case, is _enddef called only once?</div>
<div><br>
</div>
<div>Could you also point me to the program files that call PnetCDF APIs, including</div>
<div>writing attributes and variables?</div>
<div><br>
</div>
<div><br>
</div>
<div>
<div>Wei-keng </div>
<div><br>
<blockquote type="cite">
<div>On Aug 4, 2023, at 11:05 AM, Jim Edwards <<a href="mailto:jedwards@ucar.edu" target="_blank">jedwards@ucar.edu</a>> wrote:</div>
<br>
<div>
<div dir="ltr">
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
I am using the new ncar system, <a href="https://urldefense.com/v3/__https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&cad=rja&uact=8&ved=2ahUKEwigtcytrsOAAxWXHjQIHVoDD6YQFnoECBcQAQ&url=https*3A*2F*2Farc.ucar.edu*2Fknowledge_base*2F74317833&usg=AOvVaw2aXlWuOfLnua7fFmIgvfoV&opi=89978449__;JSUlJSU!!Dq0X2DkFhyF93HkjWTBQKhk!Xq6u5krREolkIRHG8AL2taDCmg6HsEdgcEoviUVyzqUINi-ipPM1EhtMcJkQfUYghDhutn7DfH5Wjm57wJ9lQhc$" target="_blank">
derecho</a>, which has a lustre parallel file system.</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
<br>
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
Looking at the difference between the two headers below makes me wonder if the issue is with variable attributes?<br>
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
<br>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
snip</div>
<div><br>
<blockquote type="cite">
<div><br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, Aug 4, 2023 at 9:39 AM Wei-Keng Liao <<a href="mailto:wkliao@northwestern.edu" target="_blank">wkliao@northwestern.edu</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>Hi, Jim
<div><br>
</div>
<div>Can your provide the test program and the file header dumped by "ncdump -h", if that is available?</div>
<div>Also, what machine was used in the tests and its the parallel file system configuration is?</div>
<div>These can help diagnose.</div>
<div><br>
</div>
<div>
<div>Wei-keng </div>
<div><br>
<blockquote type="cite">
<div>On Aug 4, 2023, at 8:49 AM, Jim Edwards <<a href="mailto:jedwards@ucar.edu" target="_blank">jedwards@ucar.edu</a>> wrote:</div>
<br>
<div>
<div dir="ltr">
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
I am using ncmpi_iput_varn and ncmpi_wait_all to write output from my model. I have a test program that does nothing but test the
<br>
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
performance of the write operation. Attached is a plot of performance in the model and in the standalone application. I'm looking for
<br>
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
clues as to why the model performance is scaling so badly with the number of variables but the standalone program performance is fine.
<br>
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
<br>
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
<br clear="all">
</div>
<br>
<span class="gmail_signature_prefix">-- </span><br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div>
<div>Jim Edwards<br>
<br>
</div>
<font size="1">CESM Software Engineer<br>
</font></div>
<font size="1">National Center for Atmospheric Research<br>
</font></div>
<font size="1">Boulder, CO</font> <br>
</div>
</div>
</div>
<span id="m_-8733781311756881360m_4857238735649148917m_2599353008599558804m_3752102915513421155m_3416875553179895269m_1419731810550872517m_-4661512969452605678cid:f_lkwn4kse0"><Screenshot 2023-07-27 at 11.49.03 AM.png></span></div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<br>
<span class="gmail_signature_prefix">-- </span><br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div>
<div>Jim Edwards<br>
<br>
</div>
<font size="1">CESM Software Engineer<br>
</font></div>
<font size="1">National Center for Atmospheric Research<br>
</font></div>
<font size="1">Boulder, CO</font> <br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote></div><br clear="all"><br><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div><div>Jim Edwards<br><br></div><font size="1">CESM Software Engineer<br></font></div><font size="1">National Center for Atmospheric Research<br></font></div><font size="1">Boulder, CO</font> <br></div></div>