performance issue

Jim Edwards jedwards at ucar.edu
Mon Aug 7 14:12:42 CDT 2023


Hi Wei-Keng,

I think that I've found the problem.   In the model I am writing a number
of scalar variables to the file as well as the decomposed variables.
for the scalar variables I use a code structure like:

ncmpi_begin_indep_data(fh);
ncmpi_put_vars_int(fh, varid, start, count, stride, data);
ncmpi_end_indep_data(fh);

In my pioperf test code I didn't write any scalars - this morning I added
one and the write performance for the decomposed variables got very very
bad.  What can I do about it?

Jim

On Fri, Aug 4, 2023 at 1:58 PM Wei-Keng Liao <wkliao at northwestern.edu>
wrote:

> I believe the inconsistent values of cb_nodes is the culprit.
> I also noticed that striping_unit is different, 16 MB vs 1 MB.
> My guess is the mechanism of setting cb_nodes in ROMIO needs a fix
> when Lustre PFL is used. At the moment, I suggest not to use it.
>
> Wei-keng
>
> On Aug 4, 2023, at 2:43 PM, Jim Edwards <jedwards at ucar.edu> wrote:
>
> Here is the info for the standalone pioperf program:
>
> dec1281.hsn.de.hpc.ucar.edu
> <https://urldefense.com/v3/__http://dec1281.hsn.de.hpc.ucar.edu__;!!Dq0X2DkFhyF93HkjWTBQKhk!V4W0Yzn6BgsuwZ345_sC5f1F_sJy9ohr09OG6TCLvrYfI-ISwBySN82UBXJrDHTzWeT8o3jBGxifC8qJlP5TXiA$>
> 0: PE 0: MPICH MPIIO environment settings ===============================
> dec1281.hsn.de.hpc.ucar.edu
> <https://urldefense.com/v3/__http://dec1281.hsn.de.hpc.ucar.edu__;!!Dq0X2DkFhyF93HkjWTBQKhk!V4W0Yzn6BgsuwZ345_sC5f1F_sJy9ohr09OG6TCLvrYfI-ISwBySN82UBXJrDHTzWeT8o3jBGxifC8qJlP5TXiA$>
> 0: PE 0:   MPICH_MPIIO_HINTS_DISPLAY                      = 1
> dec1281.hsn.de.hpc.ucar.edu
> <https://urldefense.com/v3/__http://dec1281.hsn.de.hpc.ucar.edu__;!!Dq0X2DkFhyF93HkjWTBQKhk!V4W0Yzn6BgsuwZ345_sC5f1F_sJy9ohr09OG6TCLvrYfI-ISwBySN82UBXJrDHTzWeT8o3jBGxifC8qJlP5TXiA$>
> 0: PE 0:   MPICH_MPIIO_HINTS                              = NULL
> dec1281.hsn.de.hpc.ucar.edu
> <https://urldefense.com/v3/__http://dec1281.hsn.de.hpc.ucar.edu__;!!Dq0X2DkFhyF93HkjWTBQKhk!V4W0Yzn6BgsuwZ345_sC5f1F_sJy9ohr09OG6TCLvrYfI-ISwBySN82UBXJrDHTzWeT8o3jBGxifC8qJlP5TXiA$>
> 0: PE 0:   MPICH_MPIIO_ABORT_ON_RW_ERROR                  = disable
> dec1281.hsn.de.hpc.ucar.edu
> <https://urldefense.com/v3/__http://dec1281.hsn.de.hpc.ucar.edu__;!!Dq0X2DkFhyF93HkjWTBQKhk!V4W0Yzn6BgsuwZ345_sC5f1F_sJy9ohr09OG6TCLvrYfI-ISwBySN82UBXJrDHTzWeT8o3jBGxifC8qJlP5TXiA$>
> 0: PE 0:   MPICH_MPIIO_CB_ALIGN                           = 2
> dec1281.hsn.de.hpc.ucar.edu
> <https://urldefense.com/v3/__http://dec1281.hsn.de.hpc.ucar.edu__;!!Dq0X2DkFhyF93HkjWTBQKhk!V4W0Yzn6BgsuwZ345_sC5f1F_sJy9ohr09OG6TCLvrYfI-ISwBySN82UBXJrDHTzWeT8o3jBGxifC8qJlP5TXiA$>
> 0: PE 0:   MPICH_MPIIO_DVS_MAXNODES                       = -1
> dec1281.hsn.de.hpc.ucar.edu
> <https://urldefense.com/v3/__http://dec1281.hsn.de.hpc.ucar.edu__;!!Dq0X2DkFhyF93HkjWTBQKhk!V4W0Yzn6BgsuwZ345_sC5f1F_sJy9ohr09OG6TCLvrYfI-ISwBySN82UBXJrDHTzWeT8o3jBGxifC8qJlP5TXiA$>
> 0: PE 0:   MPICH_MPIIO_AGGREGATOR_PLACEMENT_DISPLAY       = 0
> dec1281.hsn.de.hpc.ucar.edu
> <https://urldefense.com/v3/__http://dec1281.hsn.de.hpc.ucar.edu__;!!Dq0X2DkFhyF93HkjWTBQKhk!V4W0Yzn6BgsuwZ345_sC5f1F_sJy9ohr09OG6TCLvrYfI-ISwBySN82UBXJrDHTzWeT8o3jBGxifC8qJlP5TXiA$>
> 0: PE 0:   MPICH_MPIIO_AGGREGATOR_PLACEMENT_STRIDE        = -1
> dec1281.hsn.de.hpc.ucar.edu
> <https://urldefense.com/v3/__http://dec1281.hsn.de.hpc.ucar.edu__;!!Dq0X2DkFhyF93HkjWTBQKhk!V4W0Yzn6BgsuwZ345_sC5f1F_sJy9ohr09OG6TCLvrYfI-ISwBySN82UBXJrDHTzWeT8o3jBGxifC8qJlP5TXiA$>
> 0: PE 0:   MPICH_MPIIO_MAX_NUM_IRECV                      = 50
> dec1281.hsn.de.hpc.ucar.edu
> <https://urldefense.com/v3/__http://dec1281.hsn.de.hpc.ucar.edu__;!!Dq0X2DkFhyF93HkjWTBQKhk!V4W0Yzn6BgsuwZ345_sC5f1F_sJy9ohr09OG6TCLvrYfI-ISwBySN82UBXJrDHTzWeT8o3jBGxifC8qJlP5TXiA$>
> 0: PE 0:   MPICH_MPIIO_MAX_NUM_ISEND                      = 50
> dec1281.hsn.de.hpc.ucar.edu
> <https://urldefense.com/v3/__http://dec1281.hsn.de.hpc.ucar.edu__;!!Dq0X2DkFhyF93HkjWTBQKhk!V4W0Yzn6BgsuwZ345_sC5f1F_sJy9ohr09OG6TCLvrYfI-ISwBySN82UBXJrDHTzWeT8o3jBGxifC8qJlP5TXiA$>
> 0: PE 0:   MPICH_MPIIO_MAX_SIZE_ISEND                     = 10485760
> dec1281.hsn.de.hpc.ucar.edu
> <https://urldefense.com/v3/__http://dec1281.hsn.de.hpc.ucar.edu__;!!Dq0X2DkFhyF93HkjWTBQKhk!V4W0Yzn6BgsuwZ345_sC5f1F_sJy9ohr09OG6TCLvrYfI-ISwBySN82UBXJrDHTzWeT8o3jBGxifC8qJlP5TXiA$>
> 0: PE 0:   MPICH_MPIIO_OFI_STARTUP_CONNECT                = disable
> dec1281.hsn.de.hpc.ucar.edu
> <https://urldefense.com/v3/__http://dec1281.hsn.de.hpc.ucar.edu__;!!Dq0X2DkFhyF93HkjWTBQKhk!V4W0Yzn6BgsuwZ345_sC5f1F_sJy9ohr09OG6TCLvrYfI-ISwBySN82UBXJrDHTzWeT8o3jBGxifC8qJlP5TXiA$>
> 0: PE 0:   MPICH_MPIIO_OFI_STARTUP_NODES_AGGREGATOR        = 2
> dec1281.hsn.de.hpc.ucar.edu
> <https://urldefense.com/v3/__http://dec1281.hsn.de.hpc.ucar.edu__;!!Dq0X2DkFhyF93HkjWTBQKhk!V4W0Yzn6BgsuwZ345_sC5f1F_sJy9ohr09OG6TCLvrYfI-ISwBySN82UBXJrDHTzWeT8o3jBGxifC8qJlP5TXiA$>
> 0: PE 0: MPICH MPIIO statistics environment settings ====================
> dec1281.hsn.de.hpc.ucar.edu
> <https://urldefense.com/v3/__http://dec1281.hsn.de.hpc.ucar.edu__;!!Dq0X2DkFhyF93HkjWTBQKhk!V4W0Yzn6BgsuwZ345_sC5f1F_sJy9ohr09OG6TCLvrYfI-ISwBySN82UBXJrDHTzWeT8o3jBGxifC8qJlP5TXiA$>
> 0: PE 0:   MPICH_MPIIO_STATS                              = 0
> dec1281.hsn.de.hpc.ucar.edu
> <https://urldefense.com/v3/__http://dec1281.hsn.de.hpc.ucar.edu__;!!Dq0X2DkFhyF93HkjWTBQKhk!V4W0Yzn6BgsuwZ345_sC5f1F_sJy9ohr09OG6TCLvrYfI-ISwBySN82UBXJrDHTzWeT8o3jBGxifC8qJlP5TXiA$>
> 0: PE 0:   MPICH_MPIIO_TIMERS                             = 0
> dec1281.hsn.de.hpc.ucar.edu
> <https://urldefense.com/v3/__http://dec1281.hsn.de.hpc.ucar.edu__;!!Dq0X2DkFhyF93HkjWTBQKhk!V4W0Yzn6BgsuwZ345_sC5f1F_sJy9ohr09OG6TCLvrYfI-ISwBySN82UBXJrDHTzWeT8o3jBGxifC8qJlP5TXiA$>
> 0: PE 0:   MPICH_MPIIO_WRITE_EXIT_BARRIER                 = 1
> dec1281.hsn.de.hpc.ucar.edu
> <https://urldefense.com/v3/__http://dec1281.hsn.de.hpc.ucar.edu__;!!Dq0X2DkFhyF93HkjWTBQKhk!V4W0Yzn6BgsuwZ345_sC5f1F_sJy9ohr09OG6TCLvrYfI-ISwBySN82UBXJrDHTzWeT8o3jBGxifC8qJlP5TXiA$>
> 0:  (t_initf) Read in prof_inparm namelist from: pioperf.nl
> <https://urldefense.com/v3/__http://pioperf.nl__;!!Dq0X2DkFhyF93HkjWTBQKhk!V4W0Yzn6BgsuwZ345_sC5f1F_sJy9ohr09OG6TCLvrYfI-ISwBySN82UBXJrDHTzWeT8o3jBGxifC8qJ07uO4BY$>
> dec1281.hsn.de.hpc.ucar.edu
> <https://urldefense.com/v3/__http://dec1281.hsn.de.hpc.ucar.edu__;!!Dq0X2DkFhyF93HkjWTBQKhk!V4W0Yzn6BgsuwZ345_sC5f1F_sJy9ohr09OG6TCLvrYfI-ISwBySN82UBXJrDHTzWeT8o3jBGxifC8qJlP5TXiA$>
> 0:   Testing decomp:
> /glade/derecho/scratch/jedwards/piodecomps/32768/dof001.dat
> dec1281.hsn.de.hpc.ucar.edu
> <https://urldefense.com/v3/__http://dec1281.hsn.de.hpc.ucar.edu__;!!Dq0X2DkFhyF93HkjWTBQKhk!V4W0Yzn6BgsuwZ345_sC5f1F_sJy9ohr09OG6TCLvrYfI-ISwBySN82UBXJrDHTzWeT8o3jBGxifC8qJlP5TXiA$>
> 0:  iotype=           1  of            1
> dec1281.hsn.de.hpc.ucar.edu
> <https://urldefense.com/v3/__http://dec1281.hsn.de.hpc.ucar.edu__;!!Dq0X2DkFhyF93HkjWTBQKhk!V4W0Yzn6BgsuwZ345_sC5f1F_sJy9ohr09OG6TCLvrYfI-ISwBySN82UBXJrDHTzWeT8o3jBGxifC8qJlP5TXiA$>
> 0: PE 0: MPIIO hints for pioperf.2-0256-1.nc
> <https://urldefense.com/v3/__http://pioperf.2-0256-1.nc__;!!Dq0X2DkFhyF93HkjWTBQKhk!V4W0Yzn6BgsuwZ345_sC5f1F_sJy9ohr09OG6TCLvrYfI-ISwBySN82UBXJrDHTzWeT8o3jBGxifC8qJ2eymcFw$>
> :
> dec1281.hsn.de.hpc.ucar.edu
> <https://urldefense.com/v3/__http://dec1281.hsn.de.hpc.ucar.edu__;!!Dq0X2DkFhyF93HkjWTBQKhk!V4W0Yzn6BgsuwZ345_sC5f1F_sJy9ohr09OG6TCLvrYfI-ISwBySN82UBXJrDHTzWeT8o3jBGxifC8qJlP5TXiA$>
> 0:           romio_cb_pfr             = disable
> dec1281.hsn.de.hpc.ucar.edu
> <https://urldefense.com/v3/__http://dec1281.hsn.de.hpc.ucar.edu__;!!Dq0X2DkFhyF93HkjWTBQKhk!V4W0Yzn6BgsuwZ345_sC5f1F_sJy9ohr09OG6TCLvrYfI-ISwBySN82UBXJrDHTzWeT8o3jBGxifC8qJlP5TXiA$>
> 0:           romio_cb_fr_types        = aar
> dec1281.hsn.de.hpc.ucar.edu
> <https://urldefense.com/v3/__http://dec1281.hsn.de.hpc.ucar.edu__;!!Dq0X2DkFhyF93HkjWTBQKhk!V4W0Yzn6BgsuwZ345_sC5f1F_sJy9ohr09OG6TCLvrYfI-ISwBySN82UBXJrDHTzWeT8o3jBGxifC8qJlP5TXiA$>
> 0:           cb_align                 = 2
> dec1281.hsn.de.hpc.ucar.edu
> <https://urldefense.com/v3/__http://dec1281.hsn.de.hpc.ucar.edu__;!!Dq0X2DkFhyF93HkjWTBQKhk!V4W0Yzn6BgsuwZ345_sC5f1F_sJy9ohr09OG6TCLvrYfI-ISwBySN82UBXJrDHTzWeT8o3jBGxifC8qJlP5TXiA$>
> 0:           cb_buffer_size           = 16777216
> dec1281.hsn.de.hpc.ucar.edu
> <https://urldefense.com/v3/__http://dec1281.hsn.de.hpc.ucar.edu__;!!Dq0X2DkFhyF93HkjWTBQKhk!V4W0Yzn6BgsuwZ345_sC5f1F_sJy9ohr09OG6TCLvrYfI-ISwBySN82UBXJrDHTzWeT8o3jBGxifC8qJlP5TXiA$>
> 0:           romio_cb_fr_alignment    = 1
> dec1281.hsn.de.hpc.ucar.edu
> <https://urldefense.com/v3/__http://dec1281.hsn.de.hpc.ucar.edu__;!!Dq0X2DkFhyF93HkjWTBQKhk!V4W0Yzn6BgsuwZ345_sC5f1F_sJy9ohr09OG6TCLvrYfI-ISwBySN82UBXJrDHTzWeT8o3jBGxifC8qJlP5TXiA$>
> 0:           romio_cb_ds_threshold    = 0
> dec1281.hsn.de.hpc.ucar.edu
> <https://urldefense.com/v3/__http://dec1281.hsn.de.hpc.ucar.edu__;!!Dq0X2DkFhyF93HkjWTBQKhk!V4W0Yzn6BgsuwZ345_sC5f1F_sJy9ohr09OG6TCLvrYfI-ISwBySN82UBXJrDHTzWeT8o3jBGxifC8qJlP5TXiA$>
> 0:           romio_cb_alltoall        = automatic
> dec1281.hsn.de.hpc.ucar.edu
> <https://urldefense.com/v3/__http://dec1281.hsn.de.hpc.ucar.edu__;!!Dq0X2DkFhyF93HkjWTBQKhk!V4W0Yzn6BgsuwZ345_sC5f1F_sJy9ohr09OG6TCLvrYfI-ISwBySN82UBXJrDHTzWeT8o3jBGxifC8qJlP5TXiA$>
> 0:           romio_cb_read            = automatic
> dec1281.hsn.de.hpc.ucar.edu
> <https://urldefense.com/v3/__http://dec1281.hsn.de.hpc.ucar.edu__;!!Dq0X2DkFhyF93HkjWTBQKhk!V4W0Yzn6BgsuwZ345_sC5f1F_sJy9ohr09OG6TCLvrYfI-ISwBySN82UBXJrDHTzWeT8o3jBGxifC8qJlP5TXiA$>
> 0:           romio_cb_write           = automatic
> dec1281.hsn.de.hpc.ucar.edu
> <https://urldefense.com/v3/__http://dec1281.hsn.de.hpc.ucar.edu__;!!Dq0X2DkFhyF93HkjWTBQKhk!V4W0Yzn6BgsuwZ345_sC5f1F_sJy9ohr09OG6TCLvrYfI-ISwBySN82UBXJrDHTzWeT8o3jBGxifC8qJlP5TXiA$>
> 0:           romio_no_indep_rw        = false
> dec1281.hsn.de.hpc.ucar.edu
> <https://urldefense.com/v3/__http://dec1281.hsn.de.hpc.ucar.edu__;!!Dq0X2DkFhyF93HkjWTBQKhk!V4W0Yzn6BgsuwZ345_sC5f1F_sJy9ohr09OG6TCLvrYfI-ISwBySN82UBXJrDHTzWeT8o3jBGxifC8qJlP5TXiA$>
> 0:           romio_ds_write           = automatic
> dec1281.hsn.de.hpc.ucar.edu
> <https://urldefense.com/v3/__http://dec1281.hsn.de.hpc.ucar.edu__;!!Dq0X2DkFhyF93HkjWTBQKhk!V4W0Yzn6BgsuwZ345_sC5f1F_sJy9ohr09OG6TCLvrYfI-ISwBySN82UBXJrDHTzWeT8o3jBGxifC8qJlP5TXiA$>
> 0:           ind_wr_buffer_size       = 524288
> dec1281.hsn.de.hpc.ucar.edu
> <https://urldefense.com/v3/__http://dec1281.hsn.de.hpc.ucar.edu__;!!Dq0X2DkFhyF93HkjWTBQKhk!V4W0Yzn6BgsuwZ345_sC5f1F_sJy9ohr09OG6TCLvrYfI-ISwBySN82UBXJrDHTzWeT8o3jBGxifC8qJlP5TXiA$>
> 0:           romio_ds_read            = disable
> dec1281.hsn.de.hpc.ucar.edu
> <https://urldefense.com/v3/__http://dec1281.hsn.de.hpc.ucar.edu__;!!Dq0X2DkFhyF93HkjWTBQKhk!V4W0Yzn6BgsuwZ345_sC5f1F_sJy9ohr09OG6TCLvrYfI-ISwBySN82UBXJrDHTzWeT8o3jBGxifC8qJlP5TXiA$>
> 0:           ind_rd_buffer_size       = 4194304
> dec1281.hsn.de.hpc.ucar.edu
> <https://urldefense.com/v3/__http://dec1281.hsn.de.hpc.ucar.edu__;!!Dq0X2DkFhyF93HkjWTBQKhk!V4W0Yzn6BgsuwZ345_sC5f1F_sJy9ohr09OG6TCLvrYfI-ISwBySN82UBXJrDHTzWeT8o3jBGxifC8qJlP5TXiA$>
> 0:           direct_io                = false
> dec1281.hsn.de.hpc.ucar.edu
> <https://urldefense.com/v3/__http://dec1281.hsn.de.hpc.ucar.edu__;!!Dq0X2DkFhyF93HkjWTBQKhk!V4W0Yzn6BgsuwZ345_sC5f1F_sJy9ohr09OG6TCLvrYfI-ISwBySN82UBXJrDHTzWeT8o3jBGxifC8qJlP5TXiA$>
> 0:           striping_factor          = 24
> dec1281.hsn.de.hpc.ucar.edu
> <https://urldefense.com/v3/__http://dec1281.hsn.de.hpc.ucar.edu__;!!Dq0X2DkFhyF93HkjWTBQKhk!V4W0Yzn6BgsuwZ345_sC5f1F_sJy9ohr09OG6TCLvrYfI-ISwBySN82UBXJrDHTzWeT8o3jBGxifC8qJlP5TXiA$>
> 0:           striping_unit            = 16777216
> dec1281.hsn.de.hpc.ucar.edu
> <https://urldefense.com/v3/__http://dec1281.hsn.de.hpc.ucar.edu__;!!Dq0X2DkFhyF93HkjWTBQKhk!V4W0Yzn6BgsuwZ345_sC5f1F_sJy9ohr09OG6TCLvrYfI-ISwBySN82UBXJrDHTzWeT8o3jBGxifC8qJlP5TXiA$>
> 0:           romio_lustre_start_iodevice = -1
> dec1281.hsn.de.hpc.ucar.edu
> <https://urldefense.com/v3/__http://dec1281.hsn.de.hpc.ucar.edu__;!!Dq0X2DkFhyF93HkjWTBQKhk!V4W0Yzn6BgsuwZ345_sC5f1F_sJy9ohr09OG6TCLvrYfI-ISwBySN82UBXJrDHTzWeT8o3jBGxifC8qJlP5TXiA$>
> 0:           aggregator_placement_stride = -1
> dec1281.hsn.de.hpc.ucar.edu
> <https://urldefense.com/v3/__http://dec1281.hsn.de.hpc.ucar.edu__;!!Dq0X2DkFhyF93HkjWTBQKhk!V4W0Yzn6BgsuwZ345_sC5f1F_sJy9ohr09OG6TCLvrYfI-ISwBySN82UBXJrDHTzWeT8o3jBGxifC8qJlP5TXiA$>
> 0:           abort_on_rw_error        = disable
> dec1281.hsn.de.hpc.ucar.edu
> <https://urldefense.com/v3/__http://dec1281.hsn.de.hpc.ucar.edu__;!!Dq0X2DkFhyF93HkjWTBQKhk!V4W0Yzn6BgsuwZ345_sC5f1F_sJy9ohr09OG6TCLvrYfI-ISwBySN82UBXJrDHTzWeT8o3jBGxifC8qJlP5TXiA$>
> 0:           cb_config_list           = *:*
> dec1281.hsn.de.hpc.ucar.edu
> <https://urldefense.com/v3/__http://dec1281.hsn.de.hpc.ucar.edu__;!!Dq0X2DkFhyF93HkjWTBQKhk!V4W0Yzn6BgsuwZ345_sC5f1F_sJy9ohr09OG6TCLvrYfI-ISwBySN82UBXJrDHTzWeT8o3jBGxifC8qJlP5TXiA$>
> 0:           cb_nodes                 = 24
> dec1281.hsn.de.hpc.ucar.edu
> <https://urldefense.com/v3/__http://dec1281.hsn.de.hpc.ucar.edu__;!!Dq0X2DkFhyF93HkjWTBQKhk!V4W0Yzn6BgsuwZ345_sC5f1F_sJy9ohr09OG6TCLvrYfI-ISwBySN82UBXJrDHTzWeT8o3jBGxifC8qJlP5TXiA$>
> 0:           cray_fileoff_based_aggr  = false
> dec1281.hsn.de.hpc.ucar.edu
> <https://urldefense.com/v3/__http://dec1281.hsn.de.hpc.ucar.edu__;!!Dq0X2DkFhyF93HkjWTBQKhk!V4W0Yzn6BgsuwZ345_sC5f1F_sJy9ohr09OG6TCLvrYfI-ISwBySN82UBXJrDHTzWeT8o3jBGxifC8qJlP5TXiA$>
> 0:           romio_filesystem_type    = CRAY ADIO:
> dec1281.hsn.de.hpc.ucar.edu
> <https://urldefense.com/v3/__http://dec1281.hsn.de.hpc.ucar.edu__;!!Dq0X2DkFhyF93HkjWTBQKhk!V4W0Yzn6BgsuwZ345_sC5f1F_sJy9ohr09OG6TCLvrYfI-ISwBySN82UBXJrDHTzWeT8o3jBGxifC8qJlP5TXiA$>
> 0:  RESULT: write    SUBSET         1       256        64
>  12202.4897587597       24.8799532720
> dec1281.hsn.de.hpc.ucar.edu
> <https://urldefense.com/v3/__http://dec1281.hsn.de.hpc.ucar.edu__;!!Dq0X2DkFhyF93HkjWTBQKhk!V4W0Yzn6BgsuwZ345_sC5f1F_sJy9ohr09OG6TCLvrYfI-ISwBySN82UBXJrDHTzWeT8o3jBGxifC8qJlP5TXiA$>
> 0:   complete
> dec1281.hsn.de.hpc.ucar.edu
> <https://urldefense.com/v3/__http://dec1281.hsn.de.hpc.ucar.edu__;!!Dq0X2DkFhyF93HkjWTBQKhk!V4W0Yzn6BgsuwZ345_sC5f1F_sJy9ohr09OG6TCLvrYfI-ISwBySN82UBXJrDHTzWeT8o3jBGxifC8qJlP5TXiA$>
> 0:   calling mpi finalize
>
>
> And here it is for cesm:
> PE 0:   MPICH_MPIIO_HINTS_DISPLAY                      = 1
> PE 0:   MPICH_MPIIO_HINTS                              =
> *:cb_nodes=24:striping_factor=24
> PE 0:   MPICH_MPIIO_ABORT_ON_RW_ERROR                  = disable
> PE 0:   MPICH_MPIIO_CB_ALIGN                           = 2
> PE 0:   MPICH_MPIIO_DVS_MAXNODES                       = -1
> PE 0:   MPICH_MPIIO_AGGREGATOR_PLACEMENT_DISPLAY       = 0
> PE 0:   MPICH_MPIIO_AGGREGATOR_PLACEMENT_STRIDE        = -1
> PE 0:   MPICH_MPIIO_MAX_NUM_IRECV                      = 50
> PE 0:   MPICH_MPIIO_MAX_NUM_ISEND                      = 50
> PE 0:   MPICH_MPIIO_MAX_SIZE_ISEND                     = 10485760
> PE 0:   MPICH_MPIIO_OFI_STARTUP_CONNECT                = disable
> PE 0:   MPICH_MPIIO_OFI_STARTUP_NODES_AGGREGATOR        = 2
> PE 0:   MPICH_MPIIO_STATS                              = 0
> PE 0:   MPICH_MPIIO_TIMERS                             = 0
> PE 0:   MPICH_MPIIO_WRITE_EXIT_BARRIER                 = 1
>
> PE 0: MPIIO hints for
> SMS_D_Ln9.mpasa7p5_mpasa7p5_mg17.QPC6.derecho_intel.cam-outfrq9s.20230726_094231_iz24v6.cam.h0.0001-01-01-00000.nc
> <https://urldefense.com/v3/__http://SMS_D_Ln9.mpasa7p5_mpasa7p5_mg17.QPC6.derecho_intel.cam-outfrq9s.20230726_094231_iz24v6.cam.h0.0001-01-01-00000.nc__;!!Dq0X2DkFhyF93HkjWTBQKhk!V4W0Yzn6BgsuwZ345_sC5f1F_sJy9ohr09OG6TCLvrYfI-ISwBySN82UBXJrDHTzWeT8o3jBGxifC8qJ4dcMF8Q$>:
>
>
>           romio_cb_pfr             = disable
>
>
>
>           romio_cb_fr_types        = aar
>
>
>
>           cb_align                 = 2
>
>
>
>           cb_buffer_size           = 16777216
>
>
>
>           romio_cb_fr_alignment    = 1
>
>
>
>           romio_cb_ds_threshold    = 0
>
>
>
>           romio_cb_alltoall        = automatic
>
>
>
>           romio_cb_read            = automatic
>
>
>
>           romio_cb_write           = automatic
>
>
>
>           romio_no_indep_rw        = false
>
>
>
>           romio_ds_write           = automatic
>
>
>
>           ind_wr_buffer_size       = 524288
>
>
>
>           romio_ds_read            = disable
>
>
>
>           ind_rd_buffer_size       = 4194304
>
>
>
>           direct_io                = false
>
>
>
>           striping_factor          = 24
>
>
>
>           striping_unit            = 1048576
>
>
>
>           romio_lustre_start_iodevice = -1
>
>
>
>           aggregator_placement_stride = -1
>
>
>
>           abort_on_rw_error        = disable
>
>
>
>           cb_config_list           = *:*
>
>
>
>           cb_nodes                 = 1
>
>
>
>           cray_fileoff_based_aggr  = false
>
>
>
>           romio_filesystem_type    = CRAY ADIO:
>
>
>
> I'm trying to understand why cesm has cb_nodes=1 even though the
> striping_factor=24 and I explicitly set a hint for cb_nodes=24.
>
> On Fri, Aug 4, 2023 at 1:28 PM Wei-Keng Liao <wkliao at northwestern.edu>
> wrote:
>
>> Hi, Jim
>>
>> Could you print all the MPI-IO hints for using the Lustre progressive
>> file layout?
>> That may reveal some information.
>>
>> Wei-keng
>>
>> On Aug 4, 2023, at 1:45 PM, Wei-Keng Liao <wkliao at northwestern.edu>
>> wrote:
>>
>> Looks like the cesm file is using "Lustre progressive file layout", which
>> is
>> a new striping strategy. My guess is it is used center-widely by default.
>> Rob Latham at ANL has more experiences on this feature. He may have some
>> suggestions.
>>
>> In the meantime, can you write to a new folder and explicitly set its
>> Lustre striping count to a larger number, as the total write amount is
>> more
>> than 300GB?  This old fashion setting may give a consistent timing.
>>
>>
>> Wei-keng
>>
>> On Aug 4, 2023, at 1:27 PM, Jim Edwards <jedwards at ucar.edu> wrote:
>>
>> oh and the pnetcdf version is 1.12.3
>>
>> On Fri, Aug 4, 2023 at 12:25 PM Jim Edwards <jedwards at ucar.edu> wrote:
>>
>>> On my list of things to do in PIO is rewriting the error handling code -
>>> but that issue is the same for both cases and
>>> so I don't think it would play a role in the difference we are seeing.
>>>
>>> The lfs getstripe output of the two files is nearly identical, I show
>>> only the cesm file here.
>>>
>>>   lcm_layout_gen:    7
>>>   lcm_mirror_count:  1
>>>   lcm_entry_count:   4
>>>     lcme_id:             1
>>>     lcme_mirror_id:      0
>>>     lcme_flags:          init
>>>     lcme_extent.e_start: 0
>>>     lcme_extent.e_end:   16777216
>>>       lmm_stripe_count:  1
>>>       lmm_stripe_size:   1048576
>>>       lmm_pattern:       raid0
>>>       lmm_layout_gen:    0
>>>       lmm_stripe_offset: 27
>>>       lmm_objects:
>>>       - 0: { l_ost_idx: 27, l_fid: [0xa80000401:0x3f2ef8:0x0] }
>>>
>>>     lcme_id:             2
>>>     lcme_mirror_id:      0
>>>     lcme_flags:          init
>>>     lcme_extent.e_start: 16777216
>>>     lcme_extent.e_end:   17179869184
>>>       lmm_stripe_count:  4
>>>       lmm_stripe_size:   16777216
>>>       lmm_pattern:       raid0
>>>       lmm_layout_gen:    0
>>>       lmm_stripe_offset: 35
>>>       lmm_objects:
>>>       - 0: { l_ost_idx: 35, l_fid: [0xc80000401:0x3f32e5:0x0] }
>>>       - 1: { l_ost_idx: 39, l_fid: [0xcc0000402:0x3f3162:0x0] }
>>>       - 2: { l_ost_idx: 43, l_fid: [0xe80000402:0x3f2f3a:0x0] }
>>>       - 3: { l_ost_idx: 47, l_fid: [0xec0000401:0x3f3017:0x0] }
>>>
>>>     lcme_id:             3
>>>     lcme_mirror_id:      0
>>>     lcme_flags:          init
>>>     lcme_extent.e_start: 17179869184
>>>     lcme_extent.e_end:   68719476736
>>>       lmm_stripe_count:  12
>>>       lmm_stripe_size:   16777216
>>>       lmm_pattern:       raid0
>>>       lmm_layout_gen:    0
>>>       lmm_stripe_offset: 16
>>>       lmm_objects:
>>>       - 0: { l_ost_idx: 16, l_fid: [0x700000402:0x4021eb:0x0] }
>>>       - 1: { l_ost_idx: 20, l_fid: [0x740000402:0x4020ae:0x0] }
>>>       - 2: { l_ost_idx: 24, l_fid: [0x900000400:0x401f68:0x0] }
>>>       - 3: { l_ost_idx: 28, l_fid: [0x940000400:0x401f71:0x0] }
>>>       - 4: { l_ost_idx: 32, l_fid: [0xb00000402:0x40220c:0x0] }
>>>       - 5: { l_ost_idx: 36, l_fid: [0xb40000402:0x40210b:0x0] }
>>>       - 6: { l_ost_idx: 40, l_fid: [0xd00000402:0x402141:0x0] }
>>>       - 7: { l_ost_idx: 44, l_fid: [0xd40000400:0x401e90:0x0] }
>>>       - 8: { l_ost_idx: 48, l_fid: [0xf00000401:0x401e08:0x0] }
>>>       - 9: { l_ost_idx: 52, l_fid: [0xf40000400:0x401e32:0x0] }
>>>       - 10: { l_ost_idx: 56, l_fid: [0x1100000402:0x4022e0:0x0] }
>>>       - 11: { l_ost_idx: 60, l_fid: [0x1140000402:0x4020a6:0x0] }
>>>
>>>     lcme_id:             4
>>>     lcme_mirror_id:      0
>>>     lcme_flags:          init
>>>     lcme_extent.e_start: 68719476736
>>>     lcme_extent.e_end:   EOF
>>>       lmm_stripe_count:  24
>>>       lmm_stripe_size:   16777216
>>>       lmm_pattern:       raid0
>>>       lmm_layout_gen:    0
>>>       lmm_stripe_offset: 51
>>>       lmm_objects:
>>>       - 0: { l_ost_idx: 51, l_fid: [0x1080000400:0x402881:0x0] }
>>>       - 1: { l_ost_idx: 55, l_fid: [0x10c0000400:0x402955:0x0] }
>>>       - 2: { l_ost_idx: 59, l_fid: [0x1280000400:0x4027d6:0x0] }
>>>       - 3: { l_ost_idx: 63, l_fid: [0x12c0000401:0x402ab2:0x0] }
>>>       - 4: { l_ost_idx: 67, l_fid: [0x1480000400:0x402b75:0x0] }
>>>       - 5: { l_ost_idx: 71, l_fid: [0x14c0000400:0x4028d2:0x0] }
>>>       - 6: { l_ost_idx: 75, l_fid: [0x1680000401:0x402a3b:0x0] }
>>>       - 7: { l_ost_idx: 79, l_fid: [0x16c0000402:0x40294d:0x0] }
>>>       - 8: { l_ost_idx: 83, l_fid: [0x1880000401:0x40299c:0x0] }
>>>       - 9: { l_ost_idx: 87, l_fid: [0x18c0000402:0x402f5e:0x0] }
>>>       - 10: { l_ost_idx: 91, l_fid: [0x1a80000400:0x402a16:0x0] }
>>>       - 11: { l_ost_idx: 95, l_fid: [0x1ac0000400:0x402bd2:0x0] }
>>>       - 12: { l_ost_idx: 0, l_fid: [0x300000401:0x402a2e:0x0] }
>>>       - 13: { l_ost_idx: 4, l_fid: [0x340000402:0x4027d2:0x0] }
>>>       - 14: { l_ost_idx: 8, l_fid: [0x500000402:0x402a26:0x0] }
>>>       - 15: { l_ost_idx: 12, l_fid: [0x540000400:0x402943:0x0] }
>>>       - 16: { l_ost_idx: 64, l_fid: [0x1300000402:0x402c10:0x0] }
>>>       - 17: { l_ost_idx: 68, l_fid: [0x1340000401:0x4029c6:0x0] }
>>>       - 18: { l_ost_idx: 72, l_fid: [0x1500000402:0x402d11:0x0] }
>>>       - 19: { l_ost_idx: 76, l_fid: [0x1540000402:0x402be2:0x0] }
>>>       - 20: { l_ost_idx: 80, l_fid: [0x1700000400:0x402a64:0x0] }
>>>       - 21: { l_ost_idx: 84, l_fid: [0x1740000401:0x402b11:0x0] }
>>>       - 22: { l_ost_idx: 88, l_fid: [0x1900000402:0x402cb8:0x0] }
>>>       - 23: { l_ost_idx: 92, l_fid: [0x1940000400:0x402cd3:0x0] }
>>>
>>> On Fri, Aug 4, 2023 at 12:12 PM Wei-Keng Liao <wkliao at northwestern.edu>
>>> wrote:
>>>
>>>>
>>>> I can see the file header size of 20620 bytes. Because all attributes
>>>> are stored
>>>> in the header, the cost of writing them should not be an issue. I also
>>>> see no gap
>>>> between 2 consecutive variables, which is good, meaning the write
>>>> requests made
>>>> by MPI-IO will be contiguous.
>>>>
>>>> If the call sequence of PnetCDF APIs is the same between pioperf and
>>>> cesm, then
>>>> the performance should be similarly. Can you check the Lustre striping
>>>> settings
>>>> of the 2 output files, using command "lfs getstripe"?
>>>>
>>>> If you set any MPI-IO hints, they can also play a role in performance.
>>>> See the example in PnetCDF for how to dump all hints (function
>>>> print_info().)
>>>>
>>>> https://github.com/Parallel-NetCDF/PnetCDF/blob/master/examples/C/get_info.c
>>>> <https://urldefense.com/v3/__https://github.com/Parallel-NetCDF/PnetCDF/blob/master/examples/C/get_info.c__;!!Dq0X2DkFhyF93HkjWTBQKhk!S2kyuECdrloQfFGddM-B1Y9QnqTEKxfduivqOXqC5UpDlBkIBRYblv--jZO9EebTthfwwDYkKtJkHqMsgmikJ3c$>
>>>>
>>>> If all the above checked out right, then using Darshan should reveal
>>>> more information.
>>>>
>>>> BTW, what PnetCDF version is being used?
>>>>
>>>> A comment about PIOc_put_att_tc.
>>>> * calling MPI_Bcast for checking the error code may not be necessary.
>>>> PnetCDF does such
>>>>   check and all metadata consistency check at ncmpi_enddef. If the
>>>> number of variables
>>>>   and their attributes are high, then calling lots of MPI_Bcast can be
>>>> expensive.
>>>>
>>>> https://github.com/NCAR/ParallelIO/blob/f45ba898bec31e6cd662ac41f43e0cff14f928b2/src/clib/pio_getput_int.c#L213
>>>> <https://urldefense.com/v3/__https://github.com/NCAR/ParallelIO/blob/f45ba898bec31e6cd662ac41f43e0cff14f928b2/src/clib/pio_getput_int.c*L213__;Iw!!Dq0X2DkFhyF93HkjWTBQKhk!S2kyuECdrloQfFGddM-B1Y9QnqTEKxfduivqOXqC5UpDlBkIBRYblv--jZO9EebTthfwwDYkKtJkHqMsyB5l4Jg$>
>>>>
>>>>
>>>> Wei-keng
>>>>
>>>> On Aug 4, 2023, at 12:32 PM, Jim Edwards <jedwards at ucar.edu> wrote:
>>>>
>>>> Yes, _enddef is called only once.
>>>>
>>>> Here
>>>> <https://urldefense.com/v3/__https://github.com/NCAR/ParallelIO/blob/main/src/clib/pio_getput_int.c*L128__;Iw!!Dq0X2DkFhyF93HkjWTBQKhk!Rb9IHCtwLvKBflvuPIGfD-peS-Hl1-epxN7yjpgkPoFWdSFS3DFGNkKhfb7WqrC_N0TBJDe-1bKKU_hw4F6HzDo$> is
>>>> the code that writes attributes.  Here
>>>> <https://urldefense.com/v3/__https://github.com/NCAR/ParallelIO/blob/main/src/clib/pio_darray_int.c*L661__;Iw!!Dq0X2DkFhyF93HkjWTBQKhk!Rb9IHCtwLvKBflvuPIGfD-peS-Hl1-epxN7yjpgkPoFWdSFS3DFGNkKhfb7WqrC_N0TBJDe-1bKKU_hwOC-tyLY$> is
>>>> where variables are written.
>>>>
>>>> ncoffsets -sg pioperf.2-0256-1.nc
>>>> <https://urldefense.com/v3/__http://pioperf.2-0256-1.nc__;!!Dq0X2DkFhyF93HkjWTBQKhk!Rb9IHCtwLvKBflvuPIGfD-peS-Hl1-epxN7yjpgkPoFWdSFS3DFGNkKhfb7WqrC_N0TBJDe-1bKKU_hwTlZDyu8$>
>>>> netcdf pioperf.2-0256-1.nc
>>>> <https://urldefense.com/v3/__http://pioperf.2-0256-1.nc__;!!Dq0X2DkFhyF93HkjWTBQKhk!Rb9IHCtwLvKBflvuPIGfD-peS-Hl1-epxN7yjpgkPoFWdSFS3DFGNkKhfb7WqrC_N0TBJDe-1bKKU_hwTlZDyu8$>
>>>>  {
>>>> // file format: CDF-5
>>>>
>>>> file header:
>>>> size   = 7804 bytes
>>>> extent = 8192 bytes
>>>>
>>>> dimensions:
>>>> dim000001 = 10485762
>>>> dim000002 = 58
>>>> time = UNLIMITED // (1 currently)
>>>>
>>>> record variables:
>>>> double vard0001(time, dim000002, dim000001):
>>>>       start file offset =        8192    (0th record)
>>>>       end   file offset =  4865401760    (0th record)
>>>>       size in bytes     =  4865393568    (of one record)
>>>>       gap from prev var =         388
>>>> double vard0002(time, dim000002, dim000001):
>>>>       start file offset =  4865401760    (0th record)
>>>>       end   file offset =  9730795328    (0th record)
>>>>       size in bytes     =  4865393568    (of one record)
>>>>       gap from prev var =           0
>>>>
>>>>
>>>> snip
>>>>
>>>> double vard0064(time, dim000002, dim000001):
>>>>       start file offset =306519802976    (0th record)
>>>>       end   file offset =311385196544    (0th record)
>>>>       size in bytes     =  4865393568    (of one record)
>>>>       gap from prev var =           0
>>>> }
>>>>
>>>>  ncoffsets -sg run/
>>>> SMS_D_Ln9.mpasa7p5_mpasa7p5_mg17.QPC6.derecho_intel.cam-outfrq9s.20230726_094231_iz24v6.cam.h0.0001-01-01-03600.nc
>>>> <https://urldefense.com/v3/__http://SMS_D_Ln9.mpasa7p5_mpasa7p5_mg17.QPC6.derecho_intel.cam-outfrq9s.20230726_094231_iz24v6.cam.h0.0001-01-01-03600.nc__;!!Dq0X2DkFhyF93HkjWTBQKhk!Rb9IHCtwLvKBflvuPIGfD-peS-Hl1-epxN7yjpgkPoFWdSFS3DFGNkKhfb7WqrC_N0TBJDe-1bKKU_hwdkm0FHQ$>
>>>> netcdf run/
>>>> SMS_D_Ln9.mpasa7p5_mpasa7p5_mg17.QPC6.derecho_intel.cam-outfrq9s.20230726_094231_iz24v6.cam.h0.0001-01-01-03600.nc
>>>> <https://urldefense.com/v3/__http://SMS_D_Ln9.mpasa7p5_mpasa7p5_mg17.QPC6.derecho_intel.cam-outfrq9s.20230726_094231_iz24v6.cam.h0.0001-01-01-03600.nc__;!!Dq0X2DkFhyF93HkjWTBQKhk!Rb9IHCtwLvKBflvuPIGfD-peS-Hl1-epxN7yjpgkPoFWdSFS3DFGNkKhfb7WqrC_N0TBJDe-1bKKU_hwdkm0FHQ$>
>>>>  {
>>>> // file format: CDF-5
>>>>
>>>> file header:
>>>> size   = 20620 bytes
>>>> extent = 16777216 bytes
>>>>
>>>> dimensions:
>>>> ncol = 10485762
>>>> time = UNLIMITED // (1 currently)
>>>> nbnd = 2
>>>> chars = 8
>>>> lev = 58
>>>> ilev = 59
>>>>
>>>> fixed-size variables:
>>>> double lat(ncol):
>>>>       start file offset =    16777216
>>>>       end   file offset =   100663312
>>>>       size in bytes     =    83886096
>>>>       gap from prev var =    16756596
>>>> double lon(ncol):
>>>>       start file offset =   100663312
>>>>       end   file offset =   184549408
>>>>       size in bytes     =    83886096
>>>>       gap from prev var =           0
>>>>
>>>>
>>>> snip
>>>>
>>>> int    mdt:
>>>>       start file offset =   352322552
>>>>       end   file offset =   352322556
>>>>       size in bytes     =           4
>>>>       gap from prev var =           0
>>>>
>>>> record variables:
>>>> double time(time):
>>>>       start file offset =   352322556    (0th record)
>>>>       end   file offset =   352322564    (0th record)
>>>>       size in bytes     =           8    (of one record)
>>>>       gap from prev var =           0
>>>> int    date(time):
>>>>       start file offset =   352322564    (0th record)
>>>>       end   file offset =   352322568    (0th record)
>>>>       size in bytes     =           4    (of one record)
>>>>       gap from prev var =           0
>>>>
>>>>
>>>> snip
>>>>
>>>> double STEND_CLUBB(time, lev, ncol):
>>>>       start file offset =306872117448    (0th record)
>>>>       end   file offset =311737511016    (0th record)
>>>>       size in bytes     =  4865393568    (of one record)
>>>>       gap from prev var =           0
>>>> }
>>>>
>>>> On Fri, Aug 4, 2023 at 10:35 AM Wei-Keng Liao <wkliao at northwestern.edu>
>>>> wrote:
>>>>
>>>>> Can you run command "ncoffsets -sg file.nc
>>>>> <https://urldefense.com/v3/__http://file.nc__;!!Dq0X2DkFhyF93HkjWTBQKhk!Rb9IHCtwLvKBflvuPIGfD-peS-Hl1-epxN7yjpgkPoFWdSFS3DFGNkKhfb7WqrC_N0TBJDe-1bKKU_hwCtAtD9k$>"
>>>>> that shows the sizes of file header
>>>>> and all variables? For the cesm case, is _enddef called only once?
>>>>>
>>>>> Could you also point me to the program files that call PnetCDF APIs,
>>>>> including
>>>>> writing attributes and variables?
>>>>>
>>>>>
>>>>> Wei-keng
>>>>>
>>>>> On Aug 4, 2023, at 11:05 AM, Jim Edwards <jedwards at ucar.edu> wrote:
>>>>>
>>>>> I am using the new ncar system, derecho
>>>>> <https://urldefense.com/v3/__https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&cad=rja&uact=8&ved=2ahUKEwigtcytrsOAAxWXHjQIHVoDD6YQFnoECBcQAQ&url=https*3A*2F*2Farc.ucar.edu*2Fknowledge_base*2F74317833&usg=AOvVaw2aXlWuOfLnua7fFmIgvfoV&opi=89978449__;JSUlJSU!!Dq0X2DkFhyF93HkjWTBQKhk!Xq6u5krREolkIRHG8AL2taDCmg6HsEdgcEoviUVyzqUINi-ipPM1EhtMcJkQfUYghDhutn7DfH5Wjm57wJ9lQhc$>,
>>>>> which has a lustre parallel file system.
>>>>>
>>>>> Looking at the difference between the two headers below makes me
>>>>> wonder if the issue is with variable attributes?
>>>>>
>>>>>
>>>>> snip
>>>>>
>>>>>
>>>>> On Fri, Aug 4, 2023 at 9:39 AM Wei-Keng Liao <wkliao at northwestern.edu>
>>>>> wrote:
>>>>>
>>>>>> Hi, Jim
>>>>>>
>>>>>> Can your provide the test program and the file header dumped by
>>>>>> "ncdump -h", if that is available?
>>>>>> Also, what machine was used in the tests and its the parallel file
>>>>>> system configuration is?
>>>>>> These can help diagnose.
>>>>>>
>>>>>> Wei-keng
>>>>>>
>>>>>> On Aug 4, 2023, at 8:49 AM, Jim Edwards <jedwards at ucar.edu> wrote:
>>>>>>
>>>>>> I am using ncmpi_iput_varn and ncmpi_wait_all to write output from my
>>>>>> model.   I have a test program that does nothing but test the
>>>>>> performance of the write operation.   Attached is a plot of
>>>>>> performance in the model and in the standalone application.   I'm looking
>>>>>> for
>>>>>> clues as to why the model performance is scaling so badly with the
>>>>>> number of variables but the standalone program performance is fine.
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Jim Edwards
>>>>>>
>>>>>> CESM Software Engineer
>>>>>> National Center for Atmospheric Research
>>>>>> Boulder, CO
>>>>>> <Screenshot 2023-07-27 at 11.49.03 AM.png>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Jim Edwards
>>>>>
>>>>> CESM Software Engineer
>>>>> National Center for Atmospheric Research
>>>>> Boulder, CO
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> Jim Edwards
>>>>
>>>> CESM Software Engineer
>>>> National Center for Atmospheric Research
>>>> Boulder, CO
>>>>
>>>>
>>>>
>>>
>>> --
>>> Jim Edwards
>>>
>>> CESM Software Engineer
>>> National Center for Atmospheric Research
>>> Boulder, CO
>>>
>>
>>
>> --
>> Jim Edwards
>>
>> CESM Software Engineer
>> National Center for Atmospheric Research
>> Boulder, CO
>>
>>
>>
>>
>
> --
> Jim Edwards
>
> CESM Software Engineer
> National Center for Atmospheric Research
> Boulder, CO
>
>
>

-- 
Jim Edwards

CESM Software Engineer
National Center for Atmospheric Research
Boulder, CO
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/parallel-netcdf/attachments/20230807/58daa7b8/attachment-0001.html>


More information about the parallel-netcdf mailing list