program not scalable

[Wei-keng Liao]

Hi, Jialin

Non-contiguous I/O usually performs poorly. This is a well-known issue.

One trick you can try is to pass an MPI-IO hint below to see whether
the performance improves. Note this hints might help or might not,
depending on your array size, i.e. the values of your NLAT and NLON.

MPI_Info_set(info, "romio_ds_read", "disable");

and pass the info object to ncmpi_open(), such as
ncmpi_open(comm, fname, flags, info, fid);

Wei-keng

On Sep 22, 2012, at 10:31 AM, Liu, Jaln wrote:

> Hi,
> 
> I found that the problem only happened in requesting noncontiguous small I/O along those slow dimension.
> For example, there is a 4D dataset, Temperature[time][level][lat][lon], if we only read a 2D plane per time step, per level, or in other words, as is shown in the following codes:
> 
> mpi_count[0]=1;//read length in time dimension
> mpi_count[1]=1;//read length in level dimension
> mpi_count[2]=NLAT;//read length in lat dimension
> mpi_count[3]=NLON;//read length in lon dimension
> mpi_start[2]=0;//start position in lat dimension
> mpi_stat[3]=0;//start position in lon dimension
> 
> loop_start=(int)(TIME_STEPS/nprocs)*rank;
> loop_length=(int)(TIME_STEPS/nprocs);
> for (time=loop_start;time< loop_start+loop_length;time++)
> for(level=0;level<LEVEL_STEPS;level++){
> mpi_start[0]=time;
> mpi_start[1]=level;
> 
> //read 2D plane per time step and per level step
> ncmpi_get_vara_float_all(ncid, temp_varid, mpi_start, mpi_count, temp_in)
> }
> 
> I have no idea why the program is not scalable well. Or if I'm wrong in somewhere, any help is appreciated. Thanks,
> 
> Best Regards,
> Jialin Liu, Ph.D student.
> Computer Science Department
> Texas Tech University
> Phone: 806.742.3513(x241)
> Office:Engineer Center 304
> http://myweb.ttu.edu/jialliu/
> From: parallel-netcdf-bounces at lists.mcs.anl.gov [parallel-netcdf-bounces at lists.mcs.anl.gov] on behalf of Liu, Jaln [jaln.liu at ttu.edu]
> Sent: Saturday, September 22, 2012 4:20 AM
> To: parallel-netcdf at mcs.anl.gov
> Subject: program not scalable
> 
> Hi,
> 
> I use the PnetCDF to read a 4D dataset, the size is around 4G. I found the program is not scalable no matter how I distribute the data.
> e.g., I parallelize the program along the time dimension(which is the outer loop/the slowest dimension). The read time is the slowest when only one core is used. The data are striped on multiple storage nodes.
> 
> Can anybody help me with that? thanks in advance.
> 
> 
> Best Regards,
> Jialin Liu, Ph.D student.
> Computer Science Department
> Texas Tech University
> Phone: 806.742.3513(x241)
> Office:Engineer Center 304
> http://myweb.ttu.edu/jialliu/