program not scalable

[Liu, Jaln]

Hi,

I found that the problem only happened in requesting noncontiguous small I/O along those slow dimension.
For example, there is a 4D dataset, Temperature[time][level][lat][lon], if we only read a 2D plane per time step, per level, or in other words, as is shown in the following codes:

mpi_count[0]=1;//read length in time dimension
mpi_count[1]=1;//read length in level dimension
mpi_count[2]=NLAT;//read length in lat dimension
mpi_count[3]=NLON;//read length in lon dimension
mpi_start[2]=0;//start position in lat dimension
mpi_stat[3]=0;//start position in lon dimension

loop_start=(int)(TIME_STEPS/nprocs)*rank;
loop_length=(int)(TIME_STEPS/nprocs);
for (time=loop_start;time< loop_start+loop_length;time++)
for(level=0;level<LEVEL_STEPS;level++){
mpi_start[0]=time;
mpi_start[1]=level;

//read 2D plane per time step and per level step
ncmpi_get_vara_float_all(ncid, temp_varid, mpi_start, mpi_count, temp_in)
}

I have no idea why the program is not scalable well. Or if I'm wrong in somewhere, any help is appreciated. Thanks,

Best Regards,
Jialin Liu, Ph.D student.
Computer Science Department
Texas Tech University
Phone: 806.742.3513(x241)
Office:Engineer Center 304
http://myweb.ttu.edu/jialliu/
________________________________
From: parallel-netcdf-bounces at lists.mcs.anl.gov [parallel-netcdf-bounces at lists.mcs.anl.gov] on behalf of Liu, Jaln [jaln.liu at ttu.edu]
Sent: Saturday, September 22, 2012 4:20 AM
To: parallel-netcdf at mcs.anl.gov
Subject: program not scalable

Hi,

I use the PnetCDF to read a 4D dataset, the size is around 4G. I found the program is not scalable no matter how I distribute the data.
e.g., I parallelize the program along the time dimension(which is the outer loop/the slowest dimension). The read time is the slowest when only one core is used. The data are striped on multiple storage nodes.

Can anybody help me with that? thanks in advance.


Best Regards,
Jialin Liu, Ph.D student.
Computer Science Department
Texas Tech University
Phone: 806.742.3513(x241)
Office:Engineer Center 304
http://myweb.ttu.edu/jialliu/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/parallel-netcdf/attachments/20120922/6cddf9c2/attachment.html>