pnetcdf and large transfers

[Wei-keng Liao]

> Algorithm; 
> 
> - ceiling(nbytes/(1*GiB)): number of transfers
> 
> - MPI_Allreduce to find the max transfers
> 
> - carry out that many MPI_File_{write/read}_all, relying on the
>  implicit file pointer to help us keep track of where we are in the
>  file view.
> 
> I think, given the two phase overhead, we'd want N transfers of mostly
> the same size, instead of some gigabyte sized transfers and then one
> final "remainder" transfer of possibly a handful of bytes.
> When transferring multiple gigabytes, the point is probably academic
> anyway.

I am speaking from an academic point of view :)
The number of transfers may not be that straightforward.
Although in blocking APIs, the buffers are all contiguous,
fileviews may be not. The performance of a collective I/O relies
on the fact whether its aggregate access region is contiguous
or not. There can a case that the aggregate access region is contiguous
if transfer once, but individual aggregate access regions are not
if the request is broken down in multiple transfers. On the other hand,
whether I/O load is balanced among the processes is important, but
less serious. I mentioned this to you in SDAV AHM, if you remember.

I guess for blocking APIs, the worst scenario might rarely happen.
But for nonblocking APIs, the problem can occur more often.

Wei-keng