pnetcdf and large transfers

[Wei-keng Liao]

> int MPI_File_write_all(MPI_File fh, const void *buf, int count,
>                       MPI_Datatype datatype, MPI_Status *status)
> 
> So you probably see the problem: 'int count' -- integer are still 32
> bits on many systems (linux x86_64, blue gene, ppc64): how do we
> describe more than 2 GiB of data?

I thought this is a ROMIO problem which restricts the amount of
bytes in a single read/write call. In romio/adio/common/ad_write.c, these lines
below calls an assertion.

    MPI_Type_size(datatype, &datatype_size);
    len = (ADIO_Offset)datatype_size * (ADIO_Offset)count;
    ADIOI_Assert(len == (unsigned int) len); /* read takes an unsigned int parm */

The limit is 4GiB in ROMIO, instead of 2GiB in PnetCDF. If this assertion is
removed/resolved, your datatype approach will make sense for > 4GB I/O.
In POSIX read/write, the count argument is of size_t type, which is 8
bytes on 64-bit machines. ROMIO should check the size of size_t at configure
time to avoid the above assertion.

Note that the nonblocking APIs might end up with noncontiguous buffer
type when calling MPI-IO APIs. Breaking it apart for chunking I/O needs
some work, maybe requiring more temp memory space.

Wei-keng