[MPICH] question about MPI-IO Read_all

[Peter Diamessis]

Hi folks,

If I may just comment that this is a very interesting topic.
I ran into a similar situation when using MPI_WRITE_ALL &
MPI_READ_ALL to output/read-in non-contiguous 3-D data
in my CFD solver. The "global" size of the binary file was approximately
10Gb consisting of 20 3-D variables. I would encounter errors when trying
to output all 20 fields in one file. I then broke the file down into 10 files with
2 fields each, with approximate file-size equal to 1.2 Gb. Then everything
worked  smoothly. I'm wondering if this a similar issue to what Russell has
been pointing out ?

Sincerely,

Pete Diamessis




----- Original Message ----- 
From: "Rajeev Thakur" <thakur at mcs.anl.gov>
To: "'Russell L. Carter'" <rcarter at esturion.net>; <mpich-discuss at mcs.anl.gov>
Sent: Wednesday, April 25, 2007 10:19 PM
Subject: RE: [MPICH] question about MPI-IO Read_all


> 2^31 is 2 Gbytes. If you are reading 2 GB per process with a single
> Read_all, you are already doing quite well performance-wise. If you want to
> read more than that you can create a derived datatype of say 10 contiguous
> bytes and pass that as the datatype to Read_all. That would give you 20 GB.
> You read even more by using 100 or 1000 instead of 10.
> 
> In practice, you might encounter some errors, because the MPI-IO
> implementation internally may use some types that are 32-bit, not expecting
> anyone to read larger than that with a single call. So try it once, and if
> it doesn't work, read in 2GB chunks.
> 
> Rajeev
> 
> 
>> -----Original Message-----
>> From: owner-mpich-discuss at mcs.anl.gov 
>> [mailto:owner-mpich-discuss at mcs.anl.gov] On Behalf Of Russell 
>> L. Carter
>> Sent: Wednesday, April 25, 2007 6:33 PM
>> To: mpich-discuss at mcs.anl.gov
>> Subject: [MPICH] question about MPI-IO Read_all
>> 
>> Hi,
>> I have a question about the amount of data it is possible to read
>> using MPI::Create_hindex with a fundamental type of MPI::BYTE, and 
>> MPI::File::Read_all.
>> 
>> Following the discussion about irregularly distributed arras beginning
>> on p. 78 of "Using MPI-2", I want to read my data by doing this:
>> 
>> double *buf = ...;
>> int count, bufsize = ...;
>> MPI::Offset offset = ...;
>> MPI::File f = MPI::File::Open(...);
>> MPI::Datatype filetype(MPI::BYTE);
>> filetype.Create_hindexed(count, blocks, displacements);
>> f.Set_view(offset, MPI::BYTE, filetype, "native", info_);
>> f.Read_all(buf, bufsize, MPI::BYTE);
>> 
>> What I a curious about is the amount of data that can
>> be read with Read_all.  Since bufsize is an int, then
>> that would seem to imply that the maximum Read_all (per node)
>> is 2^31.  Which in bytes, is not gigantic.
>> 
>> Is there some other technique I can use to increase the amount
>> of data I can Read_all at one time?  I have different sized
>> data interspersed, so I can't offset by a larger fundamental
>> type.  My arrays are not contiguous in the fortran calling program,
>> and are of int and 4 or 8 byte reals.  If I use a Create_struct
>> to make a filetype that I use to Set_view, doesn't this have
>> the same read size limitation?  Only now it is for all the
>> arrays in the struct.  Hopefully I am missing something.
>> 
>> Thanks,
>> Russell
>> 
>> 
>