[MPICH] MPI I/O Reading 2gb or more of data

[Rajeev Thakur]

In the MPI spec, "count" is defined as an int. So, in theory, you should be
able to read up to 2G datatypes of whatever size when int is 32 bits. In
practice, within the ROMIO code, there may be cases where the size in bytes
of a single read is declared as an int, so you may be limited to 2 Gbytes.
Therefore, you may want to break up your reads into less than 2 Gbytes each.
There should be no performance difference because you are reading very very
large amounts at once already. (I wish all users read even one-hundredth of
that in a single call :-)).

Rajeev
 

> -----Original Message-----
> From: owner-mpich-discuss at mcs.anl.gov 
> [mailto:owner-mpich-discuss at mcs.anl.gov] On Behalf Of James S Perrin
> Sent: Friday, July 27, 2007 12:35 PM
> To: mpich
> Subject: [MPICH] MPI I/O Reading 2gb or more of data
> 
> Hi,
> 	I've come across a problem and need to confirm the 
> intended behaviour 
> of MPICH2 and the MPI spec when it come to reading (or handling in 
> general) more 2GB or more of data.
> 
> 	Specifically I reading contiguous data using 
> MPI_File_read, if I need 
> to read MAX_INT elements or more I chunk the reads over MAX_INT.
> 
> If nelements*sizeof(element) < 2GB then every thing is fine, ie I can 
> read 2^31-1 of chars. However is size of datatype is > 2 and 
> therefore 
> I'm reading more than 2GB of actual data then no values seem 
> to be read 
> but the return value from MPI_File_read is MPI_SUCCESS.
> 
> 	I've changed my code to make sure the actual amount of 
> data read is 
> less than 2gb but what should be the correct behaviour?
> 
> 	Using 1.05p4.
> 
> Regards
> James
> -- 
> --------------------------------------------------------------
> ----------
> James S. Perrin,                  | email: 
> james.perrin at manchester.ac.uk
> Manchester Visualization Centre,  | web:   www.mc.manchester.ac.uk
> Kilburn Building, The University, | tel:   +44 161 275 6945
> Manchester, England. M13 9PL.     | fax:   +44 161 275 0637
> --------------------------------------------------------------
> ----------
> "The test of intellect is the refusal to belabour the obvious"
> - Alfred Bester
> --------------------------------------------------------------
> ----------
> 
>