[mpich-discuss] MPI I/O using large integers in file type

[Rajeev Thakur]

Alternately, you can call the I/O functions separately for 2GB portions at a time. Won't make much of a difference in performance at that size.

Rajeev

On Sep 7, 2011, at 6:23 PM, Rajeev Thakur wrote:

> Nick,
>        With a 32-bit OS, you will have trouble using indexed types for files >4G, unless the indexing follows a regular pattern that is repeated. You may want to run on a machine with a 64-bit OS. You can use hindexed; just multiply the index by the size of the elementary type.
> 
> Rajeev
> 
> On Sep 7, 2011, at 5:38 PM, Nick Stokes wrote:
> 
>> Dear Rajeev, 
>> 
>> On Wed, Sep 7, 2011 at 3:27 PM, Nick Stokes <randomaccessiterator at gmail.com> wrote:
>> 
>> 
>> On Wed, Sep 7, 2011 at 3:22 PM, Rajeev Thakur <thakur at mcs.anl.gov> wrote:
>> MPI_Type_create_hindexed does.
>> 
>> 
>> 
>> Moreover, on my 32-bit system  MPI_Aint is typedef'ed to int (at least in my installation of MPICH2 - 1.3.3 on Windows 32-bit).  So hindexed type seems not to solve the large file problem in general.  If it were possible to create an indexed type with MPI_Offset displacements, that would have helped. 
>> 
>> Maybe I am on the wrong track here. Does my question make sense?  Perhaps a more explicit and perhaps more relate-able example would be as follows: 
>> 
>> Imagine a simulation where there are a total of 10^9  degrees of freedom, and double precision is used.  The computation result is to be output to a single file, which would add up to 8 GBs.  MPI is utilized to implement parallelization via domain partitioning, and each partition holds a certain chunk of data.   These chunks correspond to non-overlapping regions with gaps in the final output.   How would you do a collective READ/WRITE in such a case without integer overflow?
>> 
>> thanks many
>> 
>> _______________________________________________
>> mpich-discuss mailing list     mpich-discuss at mcs.anl.gov
>> To manage subscription options or unsubscribe:
>> https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss
> 
> _______________________________________________
> mpich-discuss mailing list     mpich-discuss at mcs.anl.gov
> To manage subscription options or unsubscribe:
> https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss