[mpich-discuss] MPI-IO issue

[Wei-keng Liao]

Hi Matt,

Could you please try MPI_Type_create_subarray() to see if
the same error happens, so we can rule out the possibility
that the error is from your derived datatype?

Wei-keng


On Feb 13, 2012, at 5:08 AM, Matthew J. Grismer wrote:

> Wei-keng,
> 
> Yes, that is exactly what I set the lowerbound and extent to for the derived
> type on each of the processors.  And then I went back and checked that the
> values were set correctly with MPI_Type_get_extent.  Then the view and reads
> are as in my first example in the original post, reading x, then y, then z
> in separate reads.
> 
> Matt
> 
> 
> On 2/10/12 6:07 PM, "Wei-keng Liao" <wkliao at ece.northwestern.edu> wrote:
> 
>> Matt,
>> 
>> Based on the way you use the new derived datatype in MPI_FILE_SET_VIEW,
>> the lowerbound and extent should be 0 and ie*je*ke*8, respectively.
>> 8 byte is the bytes for real*8 type. Can you verify those values?
>> 
>> Wei-keng
>> 
>> 
>> On Feb 10, 2012, at 4:25 PM, Grismer, Matthew J Civ USAF AFMC AFRL/RBAT wrote:
>> 
>>> Thanks for the info, I was not aware of the extent of the derived type.  I
>>> used MPI_Type_create_resized to change the lower bound and extent of the
>>> derived types on each processor to cover the entire array, not just the
>>> sub-blocks I wanted.  Now x reads correctly on each processor, but then y
>>> and z are still wrong.  After the MPI_File_read_all command for x, I used
>>> the MPI_File_get_position command to see where each processors file pointer
>>> was.  Only on processor 0 has the file pointer moved to the correct location
>>> in the file, on all the other processors it is different and far short of
>>> the correct location.  I'm defining the derived types and extents the same
>>> way on all the processors, and even check the extents after I set them to
>>> verify they are correct.  Any other thoughts on what I am missing?  Thanks.
>>> 
>>> Matt
>>> 
>>> -----Original Message-----
>>> From: mpich-discuss-bounces at mcs.anl.gov
>>> [mailto:mpich-discuss-bounces at mcs.anl.gov] On Behalf Of Wei-keng Liao
>>> Sent: Wednesday, February 08, 2012 7:33 PM
>>> To: mpich-discuss at mcs.anl.gov
>>> Subject: Re: [mpich-discuss] MPI-IO issue
>>> 
>>> Hi,
>>> 
>>> I am guessing the problem is the datatype you created.
>>> Please check the type extent to see if it equals to the
>>> size of entire array, using MPI_Type_extent(). If not,
>>> the file pointer after the first read will not move to
>>> the beginning of the second array you expected, which
>>> leads to reading data from a wrong file location.
>>> 
>>> Also, if each process reads a block subarray, I suggest
>>> to use MPI_Type_create_subarray to create the new datatype.
>>> 
>>> 
>>> Wei-keng
>>> 
>>> 
>>> On Feb 8, 2012, at 3:46 PM, Grismer, Matthew J Civ USAF AFMC AFRL/RBAT
>>> wrote:
>>> 
>>>> I am attempting to use MPI-IO to read block, structured data from a
>>>> file.  Three three-dimensional coordinate arrays are stored in the file,
>>>> one after another with Fortran ordering and in binary (not Fortran
>>>> unformatted):
>>>> 
>>>> header info
>>>> (((x(i,j,k),i=1,ie),j=1,je),k=1,ke)
>>>> (((y(i,j,k),i=1,ie),j=1,je),k=1,ke)
>>>> (((z(i,j,k),i=1,ie),j=1,je),k=1,ke)
>>>> 
>>>> Each processor reads a block subset of the arrays, and I've defined a
>>>> derived type using MPI_TYPE_INDEXED for the blocks that go to each
>>>> processor.  MPI_TYPE_INDEXED takes as arguments the number of blocks,
>>>> block size, and block displacement;  I created (and committed) the
>>>> datatype with the number of blocks/displacements 3 times (i.e.
>>>> subsetie*subsetje*subsetke*3) what I need for one variable above on a
>>>> given processor.  Then I used the following to read the file across the
>>>> processors:
>>>> 
>>>> count = subsetie*subsetje*subsetke
>>>> call MPI_FILE_SET_VIEW ( fh, skipheader, mpi_real8, newdatatype,
>>>> "native", mpi_info_null, ierror )
>>>> call MPI_FILE_READ_ALL ( fh, x, count, mpi_real8, ierror )
>>>> call MPI_FILE_READ_ALL ( fh, y, count, mpi_real8, ierror )
>>>> call MPI_FILE_READ_ALL ( fh, z, count, mpi_real8, ierror )
>>>> 
>>>> The result from this is x is read correctly, y and z are not.  However,
>>>> if I define one variable xyz(subsetie,subsetje,subsetke,3) and read all
>>>> the data in one call:
>>>> 
>>>> count = subsetie*subsetje*subsetke*3
>>>> call MPI_FILE_SET_VIEW ( fh, skipheader, mpi_real8, newdatatype,
>>>> "native", mpi_info_null, ierror )
>>>> call MPI_FILE_READ_ALL ( fh, xyz, count, mpi_real8, ierror )
>>>> 
>>>> everything is read correctly, which also verifies my derived type is
>>>> correct.  Alternatively I can reset the view (with correct
>>>> displacements) after each READ_ALL and read into the individual
>>>> variables.
>>>> 
>>>> Is this the expected behavior?  If so I do not understand how to make
>>>> successive collective reads from a file without resetting the view
>>>> before every read, which will take a toll on performance when I continue
>>>> to read many additional variables from the file.
>>>> 
>>>> Matt
>>>> _______________________________________________
>>>> mpich-discuss mailing list     mpich-discuss at mcs.anl.gov
>>>> To manage subscription options or unsubscribe:
>>>> https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss
>>> 
>>> _______________________________________________
>>> mpich-discuss mailing list     mpich-discuss at mcs.anl.gov
>>> To manage subscription options or unsubscribe:
>>> https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss
>>> _______________________________________________
>>> mpich-discuss mailing list     mpich-discuss at mcs.anl.gov
>>> To manage subscription options or unsubscribe:
>>> https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss
>> 
>> _______________________________________________
>> mpich-discuss mailing list     mpich-discuss at mcs.anl.gov
>> To manage subscription options or unsubscribe:
>> https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss
> 
> _______________________________________________
> mpich-discuss mailing list     mpich-discuss at mcs.anl.gov
> To manage subscription options or unsubscribe:
> https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss