[MPICH] MPI-IO, vector datatype

[Rob Ross]

Hi Russell,

The "nblocks(1)" sets that variable to 1, yes? Sorry, C++ isn't my thing.

A vector with a count of 1 is the same as a contig with a count equal to 
the blocksize of the vector. This would explain what you're seeing. The 
stride is only used if the count is greater than 1.

Regards,

Rob

Russell L. Carter wrote:
> 
>> It is easy to run on a single machine. With MPD, all you need to do is
>> % mpd &
>> % mpiexec -n 2 a.out 
> 
> Works great.  No difference between pvfs2 and unix.
> 
>> blocks of 4 ints each because you have defined INTS_PER_BLK=4.
> 
> I'm guilty of a transcription error, crap.  Sorry about that,
> that's a stupid waste of time.  Should have been INTS_PER_BLK=8.
> With INTS_PER_BLK=4, I agree with your values but the problem
> is still there.  I have found what appears to be the problem.
> The stride arg in the Create_vector method appears to be
> ignored.  It doesn't matter what I set it to, 0 on up to
> nprocs*blocksize, the block data for each proc is written
> out contiguously.
> 
> If I set the view displacement to be myrank*nints,
> the file always looks like this, without
> any holes, for any number of blocks and stride I set
> (nprocs is 2, neg is rank 0, pos is rank 1):
> 
> 0000000           0          -1          -2          -3
> 0000020          -4          -5          -6          -7
> 0000040          -8          -9         -10         -11
> 0000060         -12         -13         -14         -15
> 0000100         -16         -17         -18         -19
> 0000120         -20         -21         -22         -23
> 0000140         -24         -25         -26         -27
> 0000160         -28         -29         -30         -31
> 0000200           0           1           2           3
> 0000220           4           5           6           7
> 0000240           8           9          10          11
> 0000260          12          13          14          15
> 0000300          16          17          18          19
> 0000320          20          21          22          23
> 0000340          24          25          26          27
> 0000360          28          29          30          31
> 
> 
> If I set the view displacements to
> blocksize*sizeof(int)*myrank, the file looks like this,
> for any stride (nblocks/proc is 2 here):
> 
> 0000000           0          -1          -2          -3
> 0000020          -4          -5          -6          -7
> 0000040          -8          -9         -10         -11
> 0000060         -12         -13         -14         -15
> 0000100           0           1           2           3
> 0000120           4           5           6           7
> 0000140           8           9          10          11
> 0000160          12          13          14          15
> 0000200          16          17          18          19
> 0000220          20          21          22          23
> 0000240          24          25          26          27
> 0000260          28          29          30          31
> 
> The further reduced code is appended.  As far as I can tell
> it should produce identical datatypes and views as the program
> on p. 65 of Using MPI-2.  It was my impression that that
> program was intended to read interleaved data, maybe it's
> not?
> 
> Thanks,
> Russell
> 
> #include "mpi.h"
> #include <iostream>
> using namespace std;
> 
> struct tester
> {
>     tester()
>         : myrank(MPI::COMM_WORLD.Get_rank()),
>           nprocs(MPI::COMM_WORLD.Get_size()),
>           bufsize(FILESIZE/nprocs), nints(bufsize/sizeof(int)),
>           nblocks(1), blocksize(nints/nblocks),
>           filetype(MPI::INT),
>           //fname("pvfs2:/mnt/pvfs/tst/testfile")
>           fname("/home/rcarter/mpibin/testfile")
>     {
>         std::ios::sync_with_stdio(false);
>         filetype.Create_vector(nblocks, blocksize, nprocs * blocksize);
>         filetype.Commit();
>         obuf = new int[bufsize];
>         ibuf = new int[bufsize];
>     }
>     ~tester() {
>         delete[] obuf;
>         delete[] ibuf;
>     }
>     void write()
>     {
>         for (int i = 0; i < nints; ++i) {
>             if (myrank)
>                 obuf[i] = i;
>             else
>                 obuf[i] = -i;
>         }
> 
>         MPI::File f = open_set_view(MPI_MODE_CREATE | MPI_MODE_WRONLY);
>         f.Write_all(obuf, nints, MPI_INT, status);
>         f.Close();
>     }
>     void read()
>     {
>         MPI::File f = open_set_view(MPI_MODE_RDONLY);
>         f.Read_all(ibuf, nints, MPI_INT, status);
>         f.Close();
>         for (int i = 0; i < nints; ++i) {
>             if (obuf[i] != ibuf[i]) {
>                 cerr << "myrank, i, obuf[i], ibuf[i]: " << myrank << " "
>                      << i << " " << obuf[i] << " " << ibuf[i] << endl;
>             }
>         }
>     }
> private:
>     static const int FILESIZE = 256;
>     int myrank, nprocs, bufsize, nints, nblocks, blocksize, *obuf, *ibuf;
>     MPI::Datatype filetype;
>     string fname;
>     MPI::Status status;
> 
>     MPI::File open_set_view(int mode)
>     {
>         MPI::File f = MPI::File::Open(MPI::COMM_WORLD, fname.c_str(),
>                                       mode, MPI::INFO_NULL);
>         MPI::Offset disp = blocksize * sizeof(int) * myrank;
>         f.Set_view(disp, MPI_INT, filetype, "native", MPI_INFO_NULL);
>         return f;
>     }
> };
> int main()
> {
>     cerr << "Starting rwall.\n";
>     try {
>         MPI::Init();
>         tester t;
>         t.write();
>         MPI::COMM_WORLD.Barrier();
>         t.read();
>         MPI::Finalize();
>     } catch (exception &e) {
>         cerr << "\nCaught exception: " << e.what() << endl;
>         return -1;
>     } catch (MPI::Exception& e) {
>         cerr << "\nError:\n" << e.Get_error_string();
>         return -2;
>     }
>     cerr << "rwall end.\n";
>     return 0;
> }
>