[petsc-users] A problem with MPI Derived Data Type and 'calloc'

[Matthew Knepley]

On Thu, Apr 12, 2012 at 4:37 PM, Zhenglun (Alan) Wei <zhenglun.wei at gmail.com
> wrote:

>  Dear All,
>      I hope you're having a nice day.
>      I came across a problem while using MPI derived data type. Although
> it does not related to PETSc, I hope you guys can give me some suggestions
> or comments.
>      The reason I want to do this is that, in my PETSc code, I want to
> send a set of data. It is fairly easy to do that if data is contiguous;
> while it would be comparatively hard if it is not. I searched online and
> found the MPI derived data type may help me. What I learned are
> MPI_TYPE_CONTIGUOUS, MPI_TYPE_VECTOR  and MPI_TYPE_STRUCT. I did a test
> program, which basically combined several example programs in
> "http://computing.llnl.gov/tutorials/mpi/#Derived_Data_Types"<http://computing.llnl.gov/tutorials/mpi/#Derived_Data_Types>with all these three MPI derived data types. It works fine. (Please see my
> attachment for AlanFloatStruct.c). Briefly saying, it generates a 4 by 4
> matrix 'a[i][j]', where 'a' is a structure with members of 'u' and 'v':
>
> a[i][j].u is
>    01   02   03   04      // 2 digits just for E-mail format. Actually,
> you may find that data type for member '.u' is 'double'
>    05   06   07   08
>    09   10   11   12
>    13   14   15   16
> a[i][j].v = 10*a[i][j].u  // I will neglect .v later, because I want to
> focus on .u member
>
>      As I defined, 'j' in 'a[i][j]' is the inner loop. For example,
> a[0][0] = 1,  a[0][1] = 2, a[0][2] = 3 ... So, they are contiguous. Then,
> 'i' is the outer loop, a[0][0] = 1, a[1][0] = 5 ..., which are not
> contiguous.
>      My purpose is let rank = 0 to send the 4 contiguous data of a[n][m]
> (m = 0~3) to n'th rank and store them to 'b[n][p]' in the n'th rank. Then,
> I want to send the 4 non-contiguous data of a[m][n] (m = 0~3) to n'th rank
> and store them to 'c[p][n]' in the n'th rank. For example, if 'b' and 'c'
> are 4 by 4 matrix and initialized by '-5', after communication with rank =
> 0, I expect in rank = 1
>
> b[i][j].u
>    -5   -5   -5   -5
>    05   06   07   08
>    -5   -5   -5   -5
>    -5   -5   -5   -5
> c[i][j].v
>    -5   02   -5   -5
>    -5   06   -5   -5
>    -5   10   -5   -5
>    -5   14   -5   -5
>
>     Everything is fine in AlanFloatStruct.c. *However, when I was trying
> to use 'calloc' to allocate memory to 'a[][]' rather than directly define
> 'a[][]' as a structure matrix, the problem comes.* (see AlanTEST1.c). 'b'
> matrix is ok, yet, it keeps giving me problems for 'c' that
>

If you declare c[][] on the stack, most compilers will lay it out
contiguously. However, if you separately allocate each row, they
need not be contiguous. If you really want to do this, allocate one big
block, and setup the pointers to each row yourself.

  Thanks,

     Matt


> c[i][j].u
>    -5   02   -5   -5
>    -5   00   -5   -5
>    -5   00   -5   -5
>    -5   00   -5   -5
>
>     I guess it would be some problem for byte counting or padding of MPI
> data in 'calloc'. I refer to 'Notes' part in '
> www.mcs.anl.gov/research/projects/mpi/www/www3/MPI_Type_struct/html' and
> made AlanTEST2.c. The problem is still there.
>     I wonder what should I for this? Or, does PETS have any alternative
> way to fix this problem. BTW, if you want to run my code, you need change
> the name of .c file to AlanTEST.c. My script file 'AlanRun' is written
> based on AlanTEST.c.
>
> thank you so much in advance,
> Alan
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120412/8591366c/attachment-0001.htm>