[mpich-discuss] MPI_IN_PLACE argument
Steenhauer, Kate
k.steenhauer at abdn.ac.uk
Wed Apr 28 09:08:36 CDT 2010
For a quick start I would like to disable the error message first, does that mean have to reinstall mpich2, then when I configure it ./configure I don't understand where I have to or how to disable the bug using "CPPFLAGS=-DNDEBUG"?
Further trying to resolve the problem as well with my very limited knowledge I tried what you suggested, only updating the scatter routine, which gives me the following compiler error (I have attached the messenger file):
/opt/mpich2/bin/mpif90 -c messenger.mpi8.f90 -o obj/messenger.mpi8.o In file messenger.mpi8.f90:238
if (rank == idroot)
1
Error: Unclassifiable statement in IF-clause at (1) In file messenger.mpi8.f90:242
else
1
Error: Unexpected ELSE statement at (1)
In file messenger.mpi8.f90:252
if (myid == idroot) deallocate (sendbuf)
1
Error: Expression in DEALLOCATE statement at (1) must be ALLOCATABLE or a POINTER
make[2]: *** [messenger.mpi8.o] Error 1
make[2]: Leaving directory `/home/eng923/LES/jet/usr'
make[1]: *** [options] Error 2
make[1]: Leaving directory `/home/eng923/LES/jet/usr'
make: *** [opt] Error 2
The entire subroutine is as follows (see also attached file); Do I not need to change 'call updateBorder' then as well?
subroutine scatter(A, B)
use messenger
include "mpif.inc"
real A(nx,ny,nz), B(nx_1,ny_1,nz_1)
real, allocatable :: sendbuf(:)
integer i1(nproc), i2(nproc), &
j1(nproc), j2(nproc), &
k1(nproc), k2(nproc)
! Scatter an array among the processors
! including overlapping borders
if (myid == idroot) then
do ip=1,nproc
i1(ip) = ibmino_1(icoord(1,ip))
j1(ip) = jbmino_1(icoord(2,ip))
k1(ip) = kbmino_1(icoord(3,ip))
end do
i2 = i1 + nx_1 - 1
j2 = j1 + ny_1 - 1
k2 = k1 + nz_1 - 1
allocate (sendbuf(nproc*nxyz_1))
L = 0
do ip=1,nproc
do k=k1(ip),k2(ip)
do j=j1(ip),j2(ip)
do i=i1(ip),i2(ip)
L = L + 1
sendbuf(L) = A(i,j,k)
end do
end do
end do
end do
end if
call MPI_scatter (sendbuf, nxyz_1, MPI_REAL8, &
B, nxyz_1, MPI_REAL8, &
idroot, icomm_grid, ierr)
call updateBorder(B, nx_1,ny_1,nz_1, 1, 1)
call updateBorder(B, nx_1,ny_1,nz_1, 2, 2)
call updateBorder(B, nx_1,ny_1,nz_1, 3, 3)
if (myid == idroot) deallocate (sendbuf)
return
end
Thanks
Kate
-----Original Message-----
From: mpich-discuss-bounces at mcs.anl.gov [mailto:mpich-discuss-bounces at mcs.anl.gov] On Behalf Of Dave Goodell
Sent: 28 April 2010 14:42
To: mpich-discuss at mcs.anl.gov
Subject: Re: [mpich-discuss] MPI_IN_PLACE argument
Please see the following mpich-discuss@ threads:
https://lists.mcs.anl.gov/mailman/htdig/mpich-discuss/2010-April/006974.html
https://lists.mcs.anl.gov/mailman/htdig/mpich-discuss/2010-March/006658.html
Basically, in a pinch you can disable this error check by passing
"CPPFLAGS=-DNDEBUG" to configure. But your code is still invalid MPI
code and has some chance of erroneous behavior. The same buffer will
still be passed for both arguments of memcpy, which memcpy does not
permit and may result in undefined behavior.
If you want to convert your code to use MPI_IN_PLACE, you will need to
replace either the sendbuf or the recvbuf with MPI_IN_PLACE, but
*only* at the root process.
So your MPI_Scatter becomes something like:
if (rank == idroot)
MPI_Scatter(sendbuf, nxyz_1, MPI_REAL8, MPI_IN_PLACE, nxyz_1,
MPI_REAL8, idroot, icomm_grid, ierr);
else
MPI_Scatter(sendbuf, nxyz_1, MPI_REAL8, B, nxyz_1, MPI_REAL8,
idroot, icomm_grid, ierr);
For scatter/scatterv, MPI_IN_PLACE should be passed as the recvbuf.
For gather and most other collectives, MPI_IN_PLACE should passed as
the sendbuf.
The MPI standard provides more information about MPI_IN_PLACE: http://www.mpi-forum.org/docs/mpi-2.2/mpi22-report.pdf
-Dave
On Apr 28, 2010, at 5:09 AM, Steenhauer, Kate wrote:
> Hi,
>
> I am trying to sort the bug in the mpi program (see attached). It
> seems to me, with my very limited knowledge, that everything is
> related and the whole program needs to be restructed when upgrading
> from mpich1 to mpich2? If the MPI_Scatter routine flags a bug (The
> error message is 'memcpy argument memory ranges overlap,
> dst_=0xafd74f8 src_=0xafd750c len_=16200, internal ABORT'), then it
> is, as far as I can see, most likely that the other routines, such
> as, gather(A, B), allGather(A, B), scatterXY(A, B, nk),gatherXY(A,
> B, nk),allGatherXY(A, B, nk) etc. (see attached), will bring up a
> similar bug as well. So I don't really know where to start,
> considering this is a well-established code, run for many years
> successfully with mpich1 with sensible output.
>
> By using the MPI_IN_Place argument I have to replace the recvbuf,
> B(nx_1,ny_1,nz_1)?
> call MPI_scatter (sendbuf, nxyz_1, MPI_REAL8, &
> B, nxyz_1, MPI_REAL8, &
> idroot, icomm_grid, ierr)
> I hope you will be able to help me.
> kate
>
> -----Original Message-----
> From: mpich-discuss-bounces at mcs.anl.gov [mailto:mpich-discuss-bounces at mcs.anl.gov
> ] On Behalf Of Rajeev Thakur
> Sent: 27 April 2010 17:32
> To: mpich-discuss at mcs.anl.gov; 'Anthony Chan'
> Subject: Re: [mpich-discuss] configuration problem
>
> Kate,
> You have to use the mpif.h file that comes with the MPI
> implementation, not from some other MPI implementation. With MPICH2,
> you
> have to use MPICH2's include file.
>
> The error with MPI_Scatter indicates there is a bug in your MPI
> program.
> You are using the same buffer as sendbuf and recvbuf, which is not
> allowed in MPI. You can use the MPI_IN_PLACE argument instead, as
> described in the MPI standard.
>
> I would recommend using MPICH2 instead of trying to get MPICH-1 to
> work.
>
> Rajeev
>
>
>> -----Original Message-----
>> From: mpich-discuss-bounces at mcs.anl.gov
>> [mailto:mpich-discuss-bounces at mcs.anl.gov] On Behalf Of
>> Steenhauer, Kate
>> Sent: Tuesday, April 27, 2010 9:50 AM
>> To: Anthony Chan; mpich-discuss at mcs.anl.gov
>> Subject: Re: [mpich-discuss] configuration problem
>>
>> Initially, I tried to run my code with mpich2. The following happens.
>>
>> We have 1 workstation with 8 processors, the following
>> software: Linux Centos, Fortran Intel 11.1.
>>
>> Mpich2, Intel Fortran and Linux is not working on our pc. It
>> does not like the mpif.inc file, it immediately gets into
>> problems at the initialisation of the MPI (subroutine MPI_INI).
>> mpirun -np 8 RUN02
>>> Fatal error in MPI_Comm_size: Invalid communicator, error stack:
>>> MPI_Comm_size(111): MPI_Comm_size(comm=0x5b, size=0x7fffce629784)
>>> failed
>>> MPI_Comm_size(69).: Invalid communicator MPISTART
>>> rank 7 in job 2 cops-021026_40378 caused collective
>> abort of all ranks
>>> exit status of rank 7: killed by signal 9
>>
>> Then when I change the mpif.inc file (see attachment) and
>> direct it to the mpif.h file that came with the mpich2
>> library, it gets passed this problem but then runs into a
>> next problem further down the line at MPI_SCATTER, where it
>> is trying to distribute data to the different processors. The
>> error message is 'memcpy argument memory ranges overlap,
>> dst_=0xafd74f8 src_=0xafd750c len_=16200, internal ABORT'.
>>
>> There is something in the parametresiation within the mpi
>> that is possibly different then when the code is successfully
>> run on another cluster (this cluster uses Redhat, various
>> mpich versions, e.g. mpich-1.2.5..12, and various versions of
>> Intel FORTRAN compiler (ifort), e.g. 7, 9 and 12).
>>
>> I have attached the mpif.inc file.
>>
>> Please let me know if you have any ideas?
>>
>> Thanks
>>
>> Kate
>>
>> -----Original Message-----
>> From: mpich-discuss-bounces at mcs.anl.gov
>> [mailto:mpich-discuss-bounces at mcs.anl.gov] On Behalf Of
>> chan at mcs.anl.gov
>> Sent: 27 April 2010 15:42
>> To: mpich-discuss at mcs.anl.gov
>> Subject: Re: [mpich-discuss] configuration problem
>>
>>
>> Is there any reason you can't use mpich2 ? The latest stable
>> release of mpich2 is 1.2.1p1.
>>
>> mpich-1 is no longer officially supported. Latest fortran compilers
>> are much better supported in mpich2.
>>
>> A.Chan
>>
>> ----- "Kate Steenhauer" <k.steenhauer at abdn.ac.uk> wrote:
>>
>>> Hello,
>>>
>>> We are trying to install mpich-1.2.7p1. I downloaded and
>> unzipped this
>>> version from http://www.mcs.anl.gov/research/projects/mpi/mpich1/
>>>
>>> We have 1 workstation with 8 processors, and the following software:
>>> Linux Centos and Fortran Intel 11.1.
>>>
>>> When the documentation guidelines with regard to configuring and
>>> making are followed (see the files attached) it all seems ok, e.g.
>>> mpif90 is generated. However, when a simple parallel job is
>> tested we
>>> get the following error:
>>>
>>> mpif90 -o testA MPITEST.f90
>>> No Fortran 90 compiler specified when mpif90 was created, or
>>> configuration file does not specify a compiler.
>>>
>>> Is there a specific prefix I need to give with an ifort fortran
>>> compiler when I configure mpich?
>>>
>>> I would like to thank you in advance for your help. Please
>> let me know
>>> if you need any further details.
>>>
>>> Regards
>>>
>>> Kate Steenhauer
>>>
>>> University of Aberdeen
>>>
>>> 01224-272806
>>>
>>>
>>>
>>>
>>>
>>> The University of Aberdeen is a charity registered in Scotland, No
>>> SC013683.
>>>
>>> _______________________________________________
>>> mpich-discuss mailing list
>>> mpich-discuss at mcs.anl.gov
>>> https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss
>> _______________________________________________
>> mpich-discuss mailing list
>> mpich-discuss at mcs.anl.gov
>> https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss
>>
>>
>> The University of Aberdeen is a charity registered in
>> Scotland, No SC013683.
>>
>
> _______________________________________________
> mpich-discuss mailing list
> mpich-discuss at mcs.anl.gov
> https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss
>
>
> The University of Aberdeen is a charity registered in Scotland, No
> SC013683.
> <messenger.mpi8.f90>_______________________________________________
> mpich-discuss mailing list
> mpich-discuss at mcs.anl.gov
> https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss
_______________________________________________
mpich-discuss mailing list
mpich-discuss at mcs.anl.gov
https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss
The University of Aberdeen is a charity registered in Scotland, No SC013683.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: messenger.mpi8.f90
Type: application/octet-stream
Size: 27923 bytes
Desc: messenger.mpi8.f90
URL: <http://lists.mcs.anl.gov/pipermail/mpich-discuss/attachments/20100428/27c96c95/attachment-0001.obj>
More information about the mpich-discuss
mailing list