[mpich-discuss] MPICH deadlock error

[Gustavo Correa]

On Feb 13, 2012, at 5:03 PM, Sarika K wrote:

> Thanks! Rajeev for the quick feedback. I really appreciate it.  I have used but never never written/modified MPI code. I am assuming that I need to use the nonblocking routine MPI_Isend within the if (master) part of the sample code. Is that right?
> 
> Best regards,
> Sarika
> 

Hi Sarika

I think you may also need an MPI_Wait or MPI_Waitall after the MPI_Recv.

However, your code seems to broadcast  the same 'buf' from Master to all Workers, right?
Have you tried to use MPI_Bcast, instead of MPI_Send & MPI_Recv?
A collective call, and is likely to perform better.
Something like this:

if (Master) then
 ...load buf with data
endif

call MPI_Bcast(buf, ...)  ! every process calls it

if (Worker) then
 ... unload data from buf
endif

I hope this helps,
Gus Correa

> 
> On Mon, Feb 13, 2012 at 1:45 PM, Rajeev Thakur <thakur at mcs.anl.gov> wrote:
> This will happen if the master is also sending to itself, and calls MPI_Send(to itself) before MPI_Recv(from itself). You need to either use a nonblocking send or post a nonblocking receive before the blocking send.
> 
> Rajeev
> 
> 
> On Feb 13, 2012, at 3:28 PM, Sarika K wrote:
> 
> > Dear MPICH-discuss group:
> >
> > My work involves working with Fortran Code using MPICH for parallelization. But I have a very limited experience with the details of MPICH implementation. (I have always treated the MPICh part of the code as a black box).
> >
> > I am now working on porting the code across different machine configurations. My modeling code works fine on some machines/servers. But it also generates random MPI deadlock errors when running the simulations across other machines/servers.
> >
> > The error message is below.
> > "Fatal error in MPI_Send: Other MPI error, error stack:
> > MPI_Send(174): MPI_Send(buf=0x7f4d9b375010, count=1, dtype=USER<vector>, dest=1, tag=10001, MPI_COMM_WORLD) failed
> > MPID_Send(53): DEADLOCK: attempting to send a message to the local process without a prior matching receive"
> >
> > I searched this list/other resources for this error code and strongly believe that there is a bug in the model MPI implementation code which remains dormant in some environments and works fine due to the internal buffering threshold dependance.
> >
> > I am not sure if this is sufficient information, but attached below sample subroutine (there are many inside the code) which generates the deadlock error.
> >
> > I would really appreciate any help/pointers from the group to fix this error in our code.
> >
> > Thanks in advance for your time and assistance,
> > Sarika
> >
> > c-----------------------------------------------------------------------------------------------------------------------------
> >       subroutine int_distrib1(iend)
> > c-----------------------
> > c  Master distributes another bunch of integers to Workers
> > c-----------------------------------------------------------------------------------------------------------------------------
> > c
> >       use ParallelDataMap
> >       use CommDataTypes
> >       implicit none
> >       include 'mpif.h'
> > c
> >       include 'aqmax.param'
> >       include 'aqindx.cmm'
> > c
> >       integer :: iend
> >       integer, parameter ::  Nbuf=35
> >       integer ::  i, j, k, buf(Nbuf), Ierr, status(MPI_STATUS_SIZE)
> > c
> >       if (Master) then
> > ! arguments
> >     buf(1) = iend
> > !  /aqspid/ in aqindx.cmm stuff
> >     buf(2) = iair
> >     buf(3) = ih2o
> >     buf(4) = io2
> >     buf(5) = ico
> >     buf(6) = ino2
> >     buf(7) = iho2
> >     buf(8) = iso2
> >     buf(9) = io3
> >     buf(10)= ich4
> >     buf(11)= ico2
> >     buf(12)= ih2
> >     buf(13)= in2
> >     buf(14)= itrace
> >     k=15
> >     buf(k:k+9) = ispg_idx(1:10); k=k+10
> >     buf(k:k+9) = ispl_idx(1:10); k=k+10
> >
> >     do i=1,Nworkers
> >       call MPI_SEND(buf, Nbuf, MPI_INTEGER,
> >      &         i, i,  MPI_COMM_WORLD, Ierr)
> >
> >     enddo
> >     print*, ''
> >     print*, 'done sending int_distrib1'
> >     print*, ''
> >       endif   !   (Master)
> > c
> > c
> >       if (Worker) then
> >         call MPI_RECV(buf, Nbuf, MPI_INTEGER, 0, MyId,
> >      &                 MPI_COMM_WORLD, status, ierr)
> >     iend  = buf(1)
> > ! /aqspid/ in aqindx.cmm stuff
> >     iair  = buf(2)
> >     ih2o  = buf(3)
> >     io2   = buf(4)
> >     ico   = buf(5)
> >     ino2  = buf(6)
> >     iho2  = buf(7)
> >     iso2  = buf(8)
> >     io3   = buf(9)
> >     ich4  = buf(10)
> >     ico2  = buf(11)
> >     ih2   = buf(12)
> >     in2   = buf(13)
> >     itrace= buf(14)
> >     k=15
> >     ispg_idx(1:10) = buf(k:k+9); k=k+10
> >     ispl_idx(1:10) = buf(k:k+9); k=k+10
> >     print*, ''
> >     print*, 'done receiving int_distrib1'
> >     print*, ''
> >       endif  !    (Worker)
> > c
> >       end  subroutine int_distrib1
> >
> >
> >
> > _______________________________________________
> > mpich-discuss mailing list     mpich-discuss at mcs.anl.gov
> > To manage subscription options or unsubscribe:
> > https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss
> 
> _______________________________________________
> mpich-discuss mailing list     mpich-discuss at mcs.anl.gov
> To manage subscription options or unsubscribe:
> https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss
> 
> _______________________________________________
> mpich-discuss mailing list     mpich-discuss at mcs.anl.gov
> To manage subscription options or unsubscribe:
> https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss