[mpich-discuss] MPI_Bcast error

[Rajeev Thakur]

> 1) I thought (I've read it somewhere I can't remember on the web) that all collective MPI operations, including the MPI_BCAST function, would work as a barrier (but we could not check this in the case above). Is this right?

Only those collectives in which every process needs data from every other process act as barriers, .e.g., allreduce, allgather, all-to-all. Bcast, scatter, gather don't act as barriers.

> 2) Is there a way to trace MPI in some fashion that could help us on debugging this problem? Perhaps a MPI "debug" argument. What would you recommend? (We are running on Linux, on Amazon EC2)

For one, you could check if MPI_Bcast is being called correctly on all processes. For this, you can compile with the  option -mpe=mpicheck to mpicc or mpif90.

Rajeev


On Feb 29, 2012, at 5:39 PM, Luiz Carlos da Costa Junior wrote:

> Hi Pavan,
> 
> Thanks a lot for your quick answer.
> 
> Unfortunately, one of our main challenges right now is to identify a piece of code where program always fails.
> We tried to isolate the code and create some test programs, but we couldn't find one that reproduces it so far.
> 
> However, we tried a different approach by tracing every MPI call from our code and we discovered a curious behavior I would like to share: we have a MPI_BCAST call in which one of all the 32 processes didn't return from it. The program then got frozen and we could check that all other processes were stuck waiting it ahead in the code.
> 
> We did some batch runs and we could reproduce this same behavior some of the times.
> 
> There are two questions I would like to ask:
> 1) I thought (I've read it somewhere I can't remember on the web) that all collective MPI operations, including the MPI_BCAST function, would work as a barrier (but we could not check this in the case above). Is this right?
> 2) Is there a way to trace MPI in some fashion that could help us on debugging this problem? Perhaps a MPI "debug" argument. What would you recommend? (We are running on Linux, on Amazon EC2)
> 
> Regards,
> Luiz
> 
> 
> On 28 February 2012 20:08, Pavan Balaji <balaji at mcs.anl.gov> wrote:
> 
> Can you send us a test program where it fails?  Below are the answers to your questions.
> 
> 
> On 02/28/2012 05:03 PM, Luiz Carlos Costa Junior wrote:
> 1) Is there any limitation on the size of the buffer that is sent?
> 
> No.  You might run into a limit with the "int" parameter for the datatype count if you send more than 2 billion elements, but you can easily workaround that by creating a datatype with larger datatypes (double or larger contiguous types).  But with 12MB data, that's irrelevant.
> 
> 
> 2) If this limit exists, would it be related to the number of the
> process of the communicator? in this case, I am using 32 processes, but
> I commonly had success with bigger clusters (over 200 processes).
> 
> Number of processes don't matter.
> 
> 
> 3) Is the content of data being sent relevant? If I have some
> uninitialized data, would it be a concern? In other words, I understand
> that the only thing that matters is that the buffer size must be correct
> in all process (any combination of datatype/array size) and there must
> be enough allocated space to receive the data, right?
> 
> No.  MPI doesn't care what you are sending.
> 
> 
> 4) How is the best way to send this data? Split it in smaller broadcasts
> might be better/safer?
> 
> One broadcast is the best.  If you see better performance by doing multiple smaller broadcasts, that'll be classified as a bug in our code.
> 
> 
> 5) How should I classify a 12 MB message? Small? Big? I believe it
> should be pretty small because I also have other typical executions
> instances with messages over 100 MB that had sucess.
> 
> It's small enough that it should work fine.
> 
> -- 
> Pavan Balaji
> http://www.mcs.anl.gov/~balaji
> 
> _______________________________________________
> mpich-discuss mailing list     mpich-discuss at mcs.anl.gov
> To manage subscription options or unsubscribe:
> https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss