[mpich-discuss] MPI_Bcast error

[Luiz Carlos da Costa Junior]

Hi Pavan,

Thanks a lot for your quick answer.

Unfortunately, one of our main challenges right now is to identify a piece
of code where program always fails.
We tried to isolate the code and create some test programs, but we couldn't
find one that reproduces it so far.

However, we tried a different approach by tracing every MPI call from our
code and we discovered a curious behavior I would like to share: we have a
MPI_BCAST call in which one of all the 32 processes didn't return from it.
The program then got frozen and we could check that all other processes
were stuck waiting it ahead in the code.

We did some batch runs and we could reproduce this same behavior some of
the times.

There are two questions I would like to ask:
1) I thought (I've read it somewhere I can't remember on the web) that all
collective MPI operations, including the MPI_BCAST function, would work as
a barrier (but we could not check this in the case above). Is this right?
2) Is there a way to trace MPI in some fashion that could help us on
debugging this problem? Perhaps a MPI "debug" argument. What would you
recommend? (We are running on Linux, on Amazon EC2)

Regards,
Luiz


On 28 February 2012 20:08, Pavan Balaji <balaji at mcs.anl.gov> wrote:

>
> Can you send us a test program where it fails?  Below are the answers to
> your questions.
>
>
> On 02/28/2012 05:03 PM, Luiz Carlos Costa Junior wrote:
>
>> 1) Is there any limitation on the size of the buffer that is sent?
>>
>
> No.  You might run into a limit with the "int" parameter for the datatype
> count if you send more than 2 billion elements, but you can easily
> workaround that by creating a datatype with larger datatypes (double or
> larger contiguous types).  But with 12MB data, that's irrelevant.
>
>
>  2) If this limit exists, would it be related to the number of the
>> process of the communicator? in this case, I am using 32 processes, but
>> I commonly had success with bigger clusters (over 200 processes).
>>
>
> Number of processes don't matter.
>
>
>  3) Is the content of data being sent relevant? If I have some
>> uninitialized data, would it be a concern? In other words, I understand
>> that the only thing that matters is that the buffer size must be correct
>> in all process (any combination of datatype/array size) and there must
>> be enough allocated space to receive the data, right?
>>
>
> No.  MPI doesn't care what you are sending.
>
>
>  4) How is the best way to send this data? Split it in smaller broadcasts
>> might be better/safer?
>>
>
> One broadcast is the best.  If you see better performance by doing
> multiple smaller broadcasts, that'll be classified as a bug in our code.
>
>
>  5) How should I classify a 12 MB message? Small? Big? I believe it
>> should be pretty small because I also have other typical executions
>> instances with messages over 100 MB that had sucess.
>>
>
> It's small enough that it should work fine.
>
> --
> Pavan Balaji
> http://www.mcs.anl.gov/~balaji
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/mpich-discuss/attachments/20120229/f65b253c/attachment.htm>