[MPICH] Loopback Communication

[Elvedin Trnjanin]

I have a "ping pong" latency and bandwidth measurement program that I'm 
using for approximating performance of an Infiniband cluster. I've 
noticed that for self to self asynchronous communication with 
mpirun_ssh, to transfer a 1MB message, it takes around 3.2 ms while with 
node A to node B communication, the latency is around 2.9 ms per 1MB 
message. On the bandwidth side, self to self communication is 60MB/s 
slower. Although 60MB/s difference might not seem like a significant 
problem considering the total bandwidth for the larger message 
Infiniband transfers, on a small Gigabit Ethernet cluster I use, our 
node to node asynchronous communication bandwidth is slightly less than 
60MB/s. That difference is pretty significant to me.

My question is if anyone can explain why self to self communication is 
slower and also why the interconnect is involved instead of doing a 
memory copy? I apologize ahead of time if this issue has been discussed 
in the MPI2 standards but I'm reading the (unofficial) report and 
haven't found it yet.

Regards,
Elvedin Trnjanin