[mpich-discuss] Scalability of Intel quad core (Harpertown) cluster

[Elvedin Trnjanin]

Hello,

I've done plenty of testing on this and I suspect the problem is not 
only your network but mainly the message sizes. When your messages sizes 
are around 64KB or greater, your are starting to maximize total 
bandwidth but if your messages are smaller, bandwidth is drastically 
less. Bandwidth will show an exponential growth until about a 1MB 
message since you hit various limits. More details on the message sizes 
and (approximate) frequency of message transfers would be useful.

Infiniband, while a very good product, it's expensive and you should get 
it if your current message passing latency is longer than your compute 
latency thus hindering overall performance. If you don't need more than 
60 MB/s bandwidth or less than 5 milliseconds latency for large 
messages, then the gigabit network would be sufficient. With 10 nodes 
and 8 cores per node (80 processes?), I'm not so sure though.

Hee Il Kim wrote:
> Hi,
>
> I recently made a Rocks cluster of 10 comuting nodes with dual Intel 
> quad core cpus (Harpertown). The nodes are connected with gigabit 
> networks.
>
> The problem is I got bad scalability from the benchmark test of Cactus 
> which is well-known program for numerical relativity. It might be 
> natural with the gigabit network. However, I suspect I missed 
> something important or it's a generic problem of clusters of multicore 
> cpus because the benchmark problems does not require massive 
> communication between the computing nodes.
>
> I tested openmpi, mpich as well as mpich2 but no significant 
> differences between openmpi and mpich2. The mpich2 was configured with 
> ssm. After reading some documents on nemesis and multicore cpus, I 
> tested with nemesis but got worse results than with ssm.
>
> - Is there any optimized configuration option for multicore cpus like 
> Harpertown?
> - Could it be improved only with infiniband, myrinet,...?
> - If the gigabit network was the cause, could it be improved with Open-MX?
>
> I'm a newbie on this field. The questions must be not clear to you. I 
> appreciate any helps in advance.
>
> Kim, Hee Il