Hi everyone,<br><br> I'll echo the sentiments expressed by Tan and a few others that the culprit here, at least for Gaetano's code, is probably the memory bandwidth. The FDTD applications I've seen tend to be bandwidth-hungry, and the current Intel quad cores are not very good at this, especially those with the 1333 Mhz FSB such as the E5345. Some of the newer models support the 1600 Mhz FSB speeds and tend to deliver better results.... for example, using the SPEC fp_rate benchmarks (and specifically that of GemsFDTD) as a really rough approximation to parallel performance, an E5450 at 3.0 Ghz scores 29.0 vs. 36.7 for the E5472 processor. Same chip, but faster memory, resulting in 26.5% faster performance overall. [Note, I used the Supermicro results for this, but you can find similar results for any of them, I imagine.]<br>
<br> This isn't limited to FDTD codes, either... CFD-like applications are notoriously bandwidth hungry, and using SPEC scores again, WRF gives us 51.9 vs. 70.2 for the same configurations above. Again, these are the same processors, same compilers, and same tests done by guys who definitely like to eek the utmost performance from their rig, and this shows that simply adding faster memory improves things measurably... 35% in this case. Since fp_rate isn't really the same as parallel performance, though, let's switch to some first-hand measurements - I was recently running some WRF models on a system here using the E5440 (2.83 Ghz) processors, and here are the results:<br>
<br> Running on 128 cores as 16 nodes x 8 cores per node: 22.8 seconds / step<br> 128 cores as 32 nodes x 4 cores per node: 11.9 seconds / step<br> 128 cores as 64 nodes x 2 cores per node: 10.4 seconds / step<br>
<br> ... As you can see, the 'best' performance comes from using only 2 cores per node! In fact, I was able to run on 16 nodes and 4 cores per node, for a total of 64 cores, and it was only 2% slower than running on 128 cores (as 16 x 8). I didn't fiddle with task placement since MPI and the OS are <i>generally</i> pretty smart about such things, and the data points towards memory bandwidth being a key issue anyways. Hopefully having some of these 'hard numbers' can ease your burden so you don't go crazy trying to find a reason in MPICH, the OS, etc. ;)<br>
<br> Put another way, if you're concerned that your quad-core box isn't working properly via MPI, you can write up a code (or download a code) that does something that is not limited by bandwidth - generating random numbers, for example - and run it, and you <i>should</i> see a fully linear speedup (within the constraints of Amdahl's Law). So the only recommendations I'd make are:<br>
<br> 1) Use an up-to-date MPI implementation (such as MPICH2 1.0.7) and OS since they'll probably be 'smarter' about task placement than older versions<br> 2) Try using the Intel compilers if you haven't done so already since they tend to be superior to gfortran (and many times gcc as well)<br>
3) If you're buying hardware soon, look at the (more expensive) 1600 Mhz FSB boards / chips from Intel.<br><br> Hope that's useful! <br><br> Cheers,<br> - Brian<br><br><br>Brian Dobbins<br>Yale Engineering HPC<br>