[petsc-users] Understanding the memory bandwidth

[Justin Chang]

Hi all,

According to our University's HPC cluster (Intel Xeon E5-2680v2
<http://www.cpu-world.com/CPUs/Xeon/Intel-Xeon%20E5-2680%20v2.html>), the
online specifications says I should have a maximum BW of 59.7 GB/s. I am
guessing this number is computed by 1866 MHz * 8 Bytes * 4 memory channels.

Now, when I run the STREAMS Triad benchmark on a single compute node (two
sockets, 10 cores each, 64 GB total memory), on up to 20 processes with
MPICH, i get the following:

$ mpiexec -n 1 ./MPIVersion:
Triad:        13448.6701   Rate (MB/s)

$ mpiexec -n 2 ./MPIVersion:
Triad:        24409.1406   Rate (MB/s)

$ mpiexec -n 4 ./MPIVersion
Triad:        31914.8087   Rate (MB/s)

$ mpiexec -n 6 ./MPIVersion
Triad:        33290.2676   Rate (MB/s)

$ mpiexec -n 8 ./MPIVersion
Triad:        33618.2542   Rate (MB/s)

$ mpiexec -n 10 ./MPIVersion
Triad:        33730.1662   Rate (MB/s)

$ mpiexec -n 12 ./MPIVersion
Triad:        40835.9440   Rate (MB/s)

$ mpiexec -n 14 ./MPIVersion
Triad:        44396.0042   Rate (MB/s)

$ mpiexec -n 16 ./MPIVersion
Triad:        54647.5214   Rate (MB/s) *

$ mpiexec -n 18 ./MPIVersion
Triad:        57530.8125   Rate (MB/s) *

$ mpiexec -n 20 ./MPIVersion
Triad:        42388.0739   Rate (MB/s) *

The * numbers fluctuate greatly each time I run this. However, if I use
hydra's processor binding options:

$ mpiexec.hydra -n 2 -bind-to socket ./MPIVersion
Triad:        26879.3853   Rate (MB/s)

$ mpiexec.hydra -n 4 -bind-to socket ./MPIVersion
Triad:        48363.8441   Rate (MB/s)

$ mpiexec.hydra -n 8 -bind-to socket ./MPIVersion
Triad:        63479.9284   Rate (MB/s)

$ mpiexec.hydra -n 10 -bind-to socket ./MPIVersion
Triad:        66160.5627   Rate (MB/s)

$ mpiexec.hydra -n 16 -bind-to socket ./MPIVersion
Triad:        65975.5959   Rate (MB/s)

$ mpiexec.hydra -n 20 -bind-to socket ./MPIVersion
Triad:        64738.9336   Rate (MB/s)

I get similar metrics when i use the binding options "-bind-to hwthread
-map-by socket".

Now my question is, is 13.5 GB/s on one processor "good"? Because when I
compare this to the 59.7 GB/s it seems really inefficient. Is there a way
to browse through my system files to confirm this?

Also, when I use multiple cores and with proper binding, the streams BW
exceeds the reported max BW. Is this expected?

Thanks,
Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150813/30093088/attachment.html>