[petsc-users] Understanding the memory bandwidth

[Barry Smith]

> On Aug 13, 2015, at 10:34 AM, Justin Chang <jychang48 at gmail.com> wrote:
> 
> Hi all,
> 
> According to our University's HPC cluster (Intel Xeon E5-2680v2), the online specifications says I should have a maximum BW of 59.7 GB/s. I am guessing this number is computed by 1866 MHz * 8 Bytes * 4 memory channels. 
> 
> Now, when I run the STREAMS Triad benchmark on a single compute node (two sockets, 10 cores each, 64 GB total memory), on up to 20 processes with MPICH, i get the following:
> 
> $ mpiexec -n 1 ./MPIVersion: 
> Triad:        13448.6701   Rate (MB/s) 
> 
> $ mpiexec -n 2 ./MPIVersion: 
> Triad:        24409.1406   Rate (MB/s) 
> 
> $ mpiexec -n 4 ./MPIVersion 
> Triad:        31914.8087   Rate (MB/s)
> $ mpiexec -n 6 ./MPIVersion 
> Triad:        33290.2676   Rate (MB/s) 
> 
> 
> $ mpiexec -n 8 ./MPIVersion 
> Triad:        33618.2542   Rate (MB/s)
> 
> $ mpiexec -n 10 ./MPIVersion 
> Triad:        33730.1662   Rate (MB/s) 
> 
> 
> $ mpiexec -n 12 ./MPIVersion 
> Triad:        40835.9440   Rate (MB/s) 
> 
> 
> $ mpiexec -n 14 ./MPIVersion 
> Triad:        44396.0042   Rate (MB/s)
> 
> $ mpiexec -n 16 ./MPIVersion 
> Triad:        54647.5214   Rate (MB/s) *
> 
> $ mpiexec -n 18 ./MPIVersion 
> Triad:        57530.8125   Rate (MB/s) *
> 
> $ mpiexec -n 20 ./MPIVersion 
> Triad:        42388.0739   Rate (MB/s) *
> 
> The * numbers fluctuate greatly each time I run this.

   Yeah, MPICH's default behavior is super annoying. I think they need better defaults.

> However, if I use hydra's processor binding options:
> 
> $ mpiexec.hydra -n 2 -bind-to socket ./MPIVersion 
> Triad:        26879.3853   Rate (MB/s) 
> 
> $ mpiexec.hydra -n 4 -bind-to socket ./MPIVersion 
> Triad:        48363.8441   Rate (MB/s)
> 
> $ mpiexec.hydra -n 8 -bind-to socket ./MPIVersion 
> Triad:        63479.9284   Rate (MB/s)
> 
> $ mpiexec.hydra -n 10 -bind-to socket ./MPIVersion 
> Triad:        66160.5627   Rate (MB/s) 
> 
> $ mpiexec.hydra -n 16 -bind-to socket ./MPIVersion 
> Triad:        65975.5959   Rate (MB/s) 
> 
> $ mpiexec.hydra -n 20 -bind-to socket ./MPIVersion 
> Triad:        64738.9336   Rate (MB/s) 
> 
> I get similar metrics when i use the binding options "-bind-to hwthread -map-by socket". 
> 
> Now my question is, is 13.5 GB/s on one processor "good"?

    You mean one core. 

     Yes, that is a good number. These systems are not designed so that a single core can "saturate" (that is use) all the memory bandwidth of the node. Note that after about 8 cores you don't see any more improvement because the 8 cores has saturated the memory bandwidth. What this means is that for PETSc simulations any cores beyond 8 (or so) on the node are just unnecessary eye-candy.


> Because when I compare this to the 59.7 GB/s it seems really inefficient. Is there a way to browse through my system files to confirm this?
> 
> Also, when I use multiple cores and with proper binding, the streams BW exceeds the reported max BW. Is this expected?

   I cannot explain this, look at the exact number of loads and stores needed for the triad benchmark. Perhaps the online docs are out of date.

  Barry

> 
> Thanks,
> Justin
>