[petsc-users] interpreting petsc streams result

[Junchao Zhang]

Hi, Chris,
  I did have an MR https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/merge_requests/7651__;!!G_uCfscf7eWS!fWNsAOkuZRiMn1TuiZ0HasNdskk5heIHlt3O4unVNFd3mnPlFFPISeieHQ_DFsrasG1dwtpASUuFiR6eUOugJNvoDVDy$  to
improve mpistream.  I should rework it after Barry's !6903.  See my inlined
comments to your questions

On Fri, Oct 17, 2025 at 3:37 AM Klaij, Christiaan via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Attached is a petsc streams result kindly provided by a hardware
> vendor for a single compute node, dual socket, with two AMD epyc
> 9355 processors. Each processor has 32 cores, 12 DDR5 memory
> channels and mem BW around 600 GB/s.
>
> * It is not immediately clear which line corresponds to which
> y-axis. Could future versions of petsc please color the axis
> label with the matching line color?
>
definitely

>
>
> * Why would the achieved bandwidth be roughly 0.9 x 1e6 MB/s =
> 900 GB/s and not closer to 1200 GB/s?
>
I recall it is actually not simple to get the theoretical max bandwidth.
One has to use special SIMD instructions, compiler flags and streaming
stores etc.

>
>
> * The speed-up seems to be 12 out of 64, provided multiples of 8
> cores are used. As expected given 12 memory channels?
>
Maybe not, otherwise the speedup should be 24 as you have 24 channels.


>
> * Does the zig-zag pattern indicate a pinning problem, or is it
> unavoidable given the 8 core building block of these type of
> processors?
>
I checked and found "make mpistream" uses --map-by core.  I think we should
use --map-by socket or --map-by l3cache.


>
> Chris
> dr. ir.  Christiaan  Klaij  |  senior researcher
> Research & Development  |  CFD Development
> T +31 317 49 33 44 <+31%20317%2049%2033%2044>  |  https://urldefense.us/v3/__http://www.marin.nl__;!!G_uCfscf7eWS!fWNsAOkuZRiMn1TuiZ0HasNdskk5heIHlt3O4unVNFd3mnPlFFPISeieHQ_DFsrasG1dwtpASUuFiR6eUOugJD65Q0mk$ 
> <https://urldefense.us/v3/__https://www.marin.nl/__;!!G_uCfscf7eWS!fqSBpN3Ld5fjzXGShGI09uJke12M-5LukEHe-y-gw0Bw9msZeH7wNiId6DZxQpluR_RUWpuoQWUD2HSsBfp_z4A$>
> [image: Facebook]
> <https://urldefense.us/v3/__https://www.facebook.com/marin.wageningen__;!!G_uCfscf7eWS!fqSBpN3Ld5fjzXGShGI09uJke12M-5LukEHe-y-gw0Bw9msZeH7wNiId6DZxQpluR_RUWpuoQWUD2HSsCH7BGfA$>
> [image: LinkedIn]
> <https://urldefense.us/v3/__https://www.linkedin.com/company/marin__;!!G_uCfscf7eWS!fqSBpN3Ld5fjzXGShGI09uJke12M-5LukEHe-y-gw0Bw9msZeH7wNiId6DZxQpluR_RUWpuoQWUD2HSsDAV2fAI$>
> [image: YouTube]
> <https://urldefense.us/v3/__https://www.youtube.com/marinmultimedia__;!!G_uCfscf7eWS!fqSBpN3Ld5fjzXGShGI09uJke12M-5LukEHe-y-gw0Bw9msZeH7wNiId6DZxQpluR_RUWpuoQWUD2HSsEyu_yEs$>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20251017/df0b17c5/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image918583.png
Type: image/png
Size: 5004 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20251017/df0b17c5/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image489453.png
Type: image/png
Size: 487 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20251017/df0b17c5/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image542102.png
Type: image/png
Size: 504 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20251017/df0b17c5/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image611951.png
Type: image/png
Size: 482 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20251017/df0b17c5/attachment-0003.png>