<html><head><meta http-equiv="content-type" content="text/html; charset=us-ascii"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div><br></div> Run STREAMS with <div><br></div><div>MPI_BINDING="<i style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">-map-by socket --bind-to core --report-bindings" make mpistreams</i></div><div><font face="Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif" size="3"><i><br></i></font></div><div><font face="Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif" size="3"><i>send the result</i></font></div><div><font face="Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif" size="3"><i><br></i></font><div> Also run </div><div><br></div><div><div>lscpu</div><div>numactl -H</div></div><div><br></div><div>if they are available on your machine, send the result</div><div><br></div><div><br><blockquote type="cite"><div>On Oct 10, 2023, at 10:17 AM, Gong Yujie <yc17470@connect.um.edu.mo> wrote:</div><br class="Apple-interchange-newline"><div><meta charset="UTF-8"><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">Dear Barry,</div><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;"><br></div><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">I tried to use the binding as suggested by PETSc: </div><div class="elementToProof ContentPasted0" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;"><i>mpiexec -n 4 --map-by socket --bind-to socket --report-bindings</i><br></div><div class="elementToProof ContentPasted0" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">But it seems not improving the performance. Here is the make stream log</div><div class="elementToProof ContentPasted0" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;"><br></div><div class="elementToProof ContentPasted0" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">Best Regards,<div>Yujie</div></div><div class="elementToProof ContentPasted0" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;"><br></div><div class="elementToProof ContentPasted0 ContentPasted1" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">mpicc -o MPIVersion.o -c -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -fvisibility=hidden -g -O -I/home/tt/petsc-3.16.0/include -I/home/tt/petsc-3.16.0/arch-linux-c-opt/include `pwd`/MPIVersion.c<div class="ContentPasted1">Running streams with 'mpiexec --oversubscribe ' using 'NPMAX=16'</div><div class="ContentPasted1">1 26119.1937 Rate (MB/s)</div><div class="ContentPasted1">2 29833.4281 Rate (MB/s) 1.1422</div><div class="ContentPasted1">3 65338.5050 Rate (MB/s) 2.50155</div><div class="ContentPasted1">4 59832.7482 Rate (MB/s) 2.29076</div><div class="ContentPasted1">5 48629.8396 Rate (MB/s) 1.86184</div><div class="ContentPasted1">6 58569.4289 Rate (MB/s) 2.24239</div><div class="ContentPasted1">7 63827.1144 Rate (MB/s) 2.44369</div><div class="ContentPasted1">8 57448.5349 Rate (MB/s) 2.19948</div><div class="ContentPasted1">9 61405.3273 Rate (MB/s) 2.35097</div><div class="ContentPasted1">10 68021.6111 Rate (MB/s) 2.60428</div><div class="ContentPasted1">11 71289.0422 Rate (MB/s) 2.72937</div><div class="ContentPasted1">12 76900.6386 Rate (MB/s) 2.94422</div><div class="ContentPasted1">13 80198.6807 Rate (MB/s) 3.07049</div><div class="ContentPasted1">14 64846.3685 Rate (MB/s) 2.48271</div><div class="ContentPasted1">15 83072.8631 Rate (MB/s) 3.18053</div><div class="ContentPasted1">16 70128.0166 Rate (MB/s) 2.68492</div><div class="ContentPasted1">------------------------------------------------</div><div class="ContentPasted1">Traceback (most recent call last):</div><div class="ContentPasted1"> File "process.py", line 89, in <module></div><div class="ContentPasted1"> process(sys.argv[1],len(sys.argv)-2)</div><div class="ContentPasted1"> File "process.py", line 33, in process</div><div class="ContentPasted1"> speedups[i] = triads[i]/triads[0]</div><div class="ContentPasted1">TypeError: 'dict_values' object does not support indexing</div><div class="ContentPasted1">make[2]: [makefile:47: mpistream] Error 1 (ignored)</div><div class="ContentPasted1">Traceback (most recent call last):</div><div class="ContentPasted1"> File "process.py", line 89, in <module></div><div class="ContentPasted1"> process(sys.argv[1],len(sys.argv)-2)</div><div class="ContentPasted1"> File "process.py", line 33, in process</div><div class="ContentPasted1"> speedups[i] = triads[i]/triads[0]</div><div class="ContentPasted1">TypeError: 'dict_values' object does not support indexing</div>make[2]: [makefile:79: mpistreams] Error 1 (ignored)<br></div><div id="appendonsend" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"></div><hr tabindex="-1" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; display: inline-block; width: 922.171875px;"><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;"></span><div id="divRplyFwdMsg" dir="ltr" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><font face="Calibri, sans-serif" style="font-size: 11pt;"><b>From:</b><span class="Apple-converted-space"> </span>Barry Smith <bsmith@petsc.dev><br><b>Sent:</b><span class="Apple-converted-space"> </span>Tuesday, October 10, 2023 9:59 PM<br><b>To:</b><span class="Apple-converted-space"> </span>Gong Yujie <yc17470@connect.um.edu.mo><br><b>Cc:</b><span class="Apple-converted-space"> </span>petsc-users@mcs.anl.gov <petsc-users@mcs.anl.gov><br><b>Subject:</b><span class="Apple-converted-space"> </span>Re: [petsc-users] Scalability problem using PETSc with local installed OpenMPI</font><div> </div></div><div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; line-break: after-white-space;"><div><br></div> Take a look at <a href="https://petsc.org/release/faq/#what-kind-of-parallel-computers-or-clusters-are-needed-to-use-petsc-or-why-do-i-get-little-speedup">https://petsc.org/release/faq/#what-kind-of-parallel-computers-or-clusters-are-needed-to-use-petsc-or-why-do-i-get-little-speedup</a><div><br></div><div> Check the binding that OpenMPI is using (by the way, there are much more recent OpenMPI versions, I suggest using them). Run the STREAMS benchmark as indicated on that page.</div><div><br></div><div> Barry</div><div><br><div><br><blockquote type="cite"><div>On Oct 10, 2023, at 9:27 AM, Gong Yujie <yc17470@connect.um.edu.mo> wrote:</div><br class="x_Apple-interchange-newline"><div><div class="x_elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">Dear PETSc developers,</div><div class="x_elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;"><br></div><div class="x_elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">I installed OpenMPI3 first and then installed PETSc with that mpi. Currently, I'm facing a scalability issue, in detail, I tested that using OpenMPI to calculate an addition of two distributed arrays and I get a good scalability. The problem is when I calculate the addition of two vectors in PETSc, I don't have any scalability. For the same size of the problem, PETSc costs a lot much time than merely using OpenMPI. </div><div class="x_elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;"><br></div><div class="x_elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">My PETSc version is 3.16.0 and the version of OpenMPI is 3.1.4. Hope you can give me some suggestions.</div><div class="x_elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;"><br></div><div class="x_elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">Best Regards,<div>Yujie</div></div></div></blockquote></div></div></div></div></blockquote></div><br></div></body></html>