<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div><br class=""></div><div> It is important to check out <a href="https://petsc.org/main/faq/?highlight=faq#why-is-my-parallel-solver-slower-than-my-sequential-solver-or-i-have-poor-speed-up" class="">https://petsc.org/main/faq/?highlight=faq#why-is-my-parallel-solver-slower-than-my-sequential-solver-or-i-have-poor-speed-up</a></div><div><br class=""></div><div> In particular you will need to set appropriate binding for the mpiexec and should run the streams benchmark (make mpistreams) using the binding to find the potential performance of the system.</div><div><br class=""></div><div> If you are using a thread enabled BLAS/LAPACK that utilizes all the cores then you can get oversubscription and thus slow performance during BLAS/LAPACK calls. We try not to link with thread enabled BLAS/LAPACK by default. See <a href="https://petsc.org/main/docs/manual/blas-lapack/?highlight=thread%20blas" class="">https://petsc.org/main/docs/manual/blas-lapack/?highlight=thread%20blas</a></div><div><br class=""></div><div> Barry</div><div><br class=""></div><div><br class=""></div><div><br class=""><blockquote type="cite" class=""><div class="">On Sep 26, 2022, at 12:39 PM, Duan Junming via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" class="">petsc-users@mcs.anl.gov</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta charset="UTF-8" class=""><div id="divtagdefaultwrapper" dir="ltr" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-size: 12pt; font-family: "Times New Roman", Times, serif, Times, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;" class=""><div style="margin-top: 0px; margin-bottom: 0px;" class=""><span style="color: rgb(34, 34, 34); font-family: Arial, Helvetica, sans-serif; font-size: small;" class="">Dear all,</span></div><div style="color: rgb(34, 34, 34); font-family: Arial, Helvetica, sans-serif; font-size: small;" class=""><br class=""></div><div style="color: rgb(34, 34, 34); font-family: Arial, Helvetica, sans-serif; font-size: small;" class="">I am using PETSc 3.17.4 on a Linux server, compiling with --download-exodus --download-hdf5 --download-openmpi --download-triangle --with-fc=0 --with-debugging=0 PETSC_ARCH=arch-linux-c-opt COPTFLAGS="-g -O3" CXXOPTFLAGS="-g -O3".</div><div style="color: rgb(34, 34, 34); font-family: Arial, Helvetica, sans-serif; font-size: small;" class="">The strange thing is when I run my code with mpirun -np 1 ./main, the CPU time is 30s.</div><div style="color: rgb(34, 34, 34); font-family: Arial, Helvetica, sans-serif; font-size: small;" class="">When I use mpirun -np 2 ./main, the CPU time is 16s. It's OK.</div><div style="color: rgb(34, 34, 34); font-family: Arial, Helvetica, sans-serif; font-size: small;" class="">But when I use more than 2 CPUs, like mpirun -np 3 ./main, the CPU time is 30s.</div><div style="color: rgb(34, 34, 34); font-family: Arial, Helvetica, sans-serif; font-size: small;" class="">The output of command time is: real 0m30.189s, user 9m3.133s, sys 10m55.715s.</div><div style="color: rgb(34, 34, 34); font-family: Arial, Helvetica, sans-serif; font-size: small;" class="">I can also see that the CPU load is about 100% for each process when np = 2, but the CPU load goes to 2000%, 1000%, 1000% for each process (the server has 40 CPUs).</div><div style="color: rgb(34, 34, 34); font-family: Arial, Helvetica, sans-serif; font-size: small;" class="">Do you have any idea about this?</div><div style="color: rgb(34, 34, 34); font-family: Arial, Helvetica, sans-serif; font-size: small;" class=""><br class=""></div><div style="color: rgb(34, 34, 34); font-family: Arial, Helvetica, sans-serif; font-size: small;" class="">Thanks in advance!</div></div></div></blockquote></div><br class=""></body></html>