<div dir="ltr"><div class="gmail_default" style="color:#674ea7">I built petsc 3.13.4 and got results similar to the old ones. I am attaching the log-view output file here.</div><div class="gmail_default" style="color:#674ea7"><br></div><div class="gmail_default" style="color:#674ea7">Mohammad<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Mar 23, 2021 at 6:49 PM Satish Balay via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Tue, 23 Mar 2021, Matthew Knepley wrote:<br>
<br>
> On Tue, Mar 23, 2021 at 9:08 PM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>><br>
> wrote:<br>
> <br>
> > In the new log, I saw<br>
> ><br>
> > Summary of Stages: ----- Time ------ ----- Flop ------ --- Messages --- -- Message Lengths -- -- Reductions --<br>
> > Avg %Total Avg %Total Count %Total Avg %Total Count %Total<br>
> > 0: Main Stage: 5.4095e+00 2.3% 4.3700e+03 0.0% 4.764e+05 3.0% 3.135e+02 1.0% 2.244e+04 12.6% 1: Solute_Assembly: 1.3977e+02 59.4% 7.3353e+09 4.6% 3.263e+06 20.7% 1.278e+03 26.9% 1.059e+04 6.0%<br>
> ><br>
> ><br>
> > But I didn't see any event in this stage had a cost close to 140s. What<br>
> > happened?<br>
> ><br>
> <br>
> This is true, but all the PETSc operations are speeding up by a factor 2x.<br>
> It is hard to believe these were run on the same machine.<br>
> For example, VecScale speeds up!?! So it is not network, or optimizations.<br>
> I cannot explain this.<br>
<br>
* Using C compiler: /home/mohammad/Programs/petsc/arch-linux-c-opt/bin/mpicc -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -fvisibility=hidden -Ofast -march=native -mtune=native<br>
<br>
Perhaps the CPU is new enough that '-march=native -mtune=native' makes a difference between '18.04 to 20.04'?<br>
<br>
You can build 3.13.4 again and see if the numbers are similar to the old or new numbers you currently have..<br>
<br>
* --download-fblaslapack --download-openblas <br>
<br>
You should use one or the other - but not both. Perhaps one is using openblas in thread mode [vs single thread for the other]?<br>
<br>
Satish<br>
<br>
<br>
> <br>
> Matt<br>
> <br>
> --- Event Stage 1: Solute_Assembly<br>
> ><br>
> > BuildTwoSided 3531 1.0 2.8025e+0026.3 0.00e+00 0.0 3.6e+05 4.0e+00 3.5e+03 1 0 2 0 2 1 0 11 0 33 0<br>
> > BuildTwoSidedF 3531 1.0 2.8678e+0013.2 0.00e+00 0.0 7.1e+05 3.6e+03 3.5e+03 1 0 5 17 2 1 0 22 62 33 0<br>
> > VecScatterBegin 7062 1.0 7.1911e-02 1.9 0.00e+00 0.0 7.1e+05 3.5e+02 0.0e+00 0 0 5 2 0 0 0 22 6 0 0<br>
> > VecScatterEnd 7062 1.0 2.1248e-01 3.0 1.60e+06 2.7 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 73<br>
> > SFBcastOpBegin 3531 1.0 2.6516e-02 2.4 0.00e+00 0.0 3.6e+05 3.5e+02 0.0e+00 0 0 2 1 0 0 0 11 3 0 0<br>
> > SFBcastOpEnd 3531 1.0 9.5041e-02 4.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
> > SFReduceBegin 3531 1.0 3.8955e-02 2.1 0.00e+00 0.0 3.6e+05 3.5e+02 0.0e+00 0 0 2 1 0 0 0 11 3 0 0<br>
> > SFReduceEnd 3531 1.0 1.3791e-01 3.9 1.60e+06 2.7 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 112<br>
> > SFPack 7062 1.0 6.5591e-03 2.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
> > SFUnpack 7062 1.0 7.4186e-03 2.1 1.60e+06 2.7 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 2080<br>
> > MatAssemblyBegin 3531 1.0 4.7846e+00 1.1 0.00e+00 0.0 7.1e+05 3.6e+03 3.5e+03 2 0 5 17 2 3 0 22 62 33 0<br>
> > MatAssemblyEnd 3531 1.0 1.5468e+00 2.7 1.68e+07 2.7 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 2 0 0 0 104<br>
> > MatZeroEntries 3531 1.0 3.0998e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
> ><br>
> ><br>
> > --Junchao Zhang<br>
> ><br>
> ><br>
> ><br>
> > On Tue, Mar 23, 2021 at 5:24 PM Mohammad Gohardoust <<a href="mailto:gohardoust@gmail.com" target="_blank">gohardoust@gmail.com</a>><br>
> > wrote:<br>
> ><br>
> >> Thanks Dave for your reply.<br>
> >><br>
> >> For sure PETSc is awesome :D<br>
> >><br>
> >> Yes, in both cases petsc was configured with --with-debugging=0 and<br>
> >> fortunately I do have the old and new -log-veiw outputs which I attached.<br>
> >><br>
> >> Best,<br>
> >> Mohammad<br>
> >><br>
> >> On Tue, Mar 23, 2021 at 1:37 AM Dave May <<a href="mailto:dave.mayhem23@gmail.com" target="_blank">dave.mayhem23@gmail.com</a>> wrote:<br>
> >><br>
> >>> Nice to hear!<br>
> >>> The answer is simple, PETSc is awesome :)<br>
> >>><br>
> >>> Jokes aside, assuming both petsc builds were configured with<br>
> >>> —with-debugging=0, I don’t think there is a definitive answer to your<br>
> >>> question with the information you provided.<br>
> >>><br>
> >>> It could be as simple as one specific implementation you use was<br>
> >>> improved between petsc releases. Not being an Ubuntu expert, the change<br>
> >>> might be associated with using a different compiler, and or a more<br>
> >>> efficient BLAS implementation (non threaded vs threaded). However I doubt<br>
> >>> this is the origin of your 2x performance increase.<br>
> >>><br>
> >>> If you really want to understand where the performance improvement<br>
> >>> originated from, you’d need to send to the email list the result of<br>
> >>> -log_view from both the old and new versions, running the exact same<br>
> >>> problem.<br>
> >>><br>
> >>> From that info, we can see what implementations in PETSc are being used<br>
> >>> and where the time reduction is occurring. Knowing that, it should be<br>
> >>> clearer to provide an explanation for it.<br>
> >>><br>
> >>><br>
> >>> Thanks,<br>
> >>> Dave<br>
> >>><br>
> >>><br>
> >>> On Tue 23. Mar 2021 at 06:24, Mohammad Gohardoust <<a href="mailto:gohardoust@gmail.com" target="_blank">gohardoust@gmail.com</a>><br>
> >>> wrote:<br>
> >>><br>
> >>>> Hi,<br>
> >>>><br>
> >>>> I am using a code which is based on petsc (and also parmetis). Recently<br>
> >>>> I made the following changes and now the code is running about two times<br>
> >>>> faster than before:<br>
> >>>><br>
> >>>> - Upgraded Ubuntu 18.04 to 20.04<br>
> >>>> - Upgraded petsc 3.13.4 to 3.14.5<br>
> >>>> - This time I installed parmetis and metis directly via petsc by<br>
> >>>> --download-parmetis --download-metis flags instead of installing them<br>
> >>>> separately and using --with-parmetis-include=... and<br>
> >>>> --with-parmetis-lib=... (the version of installed parmetis was 4.0.3 before)<br>
> >>>><br>
> >>>> I was wondering what can possibly explain this speedup? Does anyone<br>
> >>>> have any suggestions?<br>
> >>>><br>
> >>>> Thanks,<br>
> >>>> Mohammad<br>
> >>>><br>
> >>><br>
> <br>
> <br>
</blockquote></div>