<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Mon, Jun 19, 2017 at 6:42 AM, Damian Kaliszan <span dir="ltr"><<a href="mailto:damian@man.poznan.pl" target="_blank">damian@man.poznan.pl</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<span style="font-family:"courier new";font-size:9pt">Hi,<br>
<br>
Regarding my previous post <br>
I looked into both logs of 64MPI/1 OMP vs. 64MPI/2 OMP.<br>
<br>
<br>
<span style="font-family:Tahoma;font-size:10pt">What attracted my attention is huge difference in MPI timings in the following places:<br>
<br>
Average time to get PetscTime(): 2.14577e-07<br>
Average time for MPI_Barrier(): 3.9196e-05<br>
Average time for zero size MPI_Send(): 5.45382e-06<br>
<br>
vs. <br>
<br>
Average time to get PetscTime(): 4.05312e-07<br>
Average time for MPI_Barrier(): 0.348399<br>
Average time for zero size MPI_Send(): 0.029937<br><br>
Isn't something wrong with PETSc library itself?...<br></span></span></div></blockquote><div><br></div><div> I don't think so. This is bad interaction of MPI and your threading mechanism. MPI_Barrier() and MPI_Send() are lower</div><div>level than PETSc. What threading mode did you choose for MPI? This can have a performance impact.</div><div><br></div><div>Also, the justifications for threading in this context are weak (or non-existent): <a href="http://www.orau.gov/hpcor2015/whitepapers/Exascale_Computing_without_Threads-Barry_Smith.pdf">http://www.orau.gov/hpcor2015/whitepapers/Exascale_Computing_without_Threads-Barry_Smith.pdf</a></div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><span style="font-family:"courier new";font-size:9pt"><span style="font-family:Tahoma;font-size:10pt">
<br>
Best,<br>
Damian<br>
<br>
<span style="font-family:"courier new";font-size:9pt">Wiadomość przekazana<br>
Od: Damian Kaliszan <<a href="mailto:damian@man.poznan.pl" target="_blank">damian@man.poznan.pl</a>><br>
Do: PETSc users list <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
Data: 16 czerwca 2017, 14:57:10<br>
Temat: [petsc-users] strange PETSc/KSP GMRES timings for MPI+OMP configuration on KNLs<br>
<br>
===8<===============Treść oryginalnej wiadomości===============<br>
<span style="font-family:Tahoma;font-size:10pt">Hi,<br>
<br>
For several days I've been trying to figure out what is going wrong<br>
with my python app timings solving Ax=b with KSP (GMRES) solver when trying to run on Intel's KNL 7210/7230.<br>
<br>
I downsized the problem to 1000x1000 A matrix and a single node and<br>
observed the following:<br>
<br>
<img width="482" height="260" alt="" style="padding: 1px;" src="cid:7753EFE8.01D2E8F9.4179B2C6.59CB5BE5_csseditor"><br>
I'm attaching 2 extreme timings where configurations differ only by 1 OMP thread (64MPI/1 OMP vs 64/2 OMPs),<br>
23321 vs 23325 slurm task ids.<br>
<br>
Any help will be appreciated....<br>
<br>
Best,<br>
Damian <br>
<br>
===8<===========Koniec treści oryginalnej wiadomości===========<br>
<br>
<br>
<br>
------------------------------<wbr>-------------------------<br>
Damian Kaliszan<br>
<br>
Poznan Supercomputing and Networking Center<br>
HPC and Data Centres Technologies<br>
ul. Jana Pawła II 10<br>
61-139 Poznan<br>
POLAND<br>
<br>
phone <a href="tel:+48%2061%20858%2051%2009" value="+48618585109" target="_blank">(+48 61) 858 5109</a><br>
e-mail </span></span></span></span><a style="font-family:"courier new";font-size:9pt" href="mailto:damian@man.poznan.pl" target="_blank">damian@man.poznan.pl</a><br>
<span style="font-family:"courier new";font-size:9pt">www - </span><a style="font-family:"courier new";font-size:9pt" href="http://www.man.poznan.pl/" target="_blank">http://www.man.poznan.pl/</a><br>
<span style="font-family:"courier new";font-size:9pt">------------------------------<wbr>------------------------- </span></div><br><br>---------- Forwarded message ----------<br>From: Damian Kaliszan <<a href="mailto:damian@man.poznan.pl">damian@man.poznan.pl</a>><br>To: PETSc users list <<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>><br>Cc: <br>Bcc: <br>Date: Fri, 16 Jun 2017 14:57:10 +0200<br>Subject: [petsc-users] strange PETSc/KSP GMRES timings for MPI+OMP configuration on KNLs<br>
<div>
<span style="font-family:Tahoma;font-size:10pt">Hi,<br>
<br>
For several days I've been trying to figure out what is going wrong<br>
with my python app timings solving Ax=b with KSP (GMRES) solver when trying to run on Intel's KNL 7210/7230.<br>
<br>
I downsized the problem to 1000x1000 A matrix and a single node and<br>
observed the following:<br>
<br>
<img width="482" height="260" alt="" style="padding: 1px;" src="cid:113AA549.01D2E6A0.1F35115E.77F210D8_csseditor"><br>
I'm attaching 2 extreme timings where configurations differ only by 1 OMP thread (64MPI/1 OMP vs 64/2 OMPs),<br>
23321 vs 23325 slurm task ids.<br>
<br>
Any help will be appreciated....<br>
<br>
Best,<br>
Damian</span></div><br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.caam.rice.edu/~mk51/" target="_blank">http://www.caam.rice.edu/~mk51/</a><br></div></div></div>
</div></div>