<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Tue, Oct 10, 2017 at 1:03 PM, Mark Adams <span dir="ltr"><<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div dir="ltr">putting this back on the list.<br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Oct 10, 2017 at 3:21 PM, Bakytzhan Kallemov <span dir="ltr">
<<a href="mailto:bkallemov@lbl.gov" target="_blank">bkallemov@lbl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<p><br>
</p>
<div class="m_5330784447021561393m_-5167551645553413926moz-forward-container"><br>
<br>
-------- Forwarded Message --------
<table class="m_5330784447021561393m_-5167551645553413926moz-email-headers-table" cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr>
<th valign="BASELINE" nowrap align="RIGHT">Subject: </th>
<td>Re: [petsc-dev] development version for MATAIJMKL type mat</td>
</tr>
<tr>
<th valign="BASELINE" nowrap align="RIGHT">Date: </th>
<td>Tue, 10 Oct 2017 12:18:08 -0700</td>
</tr>
<tr>
<th valign="BASELINE" nowrap align="RIGHT">From: </th>
<td>Bakytzhan Kallemov <a class="m_5330784447021561393m_-5167551645553413926moz-txt-link-rfc2396E" href="mailto:bkallemov@lbl.gov" target="_blank">
<bkallemov@lbl.gov></a></td>
</tr>
<tr>
<th valign="BASELINE" nowrap align="RIGHT">To: </th>
<td>Barry Smith <a class="m_5330784447021561393m_-5167551645553413926moz-txt-link-rfc2396E" href="mailto:bsmith@mcs.anl.gov" target="_blank">
<bsmith@mcs.anl.gov></a></td>
</tr>
</tbody>
</table>
<br>
<br>
<pre>Hi Barry,
Yes I am using OMP_NUM_THREADS to control the number of threads.
I talked to Mark and he suggested to use this developer version of the
code to use openmp threading when running petsc.
May be it's better to explain what I am trying to get.
Attached please see the plot for average time for a time step
advance(which reflects mostly PCApply, I believe, but I can generate the
separate plot for that later) running on a single KNL node for different
combination mpi+openmp.
My goal is to get an advantage by using openmp threads in hybrid
running, so I am trying to get the plot more flat.</pre>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>This first thing to do is verify that everything is hooked up correctly. The threaded version of hypre does look a little slower. Is this data repeatable? This could just be experimental noise.</div>
<div><br>
</div>
<div>I would run these in the same script so that you know you get the same node and "environment". We really just need one data point, say 32 threads per 2 MPI processes per socket, or whatever your goal is.</div>
<div><br>
</div>
<div>You would need to talk with hypre about (performance) debugging/optimizing hypre. We know more about gamg (I wrote it) and it uses native PETSc primitives so we get useful data with -log_view. If you do this with gamg I can look at the output and check
that everything is hooked up correctly.</div>
<div><br>
</div>
<div>Oh, actually, AIJMKL is not quite working yet for AMG because the matrix-matrix product methods are not hooked up. This should be happening in the next few months. So I'd talk with hypre. They have methods to get performance data that they can analyses.</div></div></div></div></div></blockquote><div><br></div><div>Mark is right: Try Hypre first for now. However, I'm hoping to get AIJMKL using the MKL sparse matrix-matrix multiply working this week. (So, probably sometime next week, in actual practice. =) ). I'll let everyone know when I have this working.</div><div><br></div><div>Note that my guess is that, when running with a single thread, the MKL sparse matrix-matrix multiply will be somewhat slower than the PETSc version (when compiling with the Intel compiler, which does a pretty good job). Using MKL will allow you to use MKL's threading, though, so if this is important to your application, this may be worth giving a try. Also, any performance problems we see we should report to the MKL team, which may have some motivation to improve them if we are making it easy for PETSc applications to use them.<br></div><div><br></div><div>--Richard</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">
<div><br>
</div>
<div>Mark</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div class="m_5330784447021561393m_-5167551645553413926moz-forward-container">
<pre>Is this something that I can do with new development such as aifmkl mat
type?
Thanks,
Baky
On 10/10/2017 11:50 AM, Barry Smith wrote:
>> On Oct 10, 2017, at 10:52 AM, Bakytzhan Kallemov <a class="m_5330784447021561393m_-5167551645553413926moz-txt-link-rfc2396E" href="mailto:bkallemov@lbl.gov" target="_blank"><bkallemov@lbl.gov></a> wrote:
>>
>> Hi,
>>
>> My name is Baky Kallemov.
>>
>> Currently, I am working on improving a scalibility of the Chombo-Petsc interface on cori machine at nersc system.
>>
>> I successfully build the libs from master branch with --with-openmp and hypre.
>>
>> However, I have not noticed any difference running my test problem on single node KNL node using new MATAIJMKL
> hyre uses its own matrix operations so it won't get faster when using running PETSc with MATAIJMKL or any other specific matrix type.
>>
>> type for different hybrid mpi+openmp runs compared to regular released version.
> What are you comparing? Are you using say 32 MPI processes and 2 threads or 16 MPI processes and 4 threads? How are you controlling the number of OpenMP threads, OpenMP environmental variable? What parts of the time in the code are you comparing? You should just -log_view and compare the times for PCApply and PCSetUp() between say 64 MPI process/1 thread and 32 MPI processes/2 threads and send us the output for those two cases.
>
>> It seems that it made no difference, so perhaps I am doing something wrong or my build is not configured right.
>>
>> Do you have any example that makes use of threads when running hybrid and show an advantage?
> There is not reason to think that using threads on KNL is faster than just using MPI processes. Despite what the NERSc/LBL web pages may say, just because a website says something doesn't make it true.
>
>
>> I'd like to test it and make sure that my libs are configured correctly, before start to investigate it further.
>>
>>
>> Thanks,
>>
>> Baky
>>
>>
</pre>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote></div><br></div></div>