<div dir="ltr"><div>Hi, Richard</div><div><br></div><div>thank you for the detailed information. This was our guess as well.</div><div><br></div><div>Regards,</div><div><br></div><div>Karl<br></div></div><br><div class="gmail_quote"><div class="gmail_attr" dir="ltr">On Thu, Jul 18, 2019 at 2:36 PM Mills, Richard Tran via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid">
<div bgcolor="#FFFFFF">
Hi Kun and Karl,<br>
<br>
If you are using the AIJMKL matrix types and have a recent version of MKL, the AIJMKL code uses MKL's inspector-executor sparse BLAS routines, which are described at<br>
<br>
<a href="https://software.intel.com/en-us/mkl-developer-reference-c-inspector-executor-sparse-blas-routines" target="_blank">https://software.intel.com/en-us/mkl-developer-reference-c-inspector-executor-sparse-blas-routines</a><br>
<br>
The inspector-executor analysis routines take the AIJ (compressed sparse row) format data from PETSc and then create a copy in an optimized, internal layout used by MKL. We have to keep PETSc's own, AIJ representation around, as it is needed for several operations
that MKL does not provide. This does, unfortunately, mean that roughly double (or more, depending on what MKL decides to do) the amount of memory is required. The reason you see the memory usage increase right when a MatMult() or MatMultTranspose() operation
occurs is that the we default to a "lazy" approach to calling the analysis routine (mkl_sparse_optimize()) until an operation that uses an MKL-provided kernel is requested. (You can use an "eager" approach that calls mkl_sparse_optimize() during MatAssemblyEnd()
by specifying "-mat_aijmkl_eager_inspection" in the PETSc options.)<br>
<br>
If memory is at enough of a premium for you that you can't afford the extra copy used by the MKL inspector-executor routines, then I suggest using the usual PETSc AIJ format instead of AIJMKL. AIJ is fairly well optimized for many cases (and even has some hand-optimized
kernels using Intel AVX/AVX2/AVX-512 intrinsics) and often outperforms AIJMKL. You should try both AIJ and AIJMKL, anyway, to see which is faster for your combination of problem and computing platform.<br>
<br>
Best regards,<br>
Richard<br>
<br>
<div class="gmail-m_1125677480200095237moz-cite-prefix">On 7/17/19 8:46 PM, Karl Lin via petsc-users wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">We also found that if we use MatCreateSeqAIJ, then no more memory increase with matrix vector multiplication. However, with MatCreateMPIAIJMKL, the behavior is consistent.<br>
</div>
<br>
<div class="gmail_quote">
<div class="gmail_attr" dir="ltr">On Wed, Jul 17, 2019 at 5:26 PM Karl Lin <<a href="mailto:karl.linkui@gmail.com" target="_blank">karl.linkui@gmail.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid">
<div dir="ltr">
<div dir="ltr">MatCreateMPIAIJMKL</div>
<div dir="ltr"><br>
</div>
<div>parallel and sequential exhibit the same behavior. In fact, we found that doing matmult will increase the memory by the size of matrix as well.<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div class="gmail_attr" dir="ltr">On Wed, Jul 17, 2019 at 4:55 PM Zhang, Hong <<a href="mailto:hzhang@mcs.anl.gov" target="_blank">hzhang@mcs.anl.gov</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid">
<div>
<div dir="ltr">
<div dir="ltr">Karl:<br>
</div>
<div>What matrix format do you use? Run it in parallel or sequential?</div>
<div>Hong </div>
<div><br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid">
<div dir="ltr">
<div dir="ltr">We used /proc/self/stat to track the resident set size during program run, and we saw the resident set size jumped by the size of the matrix right after we did matmulttranspose.<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div class="gmail_attr" dir="ltr">On Wed, Jul 17, 2019 at 12:04 PM hong--- via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid">
<div dir="ltr">
<div dir="ltr">Kun:<br>
</div>
<div>How do you know 'MatMultTranpose creates an extra memory copy of matrix'?</div>
<div>Hong</div>
<div><br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid">
<div lang="EN-US">
<div class="gmail-m_1125677480200095237gmail-m_-5292850215921239070gmail-m_980216059706594888gmail-m_-6664523132381501245gmail-m_-8066335926639544998gmail-m_-3538158395817514375WordSection1">
<p class="MsoNormal">Hi,</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">I was using MatMultTranpose and MatMult to solver a linear system.
</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">However we found out, MatMultTranpose create an extra memory copy of matrix for its operation. This extra memory copy is not stated everywhere in petsc manual.
</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">This basically double my memory requirement to solve my system.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">I remember mkl’s routine can do inplace matrix transpose vector product, without transposing the matrix itself.
</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Is this always the case? Or there is way to make petsc to do inplace matrix transpose vector product.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Any help is greatly appreciated.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Regards,</p>
<p class="MsoNormal">Kun</p>
<p class="MsoNormal"> </p>
<br>
<p align="Center" class="gmail-m_1125677480200095237gmail-m_-5292850215921239070gmail-m_980216059706594888gmail-m_-6664523132381501245gmail-m_-8066335926639544998gmail-m_-3538158395817514375msipfooter6d2e06ff" style="margin:0px">
<span style="color:rgb(0,0,0);font-family:Calibri;font-size:10pt">Schlumberger-Private</span></p>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
<br>
</div>
</blockquote></div>