<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap:break-word">
<div>
<div style="color:rgb(0,0,0); font-family:Calibri,sans-serif; font-size:14px">This group of Intel people has been working with me and David for past many months.</div>
<div style="color:rgb(0,0,0); font-family:Calibri,sans-serif; font-size:14px"><br>
</div>
<div><font face="Calibri,sans-serif">Do you have any recommendation for the issue they are encountering? Can we use the multithreaded routines such as <span style="font-size:15px">MatILUFactor and MatSolve from MKL (I believe for some matrix formats- but at
least for BAIJ and AIJ)?</span></font></div>
<div><font face="Calibri,sans-serif"><span style="font-size:15px"><br>
</span></font></div>
<div><font face="Calibri,sans-serif"><span style="font-size:15px">Thanks,</span></font></div>
<div style="color:rgb(0,0,0); font-family:Calibri,sans-serif; font-size:14px"><br>
</div>
<div style="color:rgb(0,0,0); font-family:Calibri,sans-serif; font-size:14px">Dinesh</div>
</div>
<div style="color:rgb(0,0,0); font-family:Calibri,sans-serif; font-size:14px"><br>
</div>
<div style="color:rgb(0,0,0); font-family:Calibri,sans-serif; font-size:14px"><br>
</div>
<span id="OLK_SRC_BODY_SECTION" style="color:rgb(0,0,0); font-family:Calibri,sans-serif; font-size:14px">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; border-bottom:medium none; border-left:medium none; padding-bottom:0in; padding-left:0in; padding-right:0in; border-top:#b5c4df 1pt solid; border-right:medium none; padding-top:3pt">
<span style="font-weight:bold">From: </span><Mudigere>, Dheevatsa <<a href="mailto:dheevatsa.mudigere@intel.com">dheevatsa.mudigere@intel.com</a>><br>
<span style="font-weight:bold">To: </span>Dinesh Kaushik <<a href="mailto:dinesh.kaushik@kaust.edu.sa">dinesh.kaushik@kaust.edu.sa</a>>, Dinesh Kaushik <<a href="mailto:kaushik@mcs.anl.gov">kaushik@mcs.anl.gov</a>><br>
<span style="font-weight:bold">Cc: </span>David E Keyes <<a href="mailto:david.keyes@kaust.edu.sa">david.keyes@kaust.edu.sa</a>>, "Deshpande, Anand M" <<a href="mailto:anand.m.deshpande@intel.com">anand.m.deshpande@intel.com</a>><br>
<span style="font-weight:bold">Subject: </span>MKL with PETSc<br>
</div>
<div><br>
</div>
<div><style>
<!--
@font-face
{font-family:Calibri}
@font-face
{font-family:Tahoma}
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif"}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline}
a:visited, span.MsoHyperlinkFollowed
{color:purple;
text-decoration:underline}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif"}
span.EmailStyle17
{font-family:"Calibri","sans-serif";
color:windowtext}
span.BalloonTextChar
{font-family:"Tahoma","sans-serif"}
.MsoChpDefault
{font-size:10.0pt;
font-family:"Calibri","sans-serif"}
@page WordSection1
{margin:1.0in 1.0in 1.0in 1.0in}
div.WordSection1
{}
-->
</style>
<div lang="EN-US">
<div class="WordSection1">
<p class="MsoNormal">Hi Dinesh,</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">I had a question regarding interfacing MKL routines with PETSC.</p>
<p class="MsoNormal"></p>
<p class="MsoNormal">Now that we have a fairly optimized flux kernel both on the Xeon and Xeon-Phi, we are progressing on to the other key kernels. Among them, the next major contributors to the execution time are the - ILU decomposition (called within the
pre-conditioner once every time step) and the direct solver (called every inner GMRES iteration, using preconditioned matrix). Form the initial performance profile (below) it can be seen that these two operations together contribute to 31% of the sequential
execution time on a single node on Xeon and on Xeon-phi the contribution of these two operations are ~50%.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">As you would already know, the following PETSc routines are used for these operations – MatILUFactor and MatSolve. These are higher level interfaces and depending on the sparse matrix storage format – AIJ or BAIJ, the more specific lower-level
MatSymbolic, MatNumeric and MatSolve routines are used. Unfortunately, these PETSc routines are not multi-threaded and can’t leverage the available fine grained parallelism. As a first step in optimizing these operations, we want to replace these PETSc calls
with multi-threaded MKL routines. This would give us a good idea of how well these operations scale on single node (Xeon and Xeon-Phi) with multiple threads.
</p>
<p class="MsoNormal">So, it’s in this regard that I wanted your help – to know what’s the best way to reverse communicate from PETSc to use MKL routines for these operations. For now, I have managed to do this by modifying the PETSC functions themselves (MatLUFactorNumeric_SeqAIJ_Inode)
to use the MKL routines. This is somewhat a “dirty hack”, where I am shunting out the actual logic and calling the MKL functions instead and I am not taking the all the precautions to maintaining compatibility with other functions. I wanted to check with
you if there is a better and more systematic way to do this, without having to hack around with the PETSc library routines ?
</p>
<p class="MsoNormal">PETSc already supports hypre, suprelu and several other such performance libraries, is there also a way to support MKL ?</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Your help on this will be greatly appreciated.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Thanks,</p>
<p class="MsoNormal">Dheevatsa</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"><img width="341" height="332" id="Chart_x0020_1" src="cid:image002.png@01CEF7FF.C9533E00"><img width="341" height="332" id="Chart_x0020_2" src="cid:image004.png@01CEF7FF.C9533E00"></p>
</div>
</div>
</div>
</span><br>
<hr>
<font face="Arial" color="Gray" size="1"><br>
This message and its contents including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system.
Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.<br>
</font>
</body>
</html>