<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Aptos;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:12.0pt;
font-family:"Aptos",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-IE" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">Hi Barry,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">Here are a bit more details.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">Under the debugger, when I break just before
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif">PetscCall(PetscOptionsInt("-mat_mkl_pardiso_65", "Suggested number of threads to use within PARDISO", "None", threads, &threads, &flg));<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif">what I see for option “-mat_mkl_pardiso_65” in the defaultoptions is<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0"> Expression: *(defaultoptions->names+16)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0"> Address: 0x17d361e0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0"> Type: $string *<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0"> Window Status:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0"> Title: *(defaultoptions->names+16) - floops.linux64.1 - 1.1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0"> Language: C++<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0"> Valid in Scope: (Global)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0"> Compiled in Scope: PetscOptionsBegin_Private (Subroutine)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0"> aoptions.c (File)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0">libpetsc.so.3.20 (Image)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0"> UPC Phase:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0"> UPC Addr:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0"> Examine Count: 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0"> Column Count: 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0">- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0">Value<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0">0x17d35b00 -> "mat_mkl_pardiso_65"<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">And
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0"> Expression: *(defaultoptions->values+16)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0"> Address: 0x17d36330<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0"> Type: $string *<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0"> Window Status:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0"> Title: *(defaultoptions->values+16) - floops.linux64.1 - 1.1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0"> Language: C++<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0"> Valid in Scope: (Global)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0"> Compiled in Scope: PetscOptionsBegin_Private (Subroutine)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0"> aoptions.c (File)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0">libpetsc.so.3.20 (Image)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0"> UPC Phase:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0"> UPC Addr:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0"> Examine Count: 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0"> Column Count: 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0">- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0">Value<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0">0x17d35b20 -> "8"<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">I.e. exactly what I set in the loop for subPC and all other options as well. So it looks ok.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">Now when I jump into (</span><span style="font-size:13.0pt;font-family:"Calibri",sans-serif">PetscOptionsInt) I turn out here:</span><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">PetscErrorCode PetscOptionsInt_Private(PetscOptionItems *PetscOptionsObject, const char opt[], const char text[], const char man[], PetscInt currentvalue,
PetscInt *value, PetscBool *set, PetscInt lb, PetscInt ub)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">{<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US"> const char *prefix = PetscOptionsObject->prefix;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US"> const PetscOptions options = PetscOptionsObject->options;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US"> PetscBool wasset;
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">PetscOptionsObject is like this:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US"><span style="color:#0070C0"> Expression: *PetscOptionsObject<o:p></o:p></span></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0;mso-fareast-language:EN-US"> Address: 0x7fffffff6d80<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0;mso-fareast-language:EN-US"> Type: PetscOptionItems<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0;mso-fareast-language:EN-US"> Window Status:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0;mso-fareast-language:EN-US"> Title: *PetscOptionsObject - PetscOptionsInt_Private - 1.1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0;mso-fareast-language:EN-US"> Language: C++<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0;mso-fareast-language:EN-US"> Valid in Scope: PetscOptionsInt_Private (Subroutine)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0;mso-fareast-language:EN-US"> aoptions.c (File)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0;mso-fareast-language:EN-US">libpetsc.so.3.20 (Image)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0;mso-fareast-language:EN-US"> Compiled in Scope: PetscOptionsInt_Private (Subroutine)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0;mso-fareast-language:EN-US"> aoptions.c (File)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0;mso-fareast-language:EN-US">libpetsc.so.3.20 (Image)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0;mso-fareast-language:EN-US"> UPC Phase:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0;mso-fareast-language:EN-US"> UPC Addr:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0;mso-fareast-language:EN-US"> Examine Count: 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0;mso-fareast-language:EN-US"> Column Count: 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0;mso-fareast-language:EN-US">- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0;mso-fareast-language:EN-US">Field Type Value<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0;mso-fareast-language:EN-US">count PetscInt 0x00000001 (1)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0;mso-fareast-language:EN-US">next PetscOptionItem 0x00000000<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0;mso-fareast-language:EN-US">prefix $string * 0x00000000<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0;mso-fareast-language:EN-US">pprefix $string * 0x00000000<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0;mso-fareast-language:EN-US">title $string * 0x17d39140 -> "MKL_PARDISO Options"<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0;mso-fareast-language:EN-US">comm _p_PetscObject::MPI_Comm 0x84000002 (-2080374782)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0;mso-fareast-language:EN-US">printhelp PetscBool PETSC_FALSE (0)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0;mso-fareast-language:EN-US">changedmethod PetscBool PETSC_FALSE (0)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0;mso-fareast-language:EN-US">alreadyprinted PetscBool PETSC_FALSE (0)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:#0070C0;mso-fareast-language:EN-US">object PetscObject 0x00000000<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;color:red;mso-fareast-language:EN-US">options _p_PetscObject::PetscOptions 0x00000000 - options are empty<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">And then, I guess, it doesn’t see my settings and overwrites the number of threads with the default 1.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">It seems like the settings I made were not associated with PetscOptionsObject named “<span style="color:#0070C0">MKL_PARDISO Options”
</span><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">Best regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">Boris<o:p></o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Barry Smith <bsmith@petsc.dev>
<br>
<b>Sent:</b> Friday, September 27, 2024 6:27 PM<br>
<b>To:</b> Voinov, Boris <boris.voinov@intel.com><br>
<b>Cc:</b> petsc-users <petsc-users@mcs.anl.gov><br>
<b>Subject:</b> Re: [petsc-users] Ask for help in tracking down bugs - setting #threads for pardiso block LU solver<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal"> I should have read your initial email more carefully:<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">----<o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:13.0pt;font-family:"Calibri",sans-serif">Loop over block PCs {</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:13.0pt;font-family:"Calibri",sans-serif"> PCSetType(subPc, PCLU); set subPC type</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:13.0pt;font-family:"Calibri",sans-serif"> PCFactorSetMatSolverType(subPc, MATSOLVERMKL_PARDISO); set solver for LU</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:13.0pt;font-family:"Calibri",sans-serif"> SetPardisoParams("pardiso"); set mkl related options including -mat_mkl_pardiso_65 equal to the desired #threads; return code after PetscOptionsSetValue
is ok, so I presume the option is set correctly.<span class="apple-converted-space"> </span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:13.0pt;font-family:"Calibri",sans-serif"> PCSetFromOptions(subPc);</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:13.0pt;font-family:"Calibri",sans-serif">}</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:13.0pt;font-family:"Calibri",sans-serif"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:13.0pt;font-family:"Calibri",sans-serif">Then what I see under the debugger</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:13.0pt;font-family:"Calibri",sans-serif">PCSetUp after a series of calls gets to MatLUFactorSymbolic_AIJMKL_PARDISO which in turn calls MatSetFromOptions_MKL_PARDISO(F, A) and there</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif">in these lines it seems to ignore what I’ve set for<span class="apple-converted-space"> </span></span><span lang="EN-US" style="font-size:13.0pt;font-family:"Calibri",sans-serif">-mat_mkl_pardiso_65
and sets #threads to the default 1</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif">PetscCall(PetscOptionsInt("-mat_mkl_pardiso_65", "Suggested number of threads to use within PARDISO", "None", threads, &threads, &flg));</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif">if (flg) PetscSetMKL_PARDISOThreads((int)threads);</span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif">-----</span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif">PCSetFromOptions() actually does not call MatSetFromOptions() on the factored matrix. There is no call to MatSetFromOptions() from PCSetFromOptions() or PetscSetFromOptions_Factor()
because the factored matrix F does not yet exist (it is created in PCSetUp()). This is why your call to PCSetFromOptions(subPc); does not process any of your mat mkl options.</span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif">Since MatSetFromOptions() cannot be called from PCSetFromOptions(), MatSetFromOptions_MKL_PARDISO(F, A) is called in </span> MatFactorSymbolic_AIJMKL_PARDISO_Private() as you
noted when the PC is being set up.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">You say "<span style="font-size:13.0pt;font-family:"Calibri",sans-serif">here in these lines it seems to ignore what I’ve set for<span class="apple-converted-space"> </span></span><span lang="EN-US" style="font-size:13.0pt;font-family:"Calibri",sans-serif">-mat_mkl_pardiso_65
and sets #threads to the default 1"</span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:13.0pt;font-family:"Calibri",sans-serif">This is the part a do not understand, if your values have been inserted in the database earlier in the code and remain in the database they should get processed
inside the</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:13.0pt;font-family:"Calibri",sans-serif">MatSetFromOptions_MKL_PARDISO(F, A) called by </span>MatFactorSymbolic_AIJMKL_PARDISO_Private(). <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Can you please put a break point at <span style="font-size:13.0pt;font-family:"Calibri",sans-serif">MatSetFromOptions_MKL_PARDISO(F, A) and then when it breaks put</span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif">call PetscOptionsView(0,0)</span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif">it should print out the contents of the options database so you can verify if the options you added are there. You can then step through the code in MatSetFromOptions_MKL_PARDISO
to see what is happening as it tries to acess the database to see why it skips </span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">PetscCall(PetscOptionsInt("-mat_mkl_pardiso_65", "Suggested number of threads to use within MKL PARDISO", "None", threads, &threads, &flg));<o:p></o:p></p>
<p class="MsoNormal"> if (flg) PetscSetMKL_PARDISOThreads((int)threads);<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif"> </span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On Sep 27, 2024, at 11:58<span style="font-family:"Arial",sans-serif"> </span>AM, Voinov, Boris <<a href="mailto:boris.voinov@intel.com">boris.voinov@intel.com</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif">Thanks a lot Barry,</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif">It looks like something wrong with my own settings though. It doesn’t make an effect. I mean this call</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:13.0pt;font-family:"Calibri",sans-serif">SetPardisoParams("pardiso");</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:13.0pt;font-family:"Calibri",sans-serif">followed by</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:13.0pt;font-family:"Calibri",sans-serif">PCSetFromOptions(subPc);</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:13.0pt;font-family:"Calibri",sans-serif">doesn’t end up with a call of PetscSetMKL_PARDISOThreads()</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:13.0pt;font-family:"Calibri",sans-serif">This function is called later when PCSetUp is invoked.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:13.0pt;font-family:"Calibri",sans-serif"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:13.0pt;font-family:"Calibri",sans-serif">Eventually what I see in the end is</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:13.0pt;font-family:"Calibri",sans-serif">Option left: name:-mat_mkl_pardiso_65 value: 8 source: code listed among unused database options</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:13.0pt;font-family:"Calibri",sans-serif">That is what I’ve set with my own settings, meaning it’s in the database, I guess.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:13.0pt;font-family:"Calibri",sans-serif"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:13.0pt;font-family:"Calibri",sans-serif">Could it be that it hits a “not proper” section in the options database and is not being used by PCSetUp?</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:13.0pt;font-family:"Calibri",sans-serif"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:13.0pt;font-family:"Calibri",sans-serif">Best regards,</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:13.0pt;font-family:"Calibri",sans-serif">Boris</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:13.0pt;font-family:"Calibri",sans-serif"> </span><o:p></o:p></p>
</div>
<div>
<div style="border:none;border-top:solid windowtext 1.0pt;padding:3.0pt 0cm 0cm 0cm;border-color:currentcolor currentcolor;border-image: none">
<div>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span class="apple-converted-space"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span></span><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">Barry
Smith <<a href="mailto:bsmith@petsc.dev"><span style="color:#0563C1">bsmith@petsc.dev</span></a>><br>
<b>Sent:</b><span class="apple-converted-space"> </span>Friday, September 27, 2024 4:21 PM<br>
<b>To:</b><span class="apple-converted-space"> </span>Voinov, Boris <<a href="mailto:boris.voinov@intel.com"><span style="color:#0563C1">boris.voinov@intel.com</span></a>><br>
<b>Cc:</b><span class="apple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov"><span style="color:#0563C1">petsc-users@mcs.anl.gov</span></a><br>
<b>Subject:</b><span class="apple-converted-space"> </span>Re: [petsc-users] Ask for help in tracking down bugs - setting #threads for pardiso block LU solver</span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"> Boris,<o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> There is nothing obviously wrong with what you outline, so let's dive down into the code with the debugger and see what is happening.<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">PetscSetMKL_PARDISOThreads() set from options is called when the command line option is provided, and this calls <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal">PETSC_EXTERN void PetscSetMKL_PARDISOThreads(int threads)<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">{<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> mkl_domain_set_num_threads(threads, MKL_DOMAIN_PARDISO);<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">}<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">so in the debugger can you check that mkl_domain_set_num_threads() gets called with your requested number of threads?<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">BTW: Since you are hardwiring the use of Pardiso in the code with function calls you couldsimply call PetscSetMKL_PARDISOThreads() directly<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">after your line of code<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif">PCFactorSetMatSolverType(subPc, MATSOLVERMKL_PARDISO); set solver for LU</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif">instead of feeding it in through the options database.</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif">Let us know how this turns out</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif"> Barry</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><br>
<br>
<br>
<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal">On Sep 27, 2024, at 7:24<span style="font-family:"Arial",sans-serif"> </span>AM, Voinov, Boris <<a href="mailto:boris.voinov@intel.com"><span style="color:#0563C1">boris.voinov@intel.com</span></a>> wrote:<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:13.0pt;font-family:"Calibri",sans-serif">Hello,</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:13.0pt;font-family:"Calibri",sans-serif"> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:13.0pt;font-family:"Calibri",sans-serif">Could you please help me to figure out what’s wrong with the way I’m trying to make mkl pardiso use the number of threads more than one while setting it as a sub
PC solver for ASM preconditioner.</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:13.0pt;font-family:"Calibri",sans-serif">Here is how I do this</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:13.0pt;font-family:"Calibri",sans-serif">Loop over block PCs {</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:13.0pt;font-family:"Calibri",sans-serif"> PCSetType(subPc, PCLU); set subPC type</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:13.0pt;font-family:"Calibri",sans-serif"> PCFactorSetMatSolverType(subPc, MATSOLVERMKL_PARDISO); set solver for LU</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:13.0pt;font-family:"Calibri",sans-serif"> SetPardisoParams("pardiso"); set mkl related options including -mat_mkl_pardiso_65 equal to the desired #threads; return code after PetscOptionsSetValue
is ok, so I presume the option is set correctly.<span class="apple-converted-space"> </span></span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:13.0pt;font-family:"Calibri",sans-serif"> PCSetFromOptions(subPc);</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:13.0pt;font-family:"Calibri",sans-serif">}</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:13.0pt;font-family:"Calibri",sans-serif"> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:13.0pt;font-family:"Calibri",sans-serif">Then what I see under the debugger</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:13.0pt;font-family:"Calibri",sans-serif">PCSetUp after a series of calls gets to MatLUFactorSymbolic_AIJMKL_PARDISO which in turn calls MatSetFromOptions_MKL_PARDISO(F, A) and there</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif">in these lines it seems to ignore what I’ve set for<span class="apple-converted-space"> </span></span><span lang="EN-US" style="font-size:13.0pt;font-family:"Calibri",sans-serif">-mat_mkl_pardiso_65
and sets #threads to the default 1</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif">PetscCall(PetscOptionsInt("-mat_mkl_pardiso_65", "Suggested number of threads to use within PARDISO", "None", threads, &threads, &flg));</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif">if (flg) PetscSetMKL_PARDISOThreads((int)threads);</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif"> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif">This is about petsc-3.20.</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif">PS when I do this in a standalone code which reads a matrix and solves the linear system and set<span class="apple-converted-space"> </span></span><span lang="EN-US" style="font-size:13.0pt;font-family:"Calibri",sans-serif">-mat_mkl_pardiso_65
in the command line it’s all right but I need the number of threads to be set in other way in my code.</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif"> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif">Thank you and best regards,</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif">Boris</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:13.0pt;font-family:"Calibri",sans-serif"> </span><o:p></o:p></p>
</div>
</div>
<p style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;text-align:start;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span style="font-size:13.5pt;font-family:"Helvetica",sans-serif">-------------------------------------------------------------<br>
Intel Ireland Limited (Branch)<br>
Collinstown Industrial Park, Leixlip, County Kildare, Ireland<br>
Registered Number: E902934<o:p></o:p></span></p>
<p style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;text-align:start;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span style="font-size:13.5pt;font-family:"Helvetica",sans-serif">This e-mail and any attachments may contain confidential material for<br>
the sole use of the intended recipient(s). Any review or distribution<br>
by others is strictly prohibited. If you are not the intended<br>
recipient, please contact the sender and delete all copies.<o:p></o:p></span></p>
</div>
</blockquote>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<p style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;text-align:start;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span style="font-size:13.5pt;font-family:"Helvetica",sans-serif">-------------------------------------------------------------<br>
Intel Ireland Limited (Branch)<br>
Collinstown Industrial Park, Leixlip, County Kildare, Ireland<br>
Registered Number: E902934<o:p></o:p></span></p>
<p style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;text-align:start;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span style="font-size:13.5pt;font-family:"Helvetica",sans-serif">This e-mail and any attachments may contain confidential material for<br>
the sole use of the intended recipient(s). Any review or distribution<br>
by others is strictly prohibited. If you are not the intended<br>
recipient, please contact the sender and delete all copies.<o:p></o:p></span></p>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<p>-------------------------------------------------------------<br>
Intel Ireland Limited (Branch)<br>
Collinstown Industrial Park, Leixlip, County Kildare, Ireland<br>
Registered Number: E902934</p>
<p>This e-mail and any attachments may contain confidential material for<br>
the sole use of the intended recipient(s). Any review or distribution<br>
by others is strictly prohibited. If you are not the intended<br>
recipient, please contact the sender and delete all copies.</p></body>
</html>