<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div><br></div> I should have read your initial email more carefully:<div><br></div><div>----</div><div><div><p class="MsoNormal" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 13pt; font-family: Calibri, sans-serif;">Loop over block PCs {</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><o:p></o:p></span></p></div><div><p class="MsoNormal" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 13pt; font-family: Calibri, sans-serif;"> PCSetType(subPc, PCLU); set subPC type</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><o:p></o:p></span></p></div><div><p class="MsoNormal" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 13pt; font-family: Calibri, sans-serif;"> PCFactorSetMatSolverType(subPc, MATSOLVERMKL_PARDISO); set solver for LU</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><o:p></o:p></span></p></div><div><p class="MsoNormal" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 13pt; font-family: Calibri, sans-serif;"> SetPardisoParams("pardiso"); set mkl related options including -mat_mkl_pardiso_65 equal to the desired #threads; return code after PetscOptionsSetValue is ok, so I presume the option is set correctly.<span class="apple-converted-space"> </span></span><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><o:p></o:p></span></p></div><div><p class="MsoNormal" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 13pt; font-family: Calibri, sans-serif;"> PCSetFromOptions(subPc);</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><o:p></o:p></span></p></div><div><p class="MsoNormal" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 13pt; font-family: Calibri, sans-serif;">}</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><o:p></o:p></span></p></div><div><p class="MsoNormal" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 13pt; font-family: Calibri, sans-serif;"> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><o:p></o:p></span></p></div><div><p class="MsoNormal" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 13pt; font-family: Calibri, sans-serif;">Then what I see under the debugger</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><o:p></o:p></span></p></div><div><p class="MsoNormal" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 13pt; font-family: Calibri, sans-serif;">PCSetUp after a series of calls gets to MatLUFactorSymbolic_AIJMKL_PARDISO which in turn calls MatSetFromOptions_MKL_PARDISO(F, A) and there</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><o:p></o:p></span></p></div><div><p class="MsoNormal" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 13pt; font-family: Calibri, sans-serif;">in these lines it seems to ignore what I’ve set for<span class="apple-converted-space"> </span></span><span lang="EN-US" style="font-size: 13pt; font-family: Calibri, sans-serif;">-mat_mkl_pardiso_65 and sets #threads to the default 1</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><o:p></o:p></span></p></div><div><p class="MsoNormal" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 13pt; font-family: Calibri, sans-serif;">PetscCall(PetscOptionsInt("-mat_mkl_pardiso_65", "Suggested number of threads to use within PARDISO", "None", threads, &threads, &flg));</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><o:p></o:p></span></p></div><div><p class="MsoNormal" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 13pt; font-family: Calibri, sans-serif;">if (flg) PetscSetMKL_PARDISOThreads((int)threads);</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><o:p></o:p></span></p><p class="MsoNormal" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 13pt; font-family: Calibri, sans-serif;"><br></span></p><p class="MsoNormal" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 13pt; font-family: Calibri, sans-serif;">-----</span></p><p class="MsoNormal" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 13pt; font-family: Calibri, sans-serif;"><br></span></p><p class="MsoNormal" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 13pt; font-family: Calibri, sans-serif;">PCSetFromOptions() actually does not call MatSetFromOptions() on the factored matrix. There is no call to MatSetFromOptions() from PCSetFromOptions() or PetscSetFromOptions_Factor() because the factored matrix F does not yet exist (it is created in PCSetUp()). This is why your call to </span><span style="font-family: Calibri, sans-serif; font-size: 17.333334px;">PCSetFromOptions(subPc); does not process any of your mat mkl options.</span></p><p class="MsoNormal" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-family: Calibri, sans-serif; font-size: 17.333334px;"><br></span></p><p class="MsoNormal" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-family: Calibri, sans-serif; font-size: 17.333334px;">Since MatSetFromOptions() cannot be called from </span><span style="font-family: Calibri, sans-serif; font-size: 17.333334px;">PCSetFromOptions(), </span><span style="font-family: Calibri, sans-serif; font-size: 17.333334px;">MatSetFromOptions_MKL_PARDISO(F, A) is called in </span> MatFactorSymbolic_AIJMKL_PARDISO_Private() as you noted when the PC is being set up.</p><p class="MsoNormal" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><br></p><p class="MsoNormal" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">You say "<span style="font-family: Calibri, sans-serif; font-size: 13pt;">here </span><span style="font-size: 13pt; font-family: Calibri, sans-serif;">in these lines it seems to ignore what I’ve set for<span class="apple-converted-space"> </span></span><span lang="EN-US" style="font-size: 13pt; font-family: Calibri, sans-serif;">-mat_mkl_pardiso_65 and sets #threads to the default 1"</span></p><p class="MsoNormal" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><br></p><p class="MsoNormal" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 13pt; font-family: Calibri, sans-serif;">This is the part a do not understand, if your values have been inserted in the database earlier in the code and remain in the database they should get processed inside the</span></p><p class="MsoNormal" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 13pt; font-family: Calibri, sans-serif;">MatSetFromOptions_MKL_PARDISO(F, A) called by </span>MatFactorSymbolic_AIJMKL_PARDISO_Private(). </p><p class="MsoNormal" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><br></p><p class="MsoNormal" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">Can you please put a break point at <span style="font-family: Calibri, sans-serif; font-size: 17.333334px;">MatSetFromOptions_MKL_PARDISO(F, A) and then when it breaks put</span></p><p class="MsoNormal" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-family: Calibri, sans-serif; font-size: 17.333334px;"><br></span></p><p class="MsoNormal" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-family: Calibri, sans-serif; font-size: 17.333334px;">call PetscOptionsView(0,0)</span></p><p class="MsoNormal" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-family: Calibri, sans-serif; font-size: 17.333334px;"><br></span></p><p class="MsoNormal" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-family: Calibri, sans-serif; font-size: 17.333334px;">it should print out the contents of the options database so you can verify if the options you added are there. You can then step through the code in </span><span style="font-family: Calibri, sans-serif; font-size: 17.333334px;">MatSetFromOptions_MKL_PARDISO to see what is happening as it tries to acess the database to see why it skips </span></p><p class="MsoNormal" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-family: Calibri, sans-serif; font-size: 17.333334px;"><br></span></p><p class="MsoNormal" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">PetscCall(PetscOptionsInt("-mat_mkl_pardiso_65", "Suggested number of threads to use within MKL PARDISO", "None", threads, &threads, &flg));</p><p class="MsoNormal" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> if (flg) PetscSetMKL_PARDISOThreads((int)threads);</p><p class="MsoNormal" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><br></p><p class="MsoNormal" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><br></p><p class="MsoNormal" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-family: Calibri, sans-serif; font-size: 17.333334px;"><br></span></p><p class="MsoNormal" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 13pt; font-family: Calibri, sans-serif;"><br></span></p><p class="MsoNormal" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><br></p><p class="MsoNormal" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><br></p><p class="MsoNormal" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 13pt; font-family: Calibri, sans-serif;"><br></span></p><p class="MsoNormal" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 13pt; font-family: Calibri, sans-serif;"><br></span></p></div><div><p class="MsoNormal" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 13pt; font-family: Calibri, sans-serif;"> </span></p><p class="MsoNormal" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 13pt; font-family: Calibri, sans-serif;"><br></span></p></div><div><br><blockquote type="cite"><div>On Sep 27, 2024, at 11:58 AM, Voinov, Boris <boris.voinov@intel.com> wrote:</div><br class="Apple-interchange-newline"><div><meta charset="UTF-8"><div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 13pt; font-family: Calibri, sans-serif;">Thanks a lot Barry,<o:p></o:p></span></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 13pt; font-family: Calibri, sans-serif;"><o:p> </o:p></span></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 13pt; font-family: Calibri, sans-serif;">It looks like something wrong with my own settings though. It doesn’t make an effect. I mean this call<o:p></o:p></span></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 13pt; font-family: Calibri, sans-serif;">SetPardisoParams("pardiso");<o:p></o:p></span></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 13pt; font-family: Calibri, sans-serif;">followed by<o:p></o:p></span></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 13pt; font-family: Calibri, sans-serif;">PCSetFromOptions(subPc);<o:p></o:p></span></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 13pt; font-family: Calibri, sans-serif;">doesn’t end up with a call of PetscSetMKL_PARDISOThreads()<o:p></o:p></span></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 13pt; font-family: Calibri, sans-serif;">This function is called later when PCSetUp is invoked.<o:p></o:p></span></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 13pt; font-family: Calibri, sans-serif;"><o:p> </o:p></span></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 13pt; font-family: Calibri, sans-serif;">Eventually what I see in the end is<o:p></o:p></span></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 13pt; font-family: Calibri, sans-serif;">Option left: name:-mat_mkl_pardiso_65 value: 8 source: code listed among unused database options<o:p></o:p></span></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 13pt; font-family: Calibri, sans-serif;">That is what I’ve set with my own settings, meaning it’s in the database, I guess.<o:p></o:p></span></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 13pt; font-family: Calibri, sans-serif;"><o:p> </o:p></span></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 13pt; font-family: Calibri, sans-serif;">Could it be that it hits a “not proper” section in the options database and is not being used by PCSetUp?<o:p></o:p></span></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 13pt; font-family: Calibri, sans-serif;"><o:p> </o:p></span></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 13pt; font-family: Calibri, sans-serif;">Best regards,<o:p></o:p></span></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 13pt; font-family: Calibri, sans-serif;">Boris<o:p></o:p></span></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 13pt; font-family: Calibri, sans-serif;"><o:p> </o:p></span></div><div><div style="border-width: 1pt medium medium; border-style: solid none none; border-color: rgb(225, 225, 225) currentcolor currentcolor; border-image: none; padding: 3pt 0cm 0cm;"><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><b><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;">From:</span></b><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;"><span class="Apple-converted-space"> </span>Barry Smith <<a href="mailto:bsmith@petsc.dev" style="color: rgb(5, 99, 193); text-decoration: underline;">bsmith@petsc.dev</a>><br><b>Sent:</b><span class="Apple-converted-space"> </span>Friday, September 27, 2024 4:21 PM<br><b>To:</b><span class="Apple-converted-space"> </span>Voinov, Boris <<a href="mailto:boris.voinov@intel.com" style="color: rgb(5, 99, 193); text-decoration: underline;">boris.voinov@intel.com</a>><br><b>Cc:</b><span class="Apple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov" style="color: rgb(5, 99, 193); text-decoration: underline;">petsc-users@mcs.anl.gov</a><br><b>Subject:</b><span class="Apple-converted-space"> </span>Re: [petsc-users] Ask for help in tracking down bugs - setting #threads for pardiso block LU solver<o:p></o:p></span></div></div></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><o:p> </o:p></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><o:p> </o:p></div></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> Boris,<o:p></o:p></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><o:p> </o:p></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> There is nothing obviously wrong with what you outline, so let's dive down into the code with the debugger and see what is happening.<o:p></o:p></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><o:p> </o:p></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">PetscSetMKL_PARDISOThreads() set from options is called when the command line option is provided, and this calls <o:p></o:p></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><o:p> </o:p></div></div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">PETSC_EXTERN void PetscSetMKL_PARDISOThreads(int threads)<o:p></o:p></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">{<o:p></o:p></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> mkl_domain_set_num_threads(threads, MKL_DOMAIN_PARDISO);<o:p></o:p></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">}<o:p></o:p></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><o:p> </o:p></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">so in the debugger can you check that mkl_domain_set_num_threads() gets called with your requested number of threads?<o:p></o:p></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><o:p> </o:p></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">BTW: Since you are hardwiring the use of Pardiso in the code with function calls you couldsimply call PetscSetMKL_PARDISOThreads() directly<o:p></o:p></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">after your line of code<o:p></o:p></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><o:p> </o:p></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 13pt; font-family: Calibri, sans-serif;">PCFactorSetMatSolverType(subPc, MATSOLVERMKL_PARDISO); set solver for LU</span><o:p></o:p></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><o:p> </o:p></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 13pt; font-family: Calibri, sans-serif;">instead of feeding it in through the options database.</span><o:p></o:p></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><o:p> </o:p></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 13pt; font-family: Calibri, sans-serif;">Let us know how this turns out</span><o:p></o:p></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><o:p> </o:p></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 13pt; font-family: Calibri, sans-serif;"> Barry</span><o:p></o:p></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><o:p> </o:p></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><o:p> </o:p></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><o:p> </o:p></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><o:p> </o:p></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><br><br><o:p></o:p></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;"><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">On Sep 27, 2024, at 7:24<span style="font-family: Arial, sans-serif;"> </span>AM, Voinov, Boris <<a href="mailto:boris.voinov@intel.com" style="color: rgb(5, 99, 193); text-decoration: underline;">boris.voinov@intel.com</a>> wrote:<o:p></o:p></div></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><o:p> </o:p></div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 13pt; font-family: Calibri, sans-serif;">Hello,</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 13pt; font-family: Calibri, sans-serif;"> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 13pt; font-family: Calibri, sans-serif;">Could you please help me to figure out what’s wrong with the way I’m trying to make mkl pardiso use the number of threads more than one while setting it as a sub PC solver for ASM preconditioner.</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 13pt; font-family: Calibri, sans-serif;">Here is how I do this</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 13pt; font-family: Calibri, sans-serif;">Loop over block PCs {</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 13pt; font-family: Calibri, sans-serif;"> PCSetType(subPc, PCLU); set subPC type</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 13pt; font-family: Calibri, sans-serif;"> PCFactorSetMatSolverType(subPc, MATSOLVERMKL_PARDISO); set solver for LU</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 13pt; font-family: Calibri, sans-serif;"> SetPardisoParams("pardiso"); set mkl related options including -mat_mkl_pardiso_65 equal to the desired #threads; return code after PetscOptionsSetValue is ok, so I presume the option is set correctly.<span class="apple-converted-space"> </span></span><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 13pt; font-family: Calibri, sans-serif;"> PCSetFromOptions(subPc);</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 13pt; font-family: Calibri, sans-serif;">}</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 13pt; font-family: Calibri, sans-serif;"> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 13pt; font-family: Calibri, sans-serif;">Then what I see under the debugger</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 13pt; font-family: Calibri, sans-serif;">PCSetUp after a series of calls gets to MatLUFactorSymbolic_AIJMKL_PARDISO which in turn calls MatSetFromOptions_MKL_PARDISO(F, A) and there</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 13pt; font-family: Calibri, sans-serif;">in these lines it seems to ignore what I’ve set for<span class="apple-converted-space"> </span></span><span lang="EN-US" style="font-size: 13pt; font-family: Calibri, sans-serif;">-mat_mkl_pardiso_65 and sets #threads to the default 1</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 13pt; font-family: Calibri, sans-serif;">PetscCall(PetscOptionsInt("-mat_mkl_pardiso_65", "Suggested number of threads to use within PARDISO", "None", threads, &threads, &flg));</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 13pt; font-family: Calibri, sans-serif;">if (flg) PetscSetMKL_PARDISOThreads((int)threads);</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 13pt; font-family: Calibri, sans-serif;"> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 13pt; font-family: Calibri, sans-serif;">This is about petsc-3.20.</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 13pt; font-family: Calibri, sans-serif;">PS when I do this in a standalone code which reads a matrix and solves the linear system and set<span class="apple-converted-space"> </span></span><span lang="EN-US" style="font-size: 13pt; font-family: Calibri, sans-serif;">-mat_mkl_pardiso_65 in the command line it’s all right but I need the number of threads to be set in other way in my code.</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 13pt; font-family: Calibri, sans-serif;"> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 13pt; font-family: Calibri, sans-serif;">Thank you and best regards,</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 13pt; font-family: Calibri, sans-serif;">Boris</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 13pt; font-family: Calibri, sans-serif;"> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><o:p></o:p></span></div></div><p style="caret-color: rgb(0, 0, 0); font-variant-caps: normal; text-align: start; -webkit-text-stroke-width: 0px; word-spacing: 0px;"><span style="font-size: 13.5pt; font-family: Helvetica, sans-serif;">-------------------------------------------------------------<br>Intel Ireland Limited (Branch)<br>Collinstown Industrial Park, Leixlip, County Kildare, Ireland<br>Registered Number: E902934<o:p></o:p></span></p><p style="caret-color: rgb(0, 0, 0); font-variant-caps: normal; text-align: start; -webkit-text-stroke-width: 0px; word-spacing: 0px;"><span style="font-size: 13.5pt; font-family: Helvetica, sans-serif;">This e-mail and any attachments may contain confidential material for<br>the sole use of the intended recipient(s). Any review or distribution<br>by others is strictly prohibited. If you are not the intended<br>recipient, please contact the sender and delete all copies.<o:p></o:p></span></p></div></blockquote></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><o:p> </o:p></div></div></div><p style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">-------------------------------------------------------------<br>Intel Ireland Limited (Branch)<br>Collinstown Industrial Park, Leixlip, County Kildare, Ireland<br>Registered Number: E902934</p><p style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">This e-mail and any attachments may contain confidential material for<br>the sole use of the intended recipient(s). Any review or distribution<br>by others is strictly prohibited. If you are not the intended<br>recipient, please contact the sender and delete all copies.</p></div></blockquote></div><br></div></body></html>