<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div><br></div> Boris,<div><br></div><div> There is nothing obviously wrong with what you outline, so let's dive down into the code with the debugger and see what is happening.</div><div><br></div><div>PetscSetMKL_PARDISOThreads() set from options is called when the command line option is provided, and this calls </div><div><br></div><div><div>PETSC_EXTERN void PetscSetMKL_PARDISOThreads(int threads)</div><div>{</div><div> mkl_domain_set_num_threads(threads, MKL_DOMAIN_PARDISO);</div><div>}</div><div><br></div><div>so in the debugger can you check that mkl_domain_set_num_threads() gets called with your requested number of threads?</div><div><br></div><div>BTW: Since you are hardwiring the use of Pardiso in the code with function calls you couldsimply call PetscSetMKL_PARDISOThreads() directly</div><div>after your line of code</div><div><br></div><div><span style="font-family: Calibri, sans-serif; font-size: 17.333334px;">PCFactorSetMatSolverType(subPc, MATSOLVERMKL_PARDISO); set solver for LU</span></div><div><span style="font-family: Calibri, sans-serif; font-size: 17.333334px;"><br></span></div><div><span style="font-family: Calibri, sans-serif; font-size: 17.333334px;">instead of feeding it in through the options database.</span></div><div><span style="font-family: Calibri, sans-serif; font-size: 17.333334px;"><br></span></div><div><span style="font-family: Calibri, sans-serif; font-size: 17.333334px;">Let us know how this turns out</span></div><div><span style="font-family: Calibri, sans-serif; font-size: 17.333334px;"><br></span></div><div><span style="font-family: Calibri, sans-serif; font-size: 17.333334px;"> Barry</span></div><div><span style="font-family: Calibri, sans-serif; font-size: 17.333334px;"><br></span></div><div><span style="font-family: Calibri, sans-serif; font-size: 17.333334px;"><br></span></div><div><span style="font-family: Calibri, sans-serif; font-size: 17.333334px;"><br></span></div><div><span style="font-family: Calibri, sans-serif; font-size: 17.333334px;"><br></span></div><div><br><blockquote type="cite"><div>On Sep 27, 2024, at 7:24 AM, Voinov, Boris <boris.voinov@intel.com> wrote:</div><br class="Apple-interchange-newline"><div><meta charset="UTF-8"><div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" style="font-size: 13pt;">Hello,<o:p></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" style="font-size: 13pt;"><o:p> </o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" style="font-size: 13pt;">Could you please help me to figure out what’s wrong with the way I’m trying to make mkl pardiso use the number of threads more than one while setting it as a sub PC solver for ASM preconditioner.<o:p></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" style="font-size: 13pt;">Here is how I do this<o:p></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" style="font-size: 13pt;">Loop over block PCs {<o:p></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" style="font-size: 13pt;"> PCSetType(subPc, PCLU); set subPC type<o:p></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" style="font-size: 13pt;"> PCFactorSetMatSolverType(subPc, MATSOLVERMKL_PARDISO); set solver for LU<o:p></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" style="font-size: 13pt;"> SetPardisoParams("pardiso"); set mkl related options including -mat_mkl_pardiso_65 equal to the desired #threads; return code after PetscOptionsSetValue is ok, so I presume the option is set correctly.<span class="Apple-converted-space"> </span><o:p></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" style="font-size: 13pt;"> PCSetFromOptions(subPc);<o:p></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" style="font-size: 13pt;">}<o:p></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" style="font-size: 13pt;"><o:p> </o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" style="font-size: 13pt;">Then what I see under the debugger<o:p></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" style="font-size: 13pt;">PCSetUp after a series of calls gets to MatLUFactorSymbolic_AIJMKL_PARDISO which in turn calls MatSetFromOptions_MKL_PARDISO(F, A) and there<o:p></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 13pt;">in these lines it seems to ignore what I’ve set for<span class="Apple-converted-space"> </span></span><span lang="EN-US" style="font-size: 13pt;">-mat_mkl_pardiso_65 and sets #threads to the default 1</span><span style="font-size: 13pt;"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 13pt;">PetscCall(PetscOptionsInt("-mat_mkl_pardiso_65", "Suggested number of threads to use within PARDISO", "None", threads, &threads, &flg));<o:p></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 13pt;">if (flg) PetscSetMKL_PARDISOThreads((int)threads);<o:p></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 13pt;"><o:p> </o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 13pt;">This is about petsc-3.20.<o:p></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 13pt;">PS when I do this in a standalone code which reads a matrix and solves the linear system and set<span class="Apple-converted-space"> </span></span><span lang="EN-US" style="font-size: 13pt;">-mat_mkl_pardiso_65 in the command line it’s all right but I need the number of threads to be set in other way in my code.</span><span style="font-size: 13pt;"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 13pt;"><o:p> </o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 13pt;">Thank you and best regards,<o:p></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 13pt;">Boris<o:p></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 13pt;"><o:p> </o:p></span></div></div><p style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">-------------------------------------------------------------<br>Intel Ireland Limited (Branch)<br>Collinstown Industrial Park, Leixlip, County Kildare, Ireland<br>Registered Number: E902934</p><p style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">This e-mail and any attachments may contain confidential material for<br>the sole use of the intended recipient(s). Any review or distribution<br>by others is strictly prohibited. If you are not the intended<br>recipient, please contact the sender and delete all copies.</p></div></blockquote></div><br></div></body></html>