[petsc-users] Ask for help in tracking down bugs - setting #threads for pardiso block LU solver

Voinov, Boris boris.voinov at intel.com
Mon Sep 30 10:57:24 CDT 2024


Hi Barry,

Sorry for messing it up. Please disregard my message.
It actually was all true but down in PetscOptionsInt_Private
PetscOptionsGetInt(options, prefix, opt, value, &wasset) returns *value = 8 and wasset = true, i.e. it’s all right.

Regards,
Boris

From: Voinov, Boris
Sent: Monday, September 30, 2024 4:36 PM
To: Barry Smith <bsmith at petsc.dev>
Cc: petsc-users <petsc-users at mcs.anl.gov>
Subject: RE: [petsc-users] Ask for help in tracking down bugs - setting #threads for pardiso block LU solver

Hi Barry,

Here are a bit more details.
Under the debugger, when I break just before
PetscCall(PetscOptionsInt("-mat_mkl_pardiso_65", "Suggested number of threads to use within PARDISO", "None", threads, &threads, &flg));
what I see for option “-mat_mkl_pardiso_65” in the defaultoptions is
         Expression:  *(defaultoptions->names+16)
            Address:  0x17d361e0
               Type:  $string *
      Window Status:
              Title:  *(defaultoptions->names+16) - floops.linux64.1 - 1.1
           Language:  C++
     Valid in Scope:  (Global)
  Compiled in Scope:      PetscOptionsBegin_Private             (Subroutine)
  aoptions.c            (File)
libpetsc.so.3.20                (Image)
          UPC Phase:
           UPC Addr:
      Examine Count:  0
       Column Count:  0
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Value
0x17d35b00 -> "mat_mkl_pardiso_65"
And
         Expression:  *(defaultoptions->values+16)
            Address:  0x17d36330
               Type:  $string *
      Window Status:
              Title:  *(defaultoptions->values+16) - floops.linux64.1 - 1.1
           Language:  C++
     Valid in Scope:  (Global)
  Compiled in Scope:      PetscOptionsBegin_Private             (Subroutine)
  aoptions.c            (File)
libpetsc.so.3.20                (Image)
          UPC Phase:
           UPC Addr:
      Examine Count:  0
       Column Count:  0
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Value
0x17d35b20 -> "8"
I.e. exactly what I set in the loop for subPC and all other options as well. So it looks ok.
Now when I jump into (PetscOptionsInt) I turn out here:
PetscErrorCode PetscOptionsInt_Private(PetscOptionItems *PetscOptionsObject, const char opt[], const char text[], const char man[], PetscInt currentvalue, PetscInt *value, PetscBool *set, PetscInt lb, PetscInt ub)
{
   const char        *prefix  = PetscOptionsObject->prefix;
   const PetscOptions options = PetscOptionsObject->options;
   PetscBool          wasset;

PetscOptionsObject is like this:
        Expression:  *PetscOptionsObject
            Address:  0x7fffffff6d80
               Type:  PetscOptionItems
      Window Status:
              Title:  *PetscOptionsObject - PetscOptionsInt_Private - 1.1
           Language:  C++
     Valid in Scope:      PetscOptionsInt_Private               (Subroutine)
  aoptions.c            (File)
libpetsc.so.3.20                (Image)
  Compiled in Scope:      PetscOptionsInt_Private               (Subroutine)
  aoptions.c            (File)
libpetsc.so.3.20                (Image)
          UPC Phase:
           UPC Addr:
      Examine Count:  0
       Column Count:  0
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Field          Type                         Value
count          PetscInt                     0x00000001 (1)
next           PetscOptionItem              0x00000000
prefix         $string *                    0x00000000
pprefix        $string *                    0x00000000
title          $string *                    0x17d39140 -> "MKL_PARDISO Options"
comm           _p_PetscObject::MPI_Comm     0x84000002 (-2080374782)
printhelp      PetscBool                    PETSC_FALSE (0)
changedmethod  PetscBool                    PETSC_FALSE (0)
alreadyprinted PetscBool                    PETSC_FALSE (0)
object         PetscObject                  0x00000000
options        _p_PetscObject::PetscOptions 0x00000000           - options are empty
And then, I guess, it doesn’t see my settings  and overwrites the number of threads with the default 1.
It seems like the settings I made were not associated with PetscOptionsObject named “MKL_PARDISO Options”

Best regards,
Boris
From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Friday, September 27, 2024 6:27 PM
To: Voinov, Boris <boris.voinov at intel.com<mailto:boris.voinov at intel.com>>
Cc: petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Ask for help in tracking down bugs - setting #threads for pardiso block LU solver


   I should have read your initial email more carefully:

----
Loop over block PCs {
  PCSetType(subPc, PCLU);             set subPC type
  PCFactorSetMatSolverType(subPc, MATSOLVERMKL_PARDISO);       set solver for LU
  SetPardisoParams("pardiso");      set mkl related options including -mat_mkl_pardiso_65 equal to the desired #threads; return code after PetscOptionsSetValue is ok, so I presume the option is set correctly.
  PCSetFromOptions(subPc);
}

Then what I see under the debugger
PCSetUp after a series of calls gets to MatLUFactorSymbolic_AIJMKL_PARDISO which in turn calls MatSetFromOptions_MKL_PARDISO(F, A) and there
in these lines it seems to ignore what I’ve set for -mat_mkl_pardiso_65 and sets #threads to the default 1
PetscCall(PetscOptionsInt("-mat_mkl_pardiso_65", "Suggested number of threads to use within PARDISO", "None", threads, &threads, &flg));
if (flg) PetscSetMKL_PARDISOThreads((int)threads);

-----

PCSetFromOptions() actually does not call MatSetFromOptions() on the factored matrix. There is no call to MatSetFromOptions() from PCSetFromOptions() or PetscSetFromOptions_Factor() because the factored matrix F does not yet exist (it is created in PCSetUp()).  This is why your call to PCSetFromOptions(subPc); does not process any of your mat mkl options.

Since MatSetFromOptions() cannot be called from PCSetFromOptions(), MatSetFromOptions_MKL_PARDISO(F, A) is called in  MatFactorSymbolic_AIJMKL_PARDISO_Private() as you noted when the PC is being set up.

You say "here in these lines it seems to ignore what I’ve set for -mat_mkl_pardiso_65 and sets #threads to the default 1"

This is the part a do not understand, if your values have been inserted in the database earlier in the code and remain in the database they should get processed inside the
MatSetFromOptions_MKL_PARDISO(F, A) called by  MatFactorSymbolic_AIJMKL_PARDISO_Private().

Can you please put a break point at MatSetFromOptions_MKL_PARDISO(F, A) and then when it breaks put

call PetscOptionsView(0,0)

it should print out the contents of the options database so you can verify if the options you added are there. You can then step through the code in MatSetFromOptions_MKL_PARDISO to see what is happening as it tries to acess the database to see why it skips

PetscCall(PetscOptionsInt("-mat_mkl_pardiso_65", "Suggested number of threads to use within MKL PARDISO", "None", threads, &threads, &flg));
  if (flg) PetscSetMKL_PARDISOThreads((int)threads);











On Sep 27, 2024, at 11:58 AM, Voinov, Boris <boris.voinov at intel.com<mailto:boris.voinov at intel.com>> wrote:

Thanks a lot Barry,

It looks like something wrong with my own settings though. It doesn’t make an effect. I mean this call
SetPardisoParams("pardiso");
followed by
PCSetFromOptions(subPc);
doesn’t end up with a call of PetscSetMKL_PARDISOThreads()
This function is called later when PCSetUp is invoked.

Eventually what I see in the end is
Option left: name:-mat_mkl_pardiso_65 value: 8 source: code listed among unused database options
That is what I’ve set with my own settings, meaning it’s in the database, I guess.

Could it be that it hits a “not proper” section in the options database and is not being used by PCSetUp?

Best regards,
Boris

From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Friday, September 27, 2024 4:21 PM
To: Voinov, Boris <boris.voinov at intel.com<mailto:boris.voinov at intel.com>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Ask for help in tracking down bugs - setting #threads for pardiso block LU solver


   Boris,

     There is nothing obviously wrong with what you outline, so let's dive down into the code with the debugger and see what is happening.

PetscSetMKL_PARDISOThreads() set from options is called when the command line option is provided, and this calls

PETSC_EXTERN void PetscSetMKL_PARDISOThreads(int threads)
{
  mkl_domain_set_num_threads(threads, MKL_DOMAIN_PARDISO);
}

so in the debugger can you check that mkl_domain_set_num_threads() gets called with your requested number of threads?

BTW: Since you are hardwiring the use of Pardiso in the code with function calls you couldsimply call PetscSetMKL_PARDISOThreads() directly
after your line of code

PCFactorSetMatSolverType(subPc, MATSOLVERMKL_PARDISO);       set solver for LU

instead of feeding it in through the options database.

Let us know how this turns out

  Barry






On Sep 27, 2024, at 7:24 AM, Voinov, Boris <boris.voinov at intel.com<mailto:boris.voinov at intel.com>> wrote:

Hello,

Could you please help me to figure out what’s wrong with the way I’m trying to make mkl pardiso use the number of threads more than one while setting it as a sub PC solver for ASM preconditioner.
Here is how I do this
Loop over block PCs {
  PCSetType(subPc, PCLU);             set subPC type
  PCFactorSetMatSolverType(subPc, MATSOLVERMKL_PARDISO);       set solver for LU
  SetPardisoParams("pardiso");      set mkl related options including -mat_mkl_pardiso_65 equal to the desired #threads; return code after PetscOptionsSetValue is ok, so I presume the option is set correctly.
  PCSetFromOptions(subPc);
}

Then what I see under the debugger
PCSetUp after a series of calls gets to MatLUFactorSymbolic_AIJMKL_PARDISO which in turn calls MatSetFromOptions_MKL_PARDISO(F, A) and there
in these lines it seems to ignore what I’ve set for -mat_mkl_pardiso_65 and sets #threads to the default 1
PetscCall(PetscOptionsInt("-mat_mkl_pardiso_65", "Suggested number of threads to use within PARDISO", "None", threads, &threads, &flg));
if (flg) PetscSetMKL_PARDISOThreads((int)threads);

This is about petsc-3.20.
PS when I do this in a standalone code which reads a matrix and solves the linear system and set -mat_mkl_pardiso_65 in the command line it’s all right but I need the number of threads to be set in other way in my code.

Thank you and best regards,
Boris


-------------------------------------------------------------
Intel Ireland Limited (Branch)
Collinstown Industrial Park, Leixlip, County Kildare, Ireland
Registered Number: E902934

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.


-------------------------------------------------------------
Intel Ireland Limited (Branch)
Collinstown Industrial Park, Leixlip, County Kildare, Ireland
Registered Number: E902934

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

-------------------------------------------------------------
Intel Ireland Limited (Branch)
Collinstown Industrial Park, Leixlip, County Kildare, Ireland
Registered Number: E902934

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20240930/1b435479/attachment-0001.html>


More information about the petsc-users mailing list