[petsc-users] PETSc MUMPS interface

Zhang, Hong hzhang at mcs.anl.gov
Wed Jan 19 17:33:52 CST 2022 

Varun,
This feature is merged to petsc main
https://gitlab.com/petsc/petsc/-/merge_requests/4727
Hong
________________________________
From: petsc-users <petsc-users-bounces at mcs.anl.gov> on behalf of Zhang, Hong via petsc-users <petsc-users at mcs.anl.gov>
Sent: Wednesday, January 19, 2022 9:37 AM
To: Varun Hiremath <varunhiremath at gmail.com>
Cc: Peder Jørgensgaard Olesen via petsc-users <petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] PETSc MUMPS interface

Varun,
Good to know it works. FactorSymbolic function is still being called twice, but the 2nd call is a no-op, thus it still appears in '-log_view'. I made changes in the low level of mumps routine, not within PCSetUp() because I feel your use case is limited to mumps, not other matrix package solvers.
Hong
________________________________
From: Varun Hiremath <varunhiremath at gmail.com>
Sent: Wednesday, January 19, 2022 2:44 AM
To: Zhang, Hong <hzhang at mcs.anl.gov>
Cc: Peder Jørgensgaard Olesen via petsc-users <petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] PETSc MUMPS interface

Hi Hong,

Thanks, I tested your branch and I think it is working fine. I don't see any increase in runtime, however with -log_view I see that the MatLUFactorSymbolic function is still being called twice, so is this expected? Is the second call a no-op?

$ ./ex52.o -use_mumps_lu -print_mumps_memory -log_view | grep MatLUFactorSym
MatLUFactorSym         2 1.0 4.4411e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  2  0  0  0  0   2  0  0  0  0     0

Thanks,
Varun

On Mon, Jan 17, 2022 at 7:49 PM Zhang, Hong <hzhang at mcs.anl.gov<mailto:hzhang at mcs.anl.gov>> wrote:
Varun,
I created a branch hzhang/feature-mumps-mem-estimate,
see https://gitlab.com/petsc/petsc/-/merge_requests/4727

You may give it a try and let me know if this is what you want.
src/ksp/ksp/tutorials/ex52.c is an example.

Hong
________________________________
From: Varun Hiremath <varunhiremath at gmail.com<mailto:varunhiremath at gmail.com>>
Sent: Monday, January 17, 2022 12:41 PM
To: Zhang, Hong <hzhang at mcs.anl.gov<mailto:hzhang at mcs.anl.gov>>
Cc: Jose E. Roman <jroman at dsic.upv.es<mailto:jroman at dsic.upv.es>>; Peder Jørgensgaard Olesen via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] PETSc MUMPS interface

Hi Hong,

Thanks for looking into this. Here is the workflow that I might use:

MatLUFactorSymbolic(F,A,perm,iperm,&info);

// get memory estimates from MUMPS e.g. INFO(3), INFOG(16), INFOG(17)
// find available memory on the system e.g. RAM size
if (estimated_memory > available_memory)
{
   // inform and stop; or
   // switch MUMPS to out-of-core factorization
   ICNTL(22) = 1;
}
else
{
   // set appropriate settings for in-core factorization
}

// Now we call the solve and inside if MatLUFactorSymbolic is already called then it should be skipped
EPSSolve(eps);

Thanks,
Varun

On Mon, Jan 17, 2022 at 9:18 AM Zhang, Hong <hzhang at mcs.anl.gov<mailto:hzhang at mcs.anl.gov>> wrote:
Varun,
I am trying to find a way to enable you to switch options after MatLUFactorSymbolic(). A hack is modifying the flag  'mumps->matstruc' inside  MatLUFactorSymbolic_AIJMUMPS() and MatFactorNumeric_MUMPS().

My understanding of what you want is:
  // collect mumps memory info
  ...
  MatLUFactorSymbolic(F,A,perm,iperm,&info);
  printMumpsMemoryInfo(F);
  //---------
if (memory is available) {
    EPSSolve(eps); --> skip calling of MatLUFactorSymbolic()
} else {
   //out-of-core (OOC) option in MUMPS
}

Am I correct? I'll let you know once I work out a solution.
Hong

________________________________
From: Varun Hiremath <varunhiremath at gmail.com<mailto:varunhiremath at gmail.com>>
Sent: Sunday, January 16, 2022 10:10 PM
To: Zhang, Hong <hzhang at mcs.anl.gov<mailto:hzhang at mcs.anl.gov>>
Cc: Jose E. Roman <jroman at dsic.upv.es<mailto:jroman at dsic.upv.es>>; Peder Jørgensgaard Olesen via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] PETSc MUMPS interface

Hi Jose, Hong,

Thanks for the explanation. I have verified using -log_view that MatLUFactorSymbolic is indeed getting called twice.

Hong, we use MUMPS solver for other things, and we typically run the symbolic analysis first and get memory estimates to ensure that we have enough memory available to run the case. If the available memory is not enough, we can stop or switch to the out-of-core (OOC) option in MUMPS. We wanted to do the same when using MUMPS via SLEPc/PETSc. Please let me know if there are other ways of getting these memory stats and switching options during runtime with PETSc.
Appreciate your help!

Thanks,
Varun

On Sun, Jan 16, 2022 at 4:01 PM Zhang, Hong <hzhang at mcs.anl.gov<mailto:hzhang at mcs.anl.gov>> wrote:
Varun,
I believe Jose is correct. You may verify it by running your code with option '-log_view', then check the number of calls to MatLUFactorSym.

I guess I can add a flag in PCSetUp() to check if user has already called MatLUFactorSymbolic() and wants to skip it. Normally, users simply allocate sufficient memory in the symbolic factorization. Why do you want to check it?
Hong

________________________________
From: Jose E. Roman <jroman at dsic.upv.es<mailto:jroman at dsic.upv.es>>
Sent: Sunday, January 16, 2022 5:11 AM
To: Varun Hiremath <varunhiremath at gmail.com<mailto:varunhiremath at gmail.com>>
Cc: Zhang, Hong <hzhang at mcs.anl.gov<mailto:hzhang at mcs.anl.gov>>; Peder Jørgensgaard Olesen via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] PETSc MUMPS interface

Hong may give a better answer, but if you look at PCSetUp_LU() https://petsc.org/main/src/ksp/pc/impls/factor/lu/lu.c.html#PCSetUp_LU you will see that MatLUFactorSymbolic() is called unconditionally during the first PCSetUp(). Currently there is no way to check if the user has already called MatLUFactorSymbolic().

Jose


> El 16 ene 2022, a las 10:40, Varun Hiremath <varunhiremath at gmail.com<mailto:varunhiremath at gmail.com>> escribió:
>
> Hi Hong,
>
> Thank you, this is very helpful!
>
> Using this method I am able to get the memory estimates before the actual solve, however, I think my code may be causing the symbolic factorization to be run twice. Attached is my code where I am using SLEPc to compute eigenvalues, and I use MUMPS for factorization. I have commented above the code that computes the memory estimates, could you please check and tell me if this would cause the symbolic factor to be computed twice (a second time inside EPSSolve?), as I am seeing a slight increase in the overall computation time?
>
> Regards,
> Varun
>
> On Wed, Jan 12, 2022 at 7:58 AM Zhang, Hong <hzhang at mcs.anl.gov<mailto:hzhang at mcs.anl.gov>> wrote:
> PCFactorSetMatSolverType(pc,MATSOLVERMUMPS);
>   PCFactorSetUpMatSolverType(pc);
>   PCFactorGetMatrix(pc,&F);
>
>   MatLUFactorSymbolic(F,A,...)
>   You must provide row and column permutations etc,    petsc/src/mat/tests/ex125.c may give you a clue on how to get these inputs.
>
> Hong
>
>
> From: petsc-users <petsc-users-bounces at mcs.anl.gov<mailto:petsc-users-bounces at mcs.anl.gov>> on behalf of Junchao Zhang <junchao.zhang at gmail.com<mailto:junchao.zhang at gmail.com>>
> Sent: Wednesday, January 12, 2022 9:03 AM
> To: Varun Hiremath <varunhiremath at gmail.com<mailto:varunhiremath at gmail.com>>
> Cc: Peder Jørgensgaard Olesen via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] PETSc MUMPS interface
>
> Calling PCSetUp() before KSPSetUp()?
>
> --Junchao Zhang
>
>
> On Wed, Jan 12, 2022 at 3:00 AM Varun Hiremath <varunhiremath at gmail.com<mailto:varunhiremath at gmail.com>> wrote:
> Hi All,
>
> I want to collect MUMPS memory estimates based on the initial symbolic factorization analysis before the actual numerical factorization starts to check if the estimated memory requirements fit the available memory.
>
> I am following the steps from https://petsc.org/main/src/ksp/ksp/tutorials/ex52.c.html
>
>   PCFactorSetMatSolverType(pc,MATSOLVERMUMPS);
>   PCFactorSetUpMatSolverType(pc);
>   PCFactorGetMatrix(pc,&F);
>
>   KSPSetUp(ksp);
>   MatMumpsGetInfog(F,...)
>
> But it appears KSPSetUp calls both symbolic and numerical factorization. So is there some other way to get these statistics before the actual factorization starts?
>
> Thanks,
> Varun
> <slepc_eps_mumps_test.cpp>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220119/4a8cb820/attachment-0001.html>

More information about the petsc-users mailing list