[petsc-users] PETSc MUMPS interface

[Varun Hiremath]

Hi Hong,

Thanks for looking into this. Here is the workflow that I might use:

MatLUFactorSymbolic(F,A,perm,iperm,&info);

// get memory estimates from MUMPS e.g. INFO(3), INFOG(16), INFOG(17)
// find available memory on the system e.g. RAM size
if (estimated_memory > available_memory)
{
   // inform and stop; or
   // switch MUMPS to out-of-core factorization
   ICNTL(22) = 1;
}
else
{
   // set appropriate settings for in-core factorization
}

// Now we call the solve and inside if MatLUFactorSymbolic is already
called then it should be skipped
EPSSolve(eps);

Thanks,
Varun

On Mon, Jan 17, 2022 at 9:18 AM Zhang, Hong <hzhang at mcs.anl.gov> wrote:

> Varun,
> I am trying to find a way to enable you to switch options after MatLUFactorSymbolic().
> A hack is modifying the flag  'mumps->matstruc'
> inside  MatLUFactorSymbolic_AIJMUMPS() and MatFactorNumeric_MUMPS().
>
> My understanding of what you want is:
>   // collect mumps memory info
>   ...
>   MatLUFactorSymbolic(F,A,perm,iperm,&info);
>   printMumpsMemoryInfo(F);
>   //---------
> if (memory is available) {
>     EPSSolve(eps); --> skip calling of MatLUFactorSymbolic()
> } else {
>    //out-of-core (OOC) option in MUMPS
> }
>
> Am I correct? I'll let you know once I work out a solution.
> Hong
>
> ------------------------------
> *From:* Varun Hiremath <varunhiremath at gmail.com>
> *Sent:* Sunday, January 16, 2022 10:10 PM
> *To:* Zhang, Hong <hzhang at mcs.anl.gov>
> *Cc:* Jose E. Roman <jroman at dsic.upv.es>; Peder Jørgensgaard Olesen via
> petsc-users <petsc-users at mcs.anl.gov>
> *Subject:* Re: [petsc-users] PETSc MUMPS interface
>
> Hi Jose, Hong,
>
> Thanks for the explanation. I have verified using -log_view that MatLUFactorSymbolic
> is indeed getting called twice.
>
> Hong, we use MUMPS solver for other things, and we typically run the
> symbolic analysis first and get memory estimates to ensure that we have
> enough memory available to run the case. If the available memory is not
> enough, we can stop or switch to the out-of-core (OOC) option in MUMPS. We
> wanted to do the same when using MUMPS via SLEPc/PETSc. Please let me know
> if there are other ways of getting these memory stats and switching options
> during runtime with PETSc.
> Appreciate your help!
>
> Thanks,
> Varun
>
> On Sun, Jan 16, 2022 at 4:01 PM Zhang, Hong <hzhang at mcs.anl.gov> wrote:
>
> Varun,
> I believe Jose is correct. You may verify it by running your code with
> option '-log_view', then check the number of calls to MatLUFactorSym.
>
> I guess I can add a flag in PCSetUp() to check if user has already called
> MatLUFactorSymbolic() and wants to skip it. Normally, users simply allocate
> sufficient memory in the symbolic factorization. Why do you want to check
> it?
> Hong
>
> ------------------------------
> *From:* Jose E. Roman <jroman at dsic.upv.es>
> *Sent:* Sunday, January 16, 2022 5:11 AM
> *To:* Varun Hiremath <varunhiremath at gmail.com>
> *Cc:* Zhang, Hong <hzhang at mcs.anl.gov>; Peder Jørgensgaard Olesen via
> petsc-users <petsc-users at mcs.anl.gov>
> *Subject:* Re: [petsc-users] PETSc MUMPS interface
>
> Hong may give a better answer, but if you look at PCSetUp_LU()
> https://petsc.org/main/src/ksp/pc/impls/factor/lu/lu.c.html#PCSetUp_LU
> you will see that MatLUFactorSymbolic() is called unconditionally during
> the first PCSetUp(). Currently there is no way to check if the user has
> already called MatLUFactorSymbolic().
>
> Jose
>
>
> > El 16 ene 2022, a las 10:40, Varun Hiremath <varunhiremath at gmail.com>
> escribió:
> >
> > Hi Hong,
> >
> > Thank you, this is very helpful!
> >
> > Using this method I am able to get the memory estimates before the
> actual solve, however, I think my code may be causing the symbolic
> factorization to be run twice. Attached is my code where I am using SLEPc
> to compute eigenvalues, and I use MUMPS for factorization. I have commented
> above the code that computes the memory estimates, could you please check
> and tell me if this would cause the symbolic factor to be computed twice (a
> second time inside EPSSolve?), as I am seeing a slight increase in the
> overall computation time?
> >
> > Regards,
> > Varun
> >
> > On Wed, Jan 12, 2022 at 7:58 AM Zhang, Hong <hzhang at mcs.anl.gov> wrote:
> > PCFactorSetMatSolverType(pc,MATSOLVERMUMPS);
> >   PCFactorSetUpMatSolverType(pc);
> >   PCFactorGetMatrix(pc,&F);
> >
> >   MatLUFactorSymbolic(F,A,...)
> >   You must provide row and column permutations etc,
> petsc/src/mat/tests/ex125.c may give you a clue on how to get these inputs.
> >
> > Hong
> >
> >
> > From: petsc-users <petsc-users-bounces at mcs.anl.gov> on behalf of
> Junchao Zhang <junchao.zhang at gmail.com>
> > Sent: Wednesday, January 12, 2022 9:03 AM
> > To: Varun Hiremath <varunhiremath at gmail.com>
> > Cc: Peder Jørgensgaard Olesen via petsc-users <petsc-users at mcs.anl.gov>
> > Subject: Re: [petsc-users] PETSc MUMPS interface
> >
> > Calling PCSetUp() before KSPSetUp()?
> >
> > --Junchao Zhang
> >
> >
> > On Wed, Jan 12, 2022 at 3:00 AM Varun Hiremath <varunhiremath at gmail.com>
> wrote:
> > Hi All,
> >
> > I want to collect MUMPS memory estimates based on the initial symbolic
> factorization analysis before the actual numerical factorization starts to
> check if the estimated memory requirements fit the available memory.
> >
> > I am following the steps from
> https://petsc.org/main/src/ksp/ksp/tutorials/ex52.c.html
> >
> >   PCFactorSetMatSolverType(pc,MATSOLVERMUMPS);
> >   PCFactorSetUpMatSolverType(pc);
> >   PCFactorGetMatrix(pc,&F);
> >
> >   KSPSetUp(ksp);
> >   MatMumpsGetInfog(F,...)
> >
> > But it appears KSPSetUp calls both symbolic and numerical factorization.
> So is there some other way to get these statistics before the actual
> factorization starts?
> >
> > Thanks,
> > Varun
> > <slepc_eps_mumps_test.cpp>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220117/2ab437c8/attachment.html>