[petsc-users] PETSc MUMPS interface

Sat Jan 22 02:11:22 CST 2022

Hi Hong,

I tested this in the latest petsc main branch and it appears to be working
fine. Thanks for implementing this so quickly!

Regards,
Varun

On Wed, Jan 19, 2022 at 3:33 PM Zhang, Hong <hzhang at mcs.anl.gov> wrote:

> Varun,
> This feature is merged to petsc main
> https://gitlab.com/petsc/petsc/-/merge_requests/4727
> Hong
> ------------------------------
> *From:* petsc-users <petsc-users-bounces at mcs.anl.gov> on behalf of Zhang,
> Hong via petsc-users <petsc-users at mcs.anl.gov>
> *Sent:* Wednesday, January 19, 2022 9:37 AM
> *To:* Varun Hiremath <varunhiremath at gmail.com>
> *Cc:* Peder Jørgensgaard Olesen via petsc-users <petsc-users at mcs.anl.gov>
> *Subject:* Re: [petsc-users] PETSc MUMPS interface
>
> Varun,
> Good to know it works. FactorSymbolic function is still being called
> twice, but the 2nd call is a no-op, thus it still appears in '-log_view'. I
> made changes in the low level of mumps routine, not within PCSetUp()
> because I feel your use case is limited to mumps, not other matrix package
> solvers.
> Hong
> ------------------------------
> *From:* Varun Hiremath <varunhiremath at gmail.com>
> *Sent:* Wednesday, January 19, 2022 2:44 AM
> *To:* Zhang, Hong <hzhang at mcs.anl.gov>
> *Cc:* Peder Jørgensgaard Olesen via petsc-users <petsc-users at mcs.anl.gov>
> *Subject:* Re: [petsc-users] PETSc MUMPS interface
>
> Hi Hong,
>
> Thanks, I tested your branch and I think it is working fine. I don't see
> any increase in runtime, however with -log_view I see that the
> MatLUFactorSymbolic function is still being called twice, so is this
> expected? Is the second call a no-op?
>
> $ ./ex52.o -use_mumps_lu -print_mumps_memory -log_view | grep
> MatLUFactorSym
> MatLUFactorSym         2 1.0 4.4411e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  2  0  0  0  0   2  0  0  0  0     0
>
> Thanks,
> Varun
>
> On Mon, Jan 17, 2022 at 7:49 PM Zhang, Hong <hzhang at mcs.anl.gov> wrote:
>
> Varun,
> I created a branch hzhang/feature-mumps-mem-estimate,
> see https://gitlab.com/petsc/petsc/-/merge_requests/4727
>
> You may give it a try and let me know if this is what you want.
> src/ksp/ksp/tutorials/ex52.c is an example.
>
> Hong
> ------------------------------
> *From:* Varun Hiremath <varunhiremath at gmail.com>
> *Sent:* Monday, January 17, 2022 12:41 PM
> *To:* Zhang, Hong <hzhang at mcs.anl.gov>
> *Cc:* Jose E. Roman <jroman at dsic.upv.es>; Peder Jørgensgaard Olesen via
> petsc-users <petsc-users at mcs.anl.gov>
> *Subject:* Re: [petsc-users] PETSc MUMPS interface
>
> Hi Hong,
>
> Thanks for looking into this. Here is the workflow that I might use:
>
> MatLUFactorSymbolic(F,A,perm,iperm,&info);
>
> // get memory estimates from MUMPS e.g. INFO(3), INFOG(16), INFOG(17)
> // find available memory on the system e.g. RAM size
> if (estimated_memory > available_memory)
> {
>    // inform and stop; or
>    // switch MUMPS to out-of-core factorization
>    ICNTL(22) = 1;
> }
> else
> {
>    // set appropriate settings for in-core factorization
> }
>
> // Now we call the solve and inside if MatLUFactorSymbolic is already
> called then it should be skipped
> EPSSolve(eps);
>
> Thanks,
> Varun
>
> On Mon, Jan 17, 2022 at 9:18 AM Zhang, Hong <hzhang at mcs.anl.gov> wrote:
>
> Varun,
> I am trying to find a way to enable you to switch options after MatLUFactorSymbolic().
> A hack is modifying the flag  'mumps->matstruc'
> inside  MatLUFactorSymbolic_AIJMUMPS() and MatFactorNumeric_MUMPS().
>
> My understanding of what you want is:
>   // collect mumps memory info
>   ...
>   MatLUFactorSymbolic(F,A,perm,iperm,&info);
>   printMumpsMemoryInfo(F);
>   //---------
> if (memory is available) {
>     EPSSolve(eps); --> skip calling of MatLUFactorSymbolic()
> } else {
>    //out-of-core (OOC) option in MUMPS
> }
>
> Am I correct? I'll let you know once I work out a solution.
> Hong
>
> ------------------------------
> *From:* Varun Hiremath <varunhiremath at gmail.com>
> *Sent:* Sunday, January 16, 2022 10:10 PM
> *To:* Zhang, Hong <hzhang at mcs.anl.gov>
> *Cc:* Jose E. Roman <jroman at dsic.upv.es>; Peder Jørgensgaard Olesen via
> petsc-users <petsc-users at mcs.anl.gov>
> *Subject:* Re: [petsc-users] PETSc MUMPS interface
>
> Hi Jose, Hong,
>
> Thanks for the explanation. I have verified using -log_view that MatLUFactorSymbolic
> is indeed getting called twice.
>
> Hong, we use MUMPS solver for other things, and we typically run the
> symbolic analysis first and get memory estimates to ensure that we have
> enough memory available to run the case. If the available memory is not
> enough, we can stop or switch to the out-of-core (OOC) option in MUMPS. We
> wanted to do the same when using MUMPS via SLEPc/PETSc. Please let me know
> if there are other ways of getting these memory stats and switching options
> during runtime with PETSc.
> Appreciate your help!
>
> Thanks,
> Varun
>
> On Sun, Jan 16, 2022 at 4:01 PM Zhang, Hong <hzhang at mcs.anl.gov> wrote:
>
> Varun,
> I believe Jose is correct. You may verify it by running your code with
> option '-log_view', then check the number of calls to MatLUFactorSym.
>
> I guess I can add a flag in PCSetUp() to check if user has already called
> MatLUFactorSymbolic() and wants to skip it. Normally, users simply allocate
> sufficient memory in the symbolic factorization. Why do you want to check
> it?
> Hong
>
> ------------------------------
> *From:* Jose E. Roman <jroman at dsic.upv.es>
> *Sent:* Sunday, January 16, 2022 5:11 AM
> *To:* Varun Hiremath <varunhiremath at gmail.com>
> *Cc:* Zhang, Hong <hzhang at mcs.anl.gov>; Peder Jørgensgaard Olesen via
> petsc-users <petsc-users at mcs.anl.gov>
> *Subject:* Re: [petsc-users] PETSc MUMPS interface
>
> Hong may give a better answer, but if you look at PCSetUp_LU()
> https://petsc.org/main/src/ksp/pc/impls/factor/lu/lu.c.html#PCSetUp_LU
> you will see that MatLUFactorSymbolic() is called unconditionally during
> the first PCSetUp(). Currently there is no way to check if the user has
> already called MatLUFactorSymbolic().
>
> Jose
>
>
> > El 16 ene 2022, a las 10:40, Varun Hiremath <varunhiremath at gmail.com>
> escribió:
> >
> > Hi Hong,
> >
> > Thank you, this is very helpful!
> >
> > Using this method I am able to get the memory estimates before the
> actual solve, however, I think my code may be causing the symbolic
> factorization to be run twice. Attached is my code where I am using SLEPc
> to compute eigenvalues, and I use MUMPS for factorization. I have commented
> above the code that computes the memory estimates, could you please check
> and tell me if this would cause the symbolic factor to be computed twice (a
> second time inside EPSSolve?), as I am seeing a slight increase in the
> overall computation time?
> >
> > Regards,
> > Varun
> >
> > On Wed, Jan 12, 2022 at 7:58 AM Zhang, Hong <hzhang at mcs.anl.gov> wrote:
> > PCFactorSetMatSolverType(pc,MATSOLVERMUMPS);
> >   PCFactorSetUpMatSolverType(pc);
> >   PCFactorGetMatrix(pc,&F);
> >
> >   MatLUFactorSymbolic(F,A,...)
> >   You must provide row and column permutations etc,
> petsc/src/mat/tests/ex125.c may give you a clue on how to get these inputs.
> >
> > Hong
> >
> >
> > From: petsc-users <petsc-users-bounces at mcs.anl.gov> on behalf of
> Junchao Zhang <junchao.zhang at gmail.com>
> > Sent: Wednesday, January 12, 2022 9:03 AM
> > To: Varun Hiremath <varunhiremath at gmail.com>
> > Cc: Peder Jørgensgaard Olesen via petsc-users <petsc-users at mcs.anl.gov>
> > Subject: Re: [petsc-users] PETSc MUMPS interface
> >
> > Calling PCSetUp() before KSPSetUp()?
> >
> > --Junchao Zhang
> >
> >
> > On Wed, Jan 12, 2022 at 3:00 AM Varun Hiremath <varunhiremath at gmail.com>
> wrote:
> > Hi All,
> >
> > I want to collect MUMPS memory estimates based on the initial symbolic
> factorization analysis before the actual numerical factorization starts to
> check if the estimated memory requirements fit the available memory.
> >
> > I am following the steps from
> https://petsc.org/main/src/ksp/ksp/tutorials/ex52.c.html
> >
> >   PCFactorSetMatSolverType(pc,MATSOLVERMUMPS);
> >   PCFactorSetUpMatSolverType(pc);
> >   PCFactorGetMatrix(pc,&F);
> >
> >   KSPSetUp(ksp);
> >   MatMumpsGetInfog(F,...)
> >
> > But it appears KSPSetUp calls both symbolic and numerical factorization.
> So is there some other way to get these statistics before the actual
> factorization starts?
> >
> > Thanks,
> > Varun
> > <slepc_eps_mumps_test.cpp>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220122/6e1f925c/attachment.html>