[petsc-users] Random Error of mumps: out of memory: INFOG(1)=-9

[Zongze Yang]

On Sat, 4 Mar 2023 at 22:03, Pierre Jolivet <pierre at joliv.et> wrote:

>
>
> On 4 Mar 2023, at 2:51 PM, Zongze Yang <yangzongze at gmail.com> wrote:
>
>
>
> On Sat, 4 Mar 2023 at 21:37, Pierre Jolivet <pierre at joliv.et> wrote:
>
>>
>>
>> > On 4 Mar 2023, at 2:30 PM, Zongze Yang <yangzongze at gmail.com> wrote:
>> >
>> > Hi,
>> >
>> > I am writing to seek your advice regarding a problem I encountered
>> while using multigrid to solve a certain issue.
>> > I am currently using multigrid with the coarse problem solved by PCLU.
>> However, the PC failed randomly with the error below (the value of INFO(2)
>> may differ):
>> > ```shell
>> > [ 0] Error reported by MUMPS in numerical factorization phase:
>> INFOG(1)=-9, INFO(2)=36
>> > ```
>> >
>> > Upon checking the documentation of MUMPS, I discovered that increasing
>> the value of ICNTL(14) may help resolve the issue. Specifically, I set the
>> option -mat_mumps_icntl_14 to a higher value (such as 40), and the error
>> seemed to disappear after I set the value of ICNTL(14) to 80. However, I am
>> still curious as to why MUMPS failed randomly in the first place.
>> >
>> > Upon further inspection, I found that the number of nonzeros of the
>> PETSc matrix and the MUMPS matrix were different every time I ran the code.
>> I am now left with the following questions:
>> >
>> > 1. What could be causing the number of nonzeros of the MUMPS matrix to
>> change every time I run the code?
>>
>> Is the Mat being fed to MUMPS distributed on a communicator of size
>> greater than one?
>> If yes, then, depending on the pivoting and the renumbering, you may get
>> non-deterministic results.
>>
>
> Hi, Pierre,
> Thank you for your prompt reply. Yes, the size of the communicator is
> greater than one.
> Even if the size of the communicator is equal, are the results
> still non-deterministic?
>
>
> In the most general case, yes.
>
> Can I assume the Mat being fed to MUMPS is the same in this case?
>
>
> Are you doing algebraic or geometric multigrid?
> Are the prolongation operators computed by Firedrake or by PETSc, e.g.,
> through GAMG?
> If it’s the latter, I believe the Mat being fed to MUMPS should always be
> the same.
> If it’s the former, you’ll have to ask the Firedrake people if there may
> be non-determinism in the coarsening process.
>

I am using geometric multigrid, and the prolongation operators, I think,
are computed by Firedrake.
Thanks for your suggestion, I will ask the Firedrake people.


>
> Is the pivoting and renumbering all done by MUMPS other than PETSc?
>
>
> You could provide your own numbering, but by default, this is outsourced
> to MUMPS indeed, which will itself outsourced this to METIS, AMD, etc.
>

I think I won't do this.
By the way, does the result of superlu_dist  have a similar
non-deterministic?

Thanks,
Zongze


> Thanks,
> Pierre
>
>
>> > 2. Why is the number of nonzeros of the MUMPS matrix significantly
>> greater than that of the PETSc matrix (as seen in the output of ksp_view,
>> 115025949 vs 7346177)?
>>
>> Exact factorizations introduce fill-in.
>> The number of nonzeros you are seeing for MUMPS is the number of nonzeros
>> in the factors.
>>
>> > 3. Is it possible that the varying number of nonzeros of the MUMPS
>> matrix is the cause of the random failure?
>>
>> Yes, MUMPS uses dynamic scheduling, which will depend on numerical
>> pivoting, and which may generate factors with different number of nonzeros.
>>
>
> Got it. Thank you for your clear explanation.
> Zongze
>
>
>> Thanks,
>> Pierre
>
>
>> > I have attached a test example written in Firedrake. The output of
>> `ksp_view` after running the code twice is included below for your
>> reference.
>> > In the output, the number of nonzeros of the MUMPS matrix was 115025949
>> and 115377847, respectively, while that of the PETSc matrix was only
>> 7346177.
>> >
>> > ```shell
>> > (complex-int32-mkl) $ mpiexec -n 32 python test_mumps.py -ksp_view
>> ::ascii_info_detail | grep -A3 "type: "
>> >   type: preonly
>> >   maximum iterations=10000, initial guess is zero
>> >   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>> >   left preconditioning
>> > --
>> >   type: lu
>> >     out-of-place factorization
>> >     tolerance for zero pivot 2.22045e-14
>> >     matrix ordering: external
>> > --
>> >           type: mumps
>> >           rows=1050625, cols=1050625
>> >           package used to perform factorization: mumps
>> >           total: nonzeros=115025949, allocated nonzeros=115025949
>> > --
>> >     type: mpiaij
>> >     rows=1050625, cols=1050625
>> >     total: nonzeros=7346177, allocated nonzeros=7346177
>> >     total number of mallocs used during MatSetValues calls=0
>> > (complex-int32-mkl) $ mpiexec -n 32 python test_mumps.py -ksp_view
>> ::ascii_info_detail | grep -A3 "type: "
>> >   type: preonly
>> >   maximum iterations=10000, initial guess is zero
>> >   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>> >   left preconditioning
>> > --
>> >   type: lu
>> >     out-of-place factorization
>> >     tolerance for zero pivot 2.22045e-14
>> >     matrix ordering: external
>> > --
>> >           type: mumps
>> >           rows=1050625, cols=1050625
>> >           package used to perform factorization: mumps
>> >           total: nonzeros=115377847, allocated nonzeros=115377847
>> > --
>> >     type: mpiaij
>> >     rows=1050625, cols=1050625
>> >     total: nonzeros=7346177, allocated nonzeros=7346177
>> >     total number of mallocs used during MatSetValues calls=0
>> > ```
>> >
>> > I would greatly appreciate any insights you may have on this matter.
>> Thank you in advance for your time and assistance.
>> >
>> > Best wishes,
>> > Zongze
>> > <test_mumps.py>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20230304/4c2fd037/attachment-0001.html>