[petsc-users] Inconsistent PETSc MUMPS statistics

[Varun Hiremath]

Hi Jose, Pierre,

Yes, I am not using ParMetis or Metis. This is how I am configuring PETSc:
$ ./configure --with-debugging=no --with-mpi-dir=<openmpi>
--with-scalar-type=complex --download-mumps --download-scalapack
--with-blaslapack-dir=<intel mkl>

Would you happen to know if there is any option available to force MUMPS to
be deterministic/repeatable in parallel? I couldn't find anything in the
MUMPS user guide that suggests non-deterministic behavior.

For the example matrix that I shared, the final computed eigenvalues are
within machine precision on multiple runs. However, for some other bigger
cases that I am testing, I get different eigenvalues on multiple runs and I
am trying to figure out the source of these inconsistencies.

Thanks,
Varun

On Sun, Jan 16, 2022 at 6:09 AM Pierre Jolivet <pierre at joliv.et> wrote:

> Default renumbering is sequential. Since --download-parmetis requires
> --download-metis, I doubt Varun is using ParMETIS since the appropriate
> ICNTL flag is not explicitly set.
> If we go back to the original “issue”, I believe this is because you are
> doing an LU factorization with a large number of off diagonal pivots, check
> INFOG(12), which are handled dynamically, and thus may yield different
> factors.
>
> Thanks,
> Pierre
>
> > On 16 Jan 2022, at 12:28 PM, Jose E. Roman <jroman at dsic.upv.es> wrote:
> >
> > Probably someone else can give a better answer, but if I remember
> correctly ParMetis relies on certain random number generator that makes it
> produce different partitions for different runs. I think this was fixed in
> the ParMetis that --download-parmetis installs, but if I am not wrong you
> are not using that version. This would explain what you get.
> >
> > Jose
> >
> >
> >> El 16 ene 2022, a las 11:37, Varun Hiremath <varunhiremath at gmail.com>
> escribió:
> >>
> >> Hi All,
> >>
> >> I am using SLEPc to compute eigenvalues and MUMPS for factorization.
> >>
> >> Please find attached:
> >> 1) A simple test program slepc_eps_mumps_test.cpp that reads a given
> PETSc matrix and computes the smallest eigenvalues using MUMPS for
> factorization
> >> 2) An example PETSc matrix MatA of size 581343 rows (sending in .gz
> format via Google drive link, please extract it "gunzip MatA.gz" before
> using). You should be able to reproduce this issue with any other matrix of
> a similar or bigger size.
> >>
> >> I notice that when I run the attached test program in parallel with the
> attached test matrix the MUMPS statistics printed (using the -eps_view
> option in the command line) change with every run.
> >>
> >> This is how I run the test:
> >> $ mpiexec -n 24 ./slepc_eps_mumps_test.o -nev 5 -f MatA -eps_view
> >> and for example, the output of this includes the following MUMPS stats
> >> ...
> >> PC Object: (st_) 24 MPI processes
> >>    type: lu
> >>      out-of-place factorization
> >>      tolerance for zero pivot 2.22045e-14
> >>      matrix ordering: external
> >>      factor fill ratio given 0., needed 0.
> >>        Factored matrix follows:
> >>          Mat Object: 24 MPI processes
> >>            type: mumps
> >>            rows=581343, cols=581343
> >>            package used to perform factorization: mumps
> >>            total: nonzeros=348236349, allocated nonzeros=348236349
> >>              MUMPS run parameters:
> >>                SYM (matrix type):                   0
> >> ...
> >>
> >> I ran this test 10 times as follows and got a different number of
> nonzeros (line highlighted above ) reported in each run. (If you save the
> full output and compare, you will notice many other differences, but I
> wouldn't have expected the nonzeros to change with every run.)
> >>
> >> $ for i in `seq 1 10`; do echo "run $i :-----"; mpiexec -n 24
> ./slepc_eps_mumps_test.o -nev 5 -f MatA -eps_view | grep -A 1
> "factorization: mumps"; done
> >>
> >> run 1 :-----
> >>            package used to perform factorization: mumps
> >>            total: nonzeros=354789915, allocated nonzeros=354789915
> >> run 2 :-----
> >>            package used to perform factorization: mumps
> >>            total: nonzeros=359811101, allocated nonzeros=359811101
> >> run 3 :-----
> >>            package used to perform factorization: mumps
> >>            total: nonzeros=354834871, allocated nonzeros=354834871
> >> run 4 :-----
> >>            package used to perform factorization: mumps
> >>            total: nonzeros=354830397, allocated nonzeros=354830397
> >> run 5 :-----
> >>            package used to perform factorization: mumps
> >>            total: nonzeros=353942929, allocated nonzeros=353942929
> >> run 6 :-----
> >>            package used to perform factorization: mumps
> >>            total: nonzeros=354147241, allocated nonzeros=354147241
> >> run 7 :-----
> >>            package used to perform factorization: mumps
> >>            total: nonzeros=354980083, allocated nonzeros=354980083
> >> run 8 :-----
> >>            package used to perform factorization: mumps
> >>            total: nonzeros=354980083, allocated nonzeros=354980083
> >> run 9 :-----
> >>            package used to perform factorization: mumps
> >>            total: nonzeros=354214219, allocated nonzeros=354214219
> >> run 10 :-----
> >>            package used to perform factorization: mumps
> >>            total: nonzeros=355894047, allocated nonzeros=355894047
> >>
> >> Can somebody please explain what causes these differences in MUMPS
> stats?
> >>
> >> Thanks,
> >> Varun
> >> MatA.gz
> >>
> >> <slepc_eps_mumps_test.cpp>
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220116/e46abf49/attachment.html>