<div dir="ltr">Hi Jose, Pierre,<div><br></div><div>Yes, I am not using ParMetis or Metis. This is how I am configuring PETSc:</div>$ ./configure --with-debugging=no --with-mpi-dir=<openmpi> --with-scalar-type=complex --download-mumps --download-scalapack --with-blaslapack-dir=<intel mkl><div><br></div><div>Would you happen to know if there is any option available to force MUMPS to be deterministic/repeatable in parallel? I couldn't find anything in the MUMPS user guide that suggests non-deterministic behavior.</div><div><br></div><div>For the example matrix that I shared, the final computed eigenvalues are within machine precision on multiple runs. However, for some other bigger cases that I am testing, I get different eigenvalues on multiple runs and I am trying to figure out the source of these inconsistencies.</div><div><br></div><div>Thanks,</div><div>Varun</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Jan 16, 2022 at 6:09 AM Pierre Jolivet <<a href="mailto:pierre@joliv.et">pierre@joliv.et</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Default renumbering is sequential. Since --download-parmetis requires --download-metis, I doubt Varun is using ParMETIS since the appropriate ICNTL flag is not explicitly set.<br>
If we go back to the original “issue”, I believe this is because you are doing an LU factorization with a large number of off diagonal pivots, check INFOG(12), which are handled dynamically, and thus may yield different factors.<br>
<br>
Thanks,<br>
Pierre<br>
<br>
> On 16 Jan 2022, at 12:28 PM, Jose E. Roman <<a href="mailto:jroman@dsic.upv.es" target="_blank">jroman@dsic.upv.es</a>> wrote:<br>
> <br>
> Probably someone else can give a better answer, but if I remember correctly ParMetis relies on certain random number generator that makes it produce different partitions for different runs. I think this was fixed in the ParMetis that --download-parmetis installs, but if I am not wrong you are not using that version. This would explain what you get.<br>
> <br>
> Jose<br>
> <br>
> <br>
>> El 16 ene 2022, a las 11:37, Varun Hiremath <<a href="mailto:varunhiremath@gmail.com" target="_blank">varunhiremath@gmail.com</a>> escribió:<br>
>> <br>
>> Hi All,<br>
>> <br>
>> I am using SLEPc to compute eigenvalues and MUMPS for factorization. <br>
>> <br>
>> Please find attached:<br>
>> 1) A simple test program slepc_eps_mumps_test.cpp that reads a given PETSc matrix and computes the smallest eigenvalues using MUMPS for factorization<br>
>> 2) An example PETSc matrix MatA of size 581343 rows (sending in .gz format via Google drive link, please extract it "gunzip MatA.gz" before using). You should be able to reproduce this issue with any other matrix of a similar or bigger size.<br>
>> <br>
>> I notice that when I run the attached test program in parallel with the attached test matrix the MUMPS statistics printed (using the -eps_view option in the command line) change with every run.<br>
>> <br>
>> This is how I run the test:<br>
>> $ mpiexec -n 24 ./slepc_eps_mumps_test.o -nev 5 -f MatA -eps_view<br>
>> and for example, the output of this includes the following MUMPS stats<br>
>> ...<br>
>> PC Object: (st_) 24 MPI processes<br>
>> type: lu<br>
>> out-of-place factorization<br>
>> tolerance for zero pivot 2.22045e-14<br>
>> matrix ordering: external<br>
>> factor fill ratio given 0., needed 0.<br>
>> Factored matrix follows:<br>
>> Mat Object: 24 MPI processes<br>
>> type: mumps<br>
>> rows=581343, cols=581343<br>
>> package used to perform factorization: mumps<br>
>> total: nonzeros=348236349, allocated nonzeros=348236349<br>
>> MUMPS run parameters:<br>
>> SYM (matrix type): 0<br>
>> ...<br>
>> <br>
>> I ran this test 10 times as follows and got a different number of nonzeros (line highlighted above ) reported in each run. (If you save the full output and compare, you will notice many other differences, but I wouldn't have expected the nonzeros to change with every run.)<br>
>> <br>
>> $ for i in `seq 1 10`; do echo "run $i :-----"; mpiexec -n 24 ./slepc_eps_mumps_test.o -nev 5 -f MatA -eps_view | grep -A 1 "factorization: mumps"; done<br>
>> <br>
>> run 1 :-----<br>
>> package used to perform factorization: mumps<br>
>> total: nonzeros=354789915, allocated nonzeros=354789915<br>
>> run 2 :-----<br>
>> package used to perform factorization: mumps<br>
>> total: nonzeros=359811101, allocated nonzeros=359811101<br>
>> run 3 :-----<br>
>> package used to perform factorization: mumps<br>
>> total: nonzeros=354834871, allocated nonzeros=354834871<br>
>> run 4 :-----<br>
>> package used to perform factorization: mumps<br>
>> total: nonzeros=354830397, allocated nonzeros=354830397<br>
>> run 5 :-----<br>
>> package used to perform factorization: mumps<br>
>> total: nonzeros=353942929, allocated nonzeros=353942929<br>
>> run 6 :-----<br>
>> package used to perform factorization: mumps<br>
>> total: nonzeros=354147241, allocated nonzeros=354147241<br>
>> run 7 :-----<br>
>> package used to perform factorization: mumps<br>
>> total: nonzeros=354980083, allocated nonzeros=354980083<br>
>> run 8 :-----<br>
>> package used to perform factorization: mumps<br>
>> total: nonzeros=354980083, allocated nonzeros=354980083<br>
>> run 9 :-----<br>
>> package used to perform factorization: mumps<br>
>> total: nonzeros=354214219, allocated nonzeros=354214219<br>
>> run 10 :-----<br>
>> package used to perform factorization: mumps<br>
>> total: nonzeros=355894047, allocated nonzeros=355894047<br>
>> <br>
>> Can somebody please explain what causes these differences in MUMPS stats? <br>
>> <br>
>> Thanks,<br>
>> Varun<br>
>> MatA.gz<br>
>> <br>
>> <slepc_eps_mumps_test.cpp><br>
> <br>
<br>
</blockquote></div>