<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Thu, Dec 11, 2014 at 10:07 AM, Marc MEDALE <span dir="ltr"><<a href="mailto:marc.medale@univ-amu.fr" target="_blank">marc.medale@univ-amu.fr</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">Dear Matt,<div><br></div><div>the output files obtained with PETSc-3.4p4 and 3.5p1 versions using the following command line:</div><div><div>
<span style="border-collapse:separate;border-spacing:0px"><span style="border-collapse:separate;text-align:-webkit-auto;text-indent:0px;border-spacing:0px"><div style="word-wrap:break-word"><span style="text-align:-webkit-auto;text-indent:0px"><div style="word-wrap:break-word"><span style="text-align:-webkit-auto;text-indent:0px"><div style="word-wrap:break-word"><span style="text-indent:0px"><div style="word-wrap:break-word"><span style="text-indent:0px"><div style="word-wrap:break-word"><div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-transform:none;white-space:normal;word-spacing:0px;border-collapse:separate;border-spacing:0px"><span style="font-size:medium">-ksp_type preonly -pc_type lu -pc_factor_mat_solver_package mumps -mat_mumps_icntl_8 0 </span><span style="font-size:medium">-ksp_monitor -ksp_view</span></div><div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-transform:none;white-space:normal;word-spacing:0px;border-collapse:separate;border-spacing:0px"><span style="font-size:medium"><br></span></div><div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-transform:none;white-space:normal;word-spacing:0px;border-collapse:separate;border-spacing:0px"><span style="font-size:medium">are attached below. If skipping flops and memory usage per core, a diff between the two output files reduces to:</span></div><div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-transform:none;white-space:normal;word-spacing:0px;border-collapse:separate;border-spacing:0px">diff Output_3.4p4.txt Output_3.5p1.txt</div><div style="border-collapse:separate;border-spacing:0px"><div style="border-collapse:separate;border-spacing:0px">14c14</div><div style="border-collapse:separate;border-spacing:0px">< Matrix Object: 64 MPI processes</div><div style="border-collapse:separate;border-spacing:0px">---</div><div style="border-collapse:separate;border-spacing:0px">> Mat Object: 64 MPI processes</div><div style="border-collapse:separate;border-spacing:0px">18c18</div><div style="border-collapse:separate;border-spacing:0px">< total: nonzeros=481059588, allocated nonzeros=481059588</div><div style="border-collapse:separate;border-spacing:0px">---</div><div style="border-collapse:separate;border-spacing:0px">> total: nonzeros=4.8106e+08, allocated nonzeros=4.8106e+08</div></div><div style="border-collapse:separate;border-spacing:0px"><div style="border-collapse:separate;border-spacing:0px">457c457</div><div style="border-collapse:separate;border-spacing:0px">< INFOG(10) (total integer space store the matrix factors after factorization): 26149876 </div><div style="border-collapse:separate;border-spacing:0px">---</div><div style="border-collapse:separate;border-spacing:0px">> INFOG(10) (total integer space store the matrix factors after factorization): 26136333 </div><div style="border-collapse:separate;border-spacing:0px">461c461</div><div style="border-collapse:separate;border-spacing:0px">< INFOG(14) (number of memory compress after factorization): 54 </div><div style="border-collapse:separate;border-spacing:0px">---</div><div style="border-collapse:separate;border-spacing:0px">> INFOG(14) (number of memory compress after factorization): 48 </div><div style="border-collapse:separate;border-spacing:0px">468,469c468,469</div><div style="border-collapse:separate;border-spacing:0px">< INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 338 </div><div style="border-collapse:separate;border-spacing:0px">< INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 19782 </div><div style="border-collapse:separate;border-spacing:0px">---</div><div style="border-collapse:separate;border-spacing:0px">> INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 334 </div><div style="border-collapse:separate;border-spacing:0px">> INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 19779 </div><div style="border-collapse:separate;border-spacing:0px">472a473,478</div><div style="border-collapse:separate;border-spacing:0px">> INFOG(28) (after factorization: number of null pivots encountered): 0</div><div style="border-collapse:separate;border-spacing:0px">> INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 470143172</div><div style="border-collapse:separate;border-spacing:0px">> INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 202, 10547</div><div style="border-collapse:separate;border-spacing:0px">> INFOG(32) (after analysis: type of analysis done): 1</div><div style="border-collapse:separate;border-spacing:0px">> INFOG(33) (value used for ICNTL(8)): 0</div><div style="border-collapse:separate;border-spacing:0px">> INFOG(34) (exponent of the determinant if determinant is requested): 0</div><div style="border-collapse:separate;border-spacing:0px">474c480</div><div style="border-collapse:separate;border-spacing:0px">< Matrix Object: 64 MPI processes</div><div style="border-collapse:separate;border-spacing:0px">---</div><div style="border-collapse:separate;border-spacing:0px">> Mat Object: 64 MPI processes</div><div style="border-collapse:separate;border-spacing:0px">477c483</div><div style="border-collapse:separate;border-spacing:0px">< total: nonzeros=63720324, allocated nonzeros=63720324</div><div style="border-collapse:separate;border-spacing:0px">---</div><div style="border-collapse:separate;border-spacing:0px">> total: nonzeros=6.37203e+07, allocated nonzeros=6.37203e+07</div><div style="border-collapse:separate;border-spacing:0px">481c487</div><div style="border-collapse:separate;border-spacing:0px">< Norme de U 1 7.37266E-02, L 1 1.00000E+00</div><div style="border-collapse:separate;border-spacing:0px">---</div><div style="border-collapse:separate;border-spacing:0px">> Norme de U 1 1.61172E-02, L 1 1.00000E+00</div><div style="border-collapse:separate;border-spacing:0px">483c489</div><div style="border-collapse:separate;border-spacing:0px">< Temps total d execution : 198.373291969299 </div><div style="border-collapse:separate;border-spacing:0px">---</div><div style="border-collapse:separate;border-spacing:0px">> Temps total d execution : 216.934082031250 </div><div style="border-collapse:separate;border-spacing:0px"><br></div></div></div></span></div></span></div></span></div></span></div></span></span></div></div></div></blockquote><div><br></div><div>These appear to be two different matrices with the same, or about the same structure. The</div><div>factorization is proceeding differently, I am guessing due to different pivots.</div><div><br></div><div>Can you write the matrix to a binary file using MatView() and load it into both versions so that</div><div>we are certain it is the same?</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div><div><span style="border-collapse:separate;border-spacing:0px"><span style="border-collapse:separate;text-align:-webkit-auto;text-indent:0px;border-spacing:0px"><div style="word-wrap:break-word"><span style="text-align:-webkit-auto;text-indent:0px"><div style="word-wrap:break-word"><span style="text-align:-webkit-auto;text-indent:0px"><div style="word-wrap:break-word"><span style="text-indent:0px"><div style="word-wrap:break-word"><span style="text-indent:0px"><div style="word-wrap:break-word"><div style="border-collapse:separate;border-spacing:0px"><div style="border-collapse:separate;border-spacing:0px"></div><div style="border-collapse:separate;border-spacing:0px">Which does not reveal any striking differences, except in the L2 norm of the solution vectors.</div><div style="border-collapse:separate;border-spacing:0px"><br></div><div style="border-collapse:separate;border-spacing:0px">I need assistance to help me to overcome this quite bizarre behavior.</div><div style="border-collapse:separate;border-spacing:0px"><br></div><div style="border-collapse:separate;border-spacing:0px">Thank you.</div><div style="border-collapse:separate;border-spacing:0px"><br></div>Marc MEDALE</div><div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-transform:none;white-space:normal;word-spacing:0px;border-collapse:separate;border-spacing:0px"><br></div><div style="color:rgb(0,0,0);font-family:Helvetica;font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-transform:none;white-space:normal;word-spacing:0px">=========================================================<br><div>Université Aix-Marseille, Polytech'Marseille, Dépt Mécanique Energétique</div>Laboratoire IUSTI, UMR 7343 CNRS-Université Aix-Marseille<br>Technopole de Chateau-Gombert, 5 rue Enrico Fermi<br>13453 MARSEILLE, Cedex 13, FRANCE<br>---------------------------------------------------------------------------------------------------<br>Tel : <a href="tel:%2B33%20%280%294.91.10.69.14" value="+33491106914" target="_blank">+33 (0)4.91.10.69.14</a> ou 38<br>Fax : <a href="tel:%2B33%20%280%294.91.10.69.69" value="+33491106969" target="_blank">+33 (0)4.91.10.69.69</a><br>e-mail : <a href="mailto:marc.medale@univ-amu.fr" target="_blank">marc.medale@univ-amu.fr</a><br>=========================================================</div><br></div></span><br></div></span></div></span></div></span></div></span></span></div></div></div><br><div style="word-wrap:break-word"><div><div><span style="border-collapse:separate;border-spacing:0px"><span style="border-collapse:separate;text-align:-webkit-auto;text-indent:0px;border-spacing:0px"><div style="word-wrap:break-word"><span style="text-align:-webkit-auto;text-indent:0px"><div style="word-wrap:break-word"><span style="text-align:-webkit-auto;text-indent:0px"><div style="word-wrap:break-word"></div></span></div></span></div></span></span></div></div></div><br><div style="word-wrap:break-word"><div><div><span style="border-collapse:separate;border-spacing:0px"><span style="border-collapse:separate;text-align:-webkit-auto;text-indent:0px;border-spacing:0px"><div style="word-wrap:break-word"><span style="text-align:-webkit-auto;text-indent:0px"><div style="word-wrap:break-word"><span style="text-align:-webkit-auto;text-indent:0px"><div style="word-wrap:break-word"></div></span><br></div></span><br></div></span><br></span><br>
</div>
<br><div><div>Le 11 déc. 2014 à 11:43, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> a écrit :</div><br><blockquote type="cite"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Thu, Dec 11, 2014 at 4:38 AM, Marc MEDALE <span dir="ltr"><<a href="mailto:marc.medale@univ-amu.fr" target="_blank">marc.medale@univ-amu.fr</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Dear PETSC Users,<br>
<br>
I have just updated to PETSc-3.5 my research code that uses PETSc for a while but I'm facing an astonishing difference between PETSc-3.4 to 3.5 versions when solving a very ill conditioned algebraic system with MUMPS (4.10.0 in both cases).<br>
<br>
The only differences that arise in my fortran source code are the following:<br>
Loma1-medale% diff ../version_3.5/solvEFL_MAN_SBIF.F ../version_3.4/solvEFL_MAN_SBIF.F<br>
336,337d335<br>
< CALL MatSetOption(MATGLOB,MAT_KEEP_NONZERO_PATTERN,<br>
< & PETSC_TRUE,IER)<br>
749,750c747,748<br>
< CALL KSPSetTolerances(KSP1,TOL,PETSC_DEFAULT_REAL,<br>
< & PETSC_DEFAULT_REAL,PETSC_DEFAULT_INTEGER,IER)<br>
---<br>
> CALL KSPSetTolerances(KSP1,TOL,PETSC_DEFAULT_DOUBLE_PRECISION,<br>
> & PETSC_DEFAULT_DOUBLE_PRECISION,PETSC_DEFAULT_INTEGER,IER)<br>
909c907,908<br>
< CALL KSPSetOperators(KSP1,MATGLOB,MATGLOB,IER)<br>
---<br>
> CALL KSPSetOperators(KSP1,MATGLOB,MATGLOB,<br>
> & SAME_NONZERO_PATTERN,IER)<br>
<br>
When I run the corresponding program versions on 128 cores of our cluster with the same input data and the following command line arguments:<br>
-ksp_type preonly -pc_type lu -pc_factor_mat_solver_package mumps -mat_mumps_icntl_8 0<br>
<br>
I get the following outputs:<br>
a) with PETSc-3.4p4:<br>
L2 norm of solution vector: 7.39640E-02,<br>
<br>
b) with PETSc-3.5p1:<br>
L2 norm of solution vector: 1.61325E-02<br>
<br>
Do I have change something else in updating my code based on KSP from PETSc-3.4 to 3.5 versions?<br>
Do any default values in the PETSc-MUMPS interface have been changed from PETSc-3.4 to 3.5?<br>
Any hints or suggestions are welcome to help me to recover the right results (obtained with PETSc-3.4).<br></blockquote><div><br></div><div>Send the output from -ksp_monitor -ksp_view for both runs. I am guessing that a MUMPS default changed between versions.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Thank you very much.<br>
<span><font color="#888888"><br>
Marc MEDALE.</font></span></blockquote></div><br><br clear="all"><div><br></div>-- <br><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div>
</div></div>
</blockquote></div><br></div></div><br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div>
</div></div>