<div dir="ltr">Great! It compiles now and both the commands -mat_mumps_icntl_7 2 and -mat_mumps_icntl_29 2 work, and perform better compared to the other ordering types. <div></div><div><div><br></div><div>Thank you Pierre! </div><div><br></div><div>As you suggested, I'll also send a new email concerning the errors I have with newer PETSc versions.</div><div><br></div><div>Best,</div><div>Victoria</div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Il giorno mar 7 nov 2023 alle ore 12:25 Pierre Jolivet <<a href="mailto:pierre@joliv.et" target="_blank">pierre@joliv.et</a>> ha scritto:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div style="line-break:after-white-space"><br id="m_-6539003080423495860m_-3732104575223888580lineBreakAtBeginningOfMessage"><div><br><blockquote type="cite"><div>On 7 Nov 2023, at 8:47 PM, Victoria Rolandi <<a href="mailto:victoria.rolandi93@gmail.com" target="_blank">victoria.rolandi93@gmail.com</a>> wrote:</div><br><div><div dir="ltr">Hi Pierre,<div><br></div><div>Thanks for your reply. I am now trying to configure PETSc with the same METIS/ParMETIS of my main code. </div><div><span style="font-family:"Helvetica Neue";font-size:13px"><br></span></div><div><span style="font-family:"Helvetica Neue";font-size:13px">I get the following error, and I still get it even if I change the option --with-precision=double/--with-precision=single</span></div><div><span style="font-family:"Helvetica Neue";font-size:13px"><br></span></div><div><span style="font-family:"Helvetica Neue";font-size:13px">Metis specified is incompatible!<br>IDXTYPEWIDTH=64 metis build appears to be specified for a default 32-bit-indices build of PETSc.<br>Suggest using --download-metis for a compatible metis<br>*******************************************************************************<br></span></div><div><br></div><div>In the cofigure.log I have: </div><div><br></div><div>compilation aborted for /tmp/petsc-yxtl_gwd/config.packages.metis/conftest.c (code 2)<br>Source:<br>#include "confdefs.h"<br>#include "conffix.h"<br>#include "metis.h"<br><br>int main() {<br>#if (IDXTYPEWIDTH != 32)<br>#error incompatible IDXTYPEWIDTH<br>#endif;<br> return 0;<br>}<br></div><div><br></div><div><br></div><div>How could I proceed?</div></div></div></blockquote><div><br></div><div>I would use --download-metis and then have your code use METIS from PETSc, not the other way around.</div><div><br></div><div>Thanks,</div><div>Pierre</div><br><blockquote type="cite"><div><div dir="ltr"><div>Thanks,</div><div>Victoria </div><div><br></div></div><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Il giorno ven 3 nov 2023 alle ore 11:34 Pierre Jolivet <<a href="mailto:pierre@joliv.et" target="_blank">pierre@joliv.et</a>> ha scritto:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div style="line-break:after-white-space"><br id="m_-6539003080423495860m_-3732104575223888580m_-9111504878726065940m_4788862655584302637lineBreakAtBeginningOfMessage"><div><br><blockquote type="cite"><div>On 3 Nov 2023, at 7:28 PM, Victoria Rolandi <<a href="mailto:victoria.rolandi93@gmail.com" target="_blank">victoria.rolandi93@gmail.com</a>> wrote:</div><br><div><div dir="ltr">Pierre, <div><br><div>Sure, I have now installed PETSc with MUMPS and PT-SCHOTCH, I got some errors at the beginning but then it worked adding <span style="font-family:"Helvetica Neue";font-size:13px">--COPTFLAGS="-D_POSIX_C_SOURCE=199309L" to the configuration. </span></div></div><div><span style="font-family:"Helvetica Neue";font-size:13px">Also, I have compilation errors when I try to use newer versions, so I kept the 3.17.0 for the moment.</span></div></div></div></blockquote><div><br></div><div>You should ask for assistance to get the latest version.</div><div>(Par)METIS snapshots may have not changed, but the MUMPS one did, with performance improvements.</div><br><blockquote type="cite"><div><div dir="ltr"><div><span style="font-family:"Helvetica Neue";font-size:13px">Now the parallel ordering works with PT-SCOTCH, however, is it normal that I do not see any difference in the performance compared to sequential ordering ? </span></div></div></div></blockquote><div><br></div><div>Impossible to tell without you providing actual figures (number of nnz, number of processes, timings with sequential ordering, etc.), but 699k is not that big of a problem, so that is not extremely surprising.</div><br><blockquote type="cite"><div><div dir="ltr"><div><span style="font-family:"Helvetica Neue";font-size:13px">Also, could the error using Metis/Parmetis be due to the fact that my main code (to which I linked PETSc) uses a different ParMetis than the one separately installed by PETSC during the configuration?</span></div></div></div></blockquote><div><br></div><div>Yes.</div><br><blockquote type="cite"><div><div dir="ltr"><div><span style="font-family:"Helvetica Neue";font-size:13px">Hence should I configure PETSc linking ParMetis to the same library used by my main code? </span><span style="font-family:"Helvetica Neue";font-size:13px"> </span></div></div></div></blockquote><div><br></div><div>Yes.</div><div><br></div><div>Thanks,</div><div>Pierre</div><br><blockquote type="cite"><div><div dir="ltr"><div><span style="font-family:"Helvetica Neue";font-size:13px">Thanks,</span></div><div><font face="Helvetica Neue">Victoria </font></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Il giorno gio 2 nov 2023 alle ore 09:35 Pierre Jolivet <<a href="mailto:pierre@joliv.et" target="_blank">pierre@joliv.et</a>> ha scritto:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div style="line-break:after-white-space"><br><div><blockquote type="cite"><div>On 2 Nov 2023, at 5:29 PM, Victoria Rolandi <<a href="mailto:victoria.rolandi93@gmail.com" target="_blank">victoria.rolandi93@gmail.com</a>> wrote:</div><br><div><div dir="ltr"><div>Pierre, </div><div>Yes, sorry, I'll keep the list in copy.</div><div>Launching with those options (-mat_mumps_icntl_28 2 -mat_mumps_icntl_29 2) I get an error during the analysis step. I also launched increasing the memory and I still have the error.</div></div></div></blockquote><div><br></div><div>Oh, OK, that’s bad.</div><div>Would you be willing to give SCOTCH and/or PT-SCOTCH a try?</div><div>You’d need to reconfigure/recompile with --download-ptscotch (and maybe --download-bison depending on your system).</div><div>Then, the option would become either -mat_mumps_icntl_28 2 -mat_mumps_icntl_29 2 (PT-SCOTCH) or -mat_mumps_icntl_7 3 (SCOTCH).</div><div>It may be worth updating PETSc as well (you are using 3.17.0, we are at 3.20.1), though I’m not sure we updated the METIS/ParMETIS snapshots since then, so it may not fix the present issue.</div><div><br></div><div>Thanks,</div><div>Pierre</div><br><blockquote type="cite"><div><div dir="ltr"><div><b>The calculations stops at :</b></div><div><br></div><div>Entering CMUMPS 5.4.1 from C interface with JOB, N = 1 699150<br> executing #MPI = 2, without OMP<br><br> =================================================<br> MUMPS compiled with option -Dmetis<br> MUMPS compiled with option -Dparmetis<br> =================================================<br>L U Solver for unsymmetric matrices<br>Type of parallelism: Working host<br><br> ****** ANALYSIS STEP ********<br><br> ** Maximum transversal (ICNTL(6)) not allowed because matrix is distributed<br> Using ParMETIS for parallel ordering<br> Structural symmetry is: 90%<br></div><div><br></div><div><br></div><div><b>The error:</b></div><div><br></div><div>[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range<br>[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br>[0]PETSC ERROR: or see <a href="https://petsc.org/release/faq/#valgrind" target="_blank">https://petsc.org/release/faq/#valgrind</a><br>[0]PETSC ERROR: or try <a href="http://valgrind.org/" target="_blank">http://valgrind.org</a> on GNU/linux and Apple MacOS to find memory corruption errors<br>[0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run<br>[0]PETSC ERROR: to get more information on the crash.<br>[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<br>[0]PETSC ERROR: Signal received<br>[0]PETSC ERROR: See <a href="https://petsc.org/release/faq/" target="_blank">https://petsc.org/release/faq/</a> for trouble shooting.<br>[0]PETSC ERROR: Petsc Release Version 3.17.0, unknown<br>[0]PETSC ERROR: ./charlin.exe on a named n1056 by vrolandi Wed Nov 1 11:38:28 2023<br>[0]PETSC ERROR: Configure options --prefix=/u/home/v/vrolandi/CODES/LIBRARY/packages/petsc/installationDir --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort CXXOPTFLAGS=-O3 --with-scalar-type=complex --with-debugging=0 --with-precision=single --download-mumps --download-scalapack --download-parmetis --download-metis</div><div><br>[0]PETSC ERROR: #1 User provided function() at unknown file:0<br>[0]PETSC ERROR: Run with -malloc_debug to check if memory corruption is causing the crash.<br>Abort(59) on node 0 (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0<br></div><div><br></div><div><br></div><div>Thanks, </div><div>Victoria </div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Il giorno mer 1 nov 2023 alle ore 10:33 Pierre Jolivet <<a href="mailto:pierre@joliv.et" target="_blank">pierre@joliv.et</a>> ha scritto:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div style="line-break:after-white-space"><div>Victoria, please keep the list in copy.</div><div><br></div><div><blockquote type="cite">I am not understanding how can I switch to ParMetis if it does not appear in the options of -mat_mumps_icntl_7.In the options I only have Metis and not ParMetis.</blockquote></div><div><br></div><div>You need to use -mat_mumps_icntl_28 2 -mat_mumps_icntl_29 2</div><div><br></div><div>Barry, I don’t think we can programmatically shut off this warning, it’s guarded by a bunch of KEEP() values, see src/dana_driver.F:4707, which are only settable/gettable by people with access to consortium releases.</div><div>I’ll ask the MUMPS people for confirmation.</div><div>Note that this warning is only printed to screen with the option -mat_mumps_icntl_4 2 (or higher), so this won’t show up for standard runs.</div><div><br></div><div>Thanks,</div><div>Pierre</div><div><br></div><div><blockquote type="cite"><div>On 1 Nov 2023, at 5:52 PM, Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>> wrote:</div><br><div><div style="line-break:after-white-space"><div><br></div> Pierre,<div><br></div><div> Could the PETSc MUMPS interface "turn-off" <span style="background-color:rgb(255,255,255)">ICNTL(6) in this situation so as to not trigger the confusing warning message from MUMPS?</span></div><div><span style="background-color:rgb(255,255,255)"><br></span></div><div><span style="background-color:rgb(255,255,255)"> Barry<br></span><div><br><blockquote type="cite"><div>On Nov 1, 2023, at 12:17 PM, Pierre Jolivet <<a href="mailto:pierre@joliv.et" target="_blank">pierre@joliv.et</a>> wrote:</div><br><div><div style="line-break:after-white-space"><br id="m_-6539003080423495860m_-3732104575223888580m_-9111504878726065940m_4788862655584302637m_72933016981574943m_-7375317822705133688m_-984065845017514147m_-3814297105497396636lineBreakAtBeginningOfMessage"><div><br><blockquote type="cite"><div>On 1 Nov 2023, at 3:33 PM, Zhang, Hong via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:</div><br><div><div style="font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt"><span style="font-size:12px;background-color:rgb(255,255,255);display:inline">Victoria,</span><br></div><div style="font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt"><span style="font-size:12px;background-color:rgb(255,255,255);display:inline"><span style="background-color:rgb(255,255,255);display:inline">"** Maximum transversal (ICNTL(6)) not allowed because matrix is distributed</span></span></div><div style="font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt"><span style="font-size:12px;background-color:rgb(255,255,255);display:inline"><span style="background-color:rgb(255,255,255);display:inline"><span style="background-color:rgb(255,255,255);display:inline">Ordering based on METIS</span>"</span></span></div></div></blockquote><div><br></div><div>This warning is benign and appears for every run using a sequential partitioner in MUMPS with a MATMPIAIJ.</div><div>(I’m not saying switching to ParMETIS will not make the issue go away)</div><div><br></div><div>Thanks,</div><div>Pierre</div><div><br></div><div><div>$ ../../../../arch-darwin-c-debug-real/bin/mpirun -n 2 ./ex2 -pc_type lu -mat_mumps_icntl_4 2</div><div>Entering DMUMPS 5.6.2 from C interface with JOB, N = 1 56</div><div> executing #MPI = 2, without OMP</div><div><br></div><div> =================================================</div><div> MUMPS compiled with option -Dmetis</div><div> MUMPS compiled with option -Dparmetis</div><div> MUMPS compiled with option -Dpord</div><div> MUMPS compiled with option -Dptscotch</div><div> MUMPS compiled with option -Dscotch</div><div> =================================================</div><div>L U Solver for unsymmetric matrices</div><div>Type of parallelism: Working host</div><div><br></div><div> ****** ANALYSIS STEP ********</div><div><br></div><div> ** Maximum transversal (ICNTL(6)) not allowed because matrix is distributed</div><div> Processing a graph of size: 56 with 194 edges</div><div> Ordering based on AMF </div><div> WARNING: Largest root node of size 26 not selected for parallel execution</div><div><br></div><div>Leaving analysis phase with ...</div><div> INFOG(1) = 0</div><div> INFOG(2) = 0</div><div>[…]</div></div><br><blockquote type="cite"><div><div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><font face="Segoe UI Web (West European), Segoe UI, -apple-system, system-ui, Roboto, Helvetica Neue, sans-serif"><span style="font-size:12px">Try parmetis.</span></font></div><div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><font face="Segoe UI Web (West European), Segoe UI, -apple-system, system-ui, Roboto, Helvetica Neue, sans-serif"><span style="font-size:12px">Hong</span></font></div><hr style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;display:inline-block;width:806.53125px"><span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;float:none;display:inline"></span><div id="m_-6539003080423495860m_-3732104575223888580m_-9111504878726065940m_4788862655584302637m_72933016981574943m_-7375317822705133688m_-984065845017514147m_-3814297105497396636divRplyFwdMsg" dir="ltr" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><font face="Calibri, sans-serif" style="font-size:11pt"><b>From:</b><span> </span>petsc-users <<a href="mailto:petsc-users-bounces@mcs.anl.gov" target="_blank">petsc-users-bounces@mcs.anl.gov</a>> on behalf of Victoria Rolandi <<a href="mailto:victoria.rolandi93@gmail.com" target="_blank">victoria.rolandi93@gmail.com</a>><br><b>Sent:</b><span> </span>Tuesday, October 31, 2023 10:30 PM<br><b>To:</b><span> </span><a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br><b>Subject:</b><span> </span>[petsc-users] Error using Metis with PETSc installed with MUMPS</font><div> </div></div><div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><div dir="ltr">Hi, <div><br></div><div>I'm solving a large sparse linear system in parallel and I am using PETSc with MUMPS. I am trying to test different options, like the ordering of the matrix. Everything works if I use the <span style="box-sizing:border-box"><span style="box-sizing:border-box"><font><i>-mat_mumps_icntl_7 2 </i>or<i> </i></font></span></span><span style="box-sizing:border-box"><span style="box-sizing:border-box"><font><i>-mat_mumps_icntl_7 0 </i></font></span></span>options<i> </i><span>(with the first one, AMF, performing better than AMD), however when I test METIS </span><span><i>-mat_mumps_icntl_7</i><span><i> 5<span> </span></i>I get an error (reported at the end of the email).</span></span></div><div><span><span><br></span></span></div><div>I have configured PETSc with the following options: </div><div><br></div><div>--with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-scalar-type=complex --with-debugging=0 --with-precision=single --download-mumps --download-scalapack --download-parmetis --download-metis<br></div><div><br></div><div>and the installation didn't give any problems.</div><div><br></div><div>Could you help me understand why metis is not working? </div><div><br></div><div>Thank you in advance,</div><div>Victoria </div><div><br></div><div>Error:</div><div><br></div><div> ****** ANALYSIS STEP ********<br> ** Maximum transversal (ICNTL(6)) not allowed because matrix is distributed<br> Processing a graph of size: 699150 with 69238690 edges<br> Ordering based on METIS<br>510522 37081376 [100] [10486 699150]<br>Error! Unknown CType: -1</div></div></div></div></blockquote></div><br></div></div></blockquote></div><br></div></div></div></blockquote></div><br></div></blockquote></div>
</div></blockquote></div><br></div></blockquote></div>
</div></blockquote></div><br></div></blockquote></div>
</div></blockquote></div><br></div></blockquote></div>