<div dir="ltr"><div dir="ltr">On Wed, Dec 7, 2022 at 8:15 AM 김성익 <<a href="mailto:ksi2443@gmail.com">ksi2443@gmail.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Following your comments,<br>I used below command<div><div>mpirun -np 4 ./app -ksp_type preonly -pc_type mpi -mpi_linear_solver_server -mpi_pc_type lu -mpi_pc_factor_mat_solver_type mumps -mpi_mat_mumps_icntl_7 5 -mpi_ksp_view<br></div><div><br></div><div><br></div><div>so the output is as below</div><div><br></div><div>KSP Object: (mpi_) 1 MPI process<br> type: gmres<br> restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement<br> happy breakdown tolerance 1e-30<br> maximum iterations=10000, initial guess is zero<br> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.<br> left preconditioning<br> using PRECONDITIONED norm type for convergence test<br>PC Object: (mpi_) 1 MPI process<br> type: lu<br> out-of-place factorization<br> tolerance for zero pivot 2.22045e-14<br> matrix ordering: external<br> factor fill ratio given 0., needed 0.<br> Factored matrix follows:<br> Mat Object: (mpi_) 1 MPI process<br> type: mumps<br> rows=192, cols=192<br> package used to perform factorization: mumps<br> total: nonzeros=17334, allocated nonzeros=17334<br> MUMPS run parameters:<br> Use -mpi_ksp_view ::ascii_info_detail to display information for all processes<br> RINFOG(1) (global estimated flops for the elimination after analysis): 949441.<br> RINFOG(2) (global estimated flops for the assembly after factorization): 18774.<br> RINFOG(3) (global estimated flops for the elimination after factorization): 949441.<br> (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0.,0.)*(2^0)<br> INFOG(3) (estimated real workspace for factors on all processors after analysis): 17334<br> INFOG(4) (estimated integer workspace for factors on all processors after analysis): 1724<br> INFOG(5) (estimated maximum front size in the complete tree): 96<br> INFOG(6) (number of nodes in the complete tree): 16<br> INFOG(7) (ordering option effectively used after analysis): 5<br> INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 100<br> INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 17334<br> INFOG(10) (total integer space store the matrix factors after factorization): 1724<br> INFOG(11) (order of largest frontal matrix after factorization): 96<br> INFOG(12) (number of off-diagonal pivots): 0<br> INFOG(13) (number of delayed pivots after factorization): 0<br> INFOG(14) (number of memory compress after factorization): 0<br> INFOG(15) (number of steps of iterative refinement after solution): 0<br> INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 1<br> INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 1<br> INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 1<br> INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 1<br> INFOG(20) (estimated number of entries in the factors): 17334<br> INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 1<br> INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 1<br> INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0<br> INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1<br> INFOG(25) (after factorization: number of pivots modified by static pivoting): 0<br> INFOG(28) (after factorization: number of null pivots encountered): 0<br> INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 17334<br> INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 0, 0<br> INFOG(32) (after analysis: type of analysis done): 1<br> INFOG(33) (value used for ICNTL(8)): 7<br> INFOG(34) (exponent of the determinant if determinant is requested): 0<br> INFOG(35) (after factorization: number of entries taking into account BLR factor compression - sum over all processors): 17334<br> INFOG(36) (after analysis: estimated size of all MUMPS internal data for running BLR in-core - value on the most memory consuming processor): 0<br> INFOG(37) (after analysis: estimated size of all MUMPS internal data for running BLR in-core - sum over all processors): 0<br> INFOG(38) (after analysis: estimated size of all MUMPS internal data for running BLR out-of-core - value on the most memory consuming processor): 0<br> INFOG(39) (after analysis: estimated size of all MUMPS internal data for running BLR out-of-core - sum over all processors): 0<br> linear system matrix = precond matrix:<br> Mat Object: 1 MPI process<br> type: seqaij<br> rows=192, cols=192<br> total: nonzeros=9000, allocated nonzeros=36864<br> total number of mallocs used during MatSetValues calls=0<br> using I-node routines: found 64 nodes, limit used is 5<br></div><div><br></div><div>Is it correct that I successfully computed with mumps by using metis(icntl_7 5)?</div></div></div></blockquote><div><br></div><div>Yes, first you know it is MUMPS</div><div><br></div><div> Mat Object: (mpi_) 1 MPI process<br> type: mumps<br></div><div><br></div><div>Second you know METIS was used</div><div><br></div><div> INFOG(7) (ordering option effectively used after analysis): 5<br></div><div><br></div><div>I would not expect much speedup on 2 processes with such a small matrix</div><div><br></div><div> rows=192, cols=192<br></div><div><br></div><div> Thanks</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Thanks,</div><div>Hyung Kim</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">2022년 12월 7일 (수) 오후 8:41, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>>님이 작성:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Wed, Dec 7, 2022 at 6:15 AM 김성익 <<a href="mailto:ksi2443@gmail.com" target="_blank">ksi2443@gmail.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">I think I don't understand the meaning of <br>-pc_type mpi</div></blockquote><div><br></div><div>This option says to use the PCMPI preconditioner. This allows you to parallelize the</div><div>solver in what is otherwise a serial code.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>-mpi_pc_type lu</div></div></blockquote><div><br></div><div>This tells the underlying solver in PCMPI to use the LU preconditioner.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>What's the exact meaning of -pc_type mpi and -mpi_pc_type lu??</div><div>Is this difference coming from 'mpi_linear_solver_server' option??</div></div></blockquote><div><br></div><div>Please use -ksp_view as I asked to look at the entire solver. Send it anytime you mail about solver questions.</div><div><br></div><div> Thanks</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Thanks,</div><div>Hyung Kim</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">2022년 12월 7일 (수) 오후 8:05, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>>님이 작성:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Wed, Dec 7, 2022 at 5:13 AM 김성익 <<a href="mailto:ksi2443@gmail.com" target="_blank">ksi2443@gmail.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">I want to use METIS for ordering. <br>I heard the MUMPS has good performance with METIS ordering.<div><br></div><div>However there are some wonder things.</div><div>1. With option "-mpi_linear_solver_server -ksp_type preonly -pc_type mpi -mpi_pc_type lu " the MUMPS solving is slower than with option "-mpi_linear_solver_server -pc_type mpi -ksp_type preonly". <br> Why does this result happen?</div></div></blockquote><div><br></div><div>You are probably not using MUMPS. Always always always use -ksp_view to see exactly what solver you are using.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>2. (MPIRUN case (actually, mpi_linear_solver_server case))) In my code, there is already has "PetscCall(PCSetType(pc,PCLU))" . However, to use METIS by using "-mpi_mat_mumps_icntl_7 5" I must append this option "-mpi_pc_type pu".<br>If I don't apply "-mpi_pc_type lu", the metis option ("-mpi_mat_mumps_icntl_7 5"). Can I get some information about this? </div></div></blockquote><div><br></div><div>Again, it seems like the solver configuration is not what you think it is.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div>Thanks, </div><div>Hyung Kim</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">2022년 12월 7일 (수) 오전 12:24, Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>>님이 작성:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><br><div><br><blockquote type="cite"><div>On Dec 6, 2022, at 5:15 AM, 김성익 <<a href="mailto:ksi2443@gmail.com" target="_blank">ksi2443@gmail.com</a>> wrote:</div><br><div><div dir="ltr"><p class="MsoNormal" style="margin:0cm 0cm 8pt;text-align:justify;line-height:107%;font-size:10pt;font-family:"\00b9d1\00c740 \00ace0\00b515""><span lang="EN-US">Hello,</span></p><div style="margin:0cm 0cm 8pt;text-align:justify;line-height:107%;font-size:10pt;font-family:"\00b9d1\00c740 \00ace0\00b515""><span lang="EN-US"> </span><br></div><p class="MsoNormal" style="margin:0cm 0cm 8pt;text-align:justify;line-height:107%;font-size:10pt;font-family:"\00b9d1\00c740 \00ace0\00b515""><span lang="EN-US">I have some questions about pc and mumps_icntl.</span></p><p style="margin:0cm 0cm 8pt 38pt;text-align:justify;line-height:107%;font-size:10pt;font-family:"\00b9d1\00c740 \00ace0\00b515""><span lang="EN-US">1.<span style="font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:7pt;line-height:normal;font-family:"Times New Roman"">
</span></span><span lang="EN-US">What’s the difference between adopt
preconditioner by code (for example, PetscCall(PCSetType(pc,PCLU)) and option -pc_type
lu??<br>
And also, What’s the priority between code pcsettype and option -pc_type ??<br>
<br>
</span></p><p style="margin:0cm 0cm 8pt 38pt;text-align:justify;line-height:107%;font-size:10pt;font-family:"\00b9d1\00c740 \00ace0\00b515""><span lang="EN-US">2.<span style="font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:7pt;line-height:normal;font-family:"Times New Roman"">
</span></span><span lang="EN-US">When I tried to use METIS in MUMPS,
I adopted metis by option (for example, -mat_mumps_icntl_7 5). In this situation,
it is impossible to use metis without pc_type lu. However, in my case pc type lu
makes the performance poor. So I don’t want to use lu preconditioner. How can I
do this?</span></p></div></div></blockquote><div> The package MUMPS has an option to use metis in its ordering process which can be turned on as indicated while using MUMPS. Most preconditioners that PETSc can use do not use metis for any purpose hence there is no option to turn on its use. For what purpose do you wish to use metis? Partitioning, ordering, ?</div><div><br></div><div><br></div> <br><blockquote type="cite"><div><div dir="ltr"><div style="margin:0cm 0cm 8pt;text-align:justify;line-height:107%;font-size:10pt;font-family:"\00b9d1\00c740 \00ace0\00b515""><span lang="EN-US"> </span><br></div><p class="MsoNormal" style="margin:0cm 0cm 8pt;text-align:justify;line-height:107%;font-size:10pt;font-family:"\00b9d1\00c740 \00ace0\00b515""><span lang="EN-US">Thanks,</span></p><p class="MsoNormal" style="margin:0cm 0cm 8pt;text-align:justify;line-height:107%;font-size:10pt;font-family:"\00b9d1\00c740 \00ace0\00b515""><span lang="EN-US">Hyung Kim</span></p></div>
</div></blockquote></div><br></div></blockquote></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>