<div dir="ltr"><div class="gmail_default" style="font-family:comic sans ms,sans-serif;font-size:small">Some graph preprocessing steps can be skipped ONLY IF a previous factorization was done, and the information can be reused (AS INPUT) to the new factorization. <br><br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;font-size:small">In general, the driver routine SRC/pdgssvx.c() performs the LU factorization of the following (preprocessed) matrix:<br> Pc*Pr*diag(R)*A*diag(C)*Pc^T = L*U<br><br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;font-size:small">The default is to do LU from scratch, including all the steps to compute equilibration (R, C), pivot ordering (Pr), and sparsity ordering (Pc). <br><br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;font-size:small">-- The default should be set as options.Fact = DOFACT.<br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;font-size:small"> <br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;font-size:small">-- When you set options.Fact = SamePattern, the sparsity ordering step is skipped, but you need to input Pc which was obtained from a previous factorization.<br><br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;font-size:small">-- When you set options.Fact = SamePattern_SameRowPerm, both sparsity reordering and pivoting ordering steps are skipped, but you need to input both Pr and Pc.<br><br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;font-size:small">Please see Lines 258 - 307 comments in SRC/pdgssvx.c for details, regarding which data structures should be inputs and which are outputs. The Users Guide also explains this.<br><br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;font-size:small">In EXAMPLE/ directory, I have various examples of these usage situations, see EXAMPLE/README.<br><br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;font-size:small">I am a little puzzled why in PETSc, the default is set to SamePattern ??<br><br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;font-size:small">Sherry<br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;font-size:small"><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Oct 25, 2016 at 9:18 AM, Hong <span dir="ltr"><<a href="mailto:hzhang@mcs.anl.gov" target="_blank">hzhang@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Sherry,<div><br><div>We set <span style="font-size:12.8px">'-mat_superlu_dist_fact </span><span class="m_-4562941754056311974gmail-il" style="background-color:rgb(255,255,255);font-size:12.8px">Sa<wbr>mePattern</span><span style="font-size:12.8px">' as default in petsc/superlu_dist on 12/6/15 (see attached email below). </span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">However, Anton must set '</span><span style="font-size:12.8px">SamePattern_SameRowPerm' to avoid crash in his code. Checking </span></div><div><a href="http://crd-legacy.lbl.gov/~xiaoye/SuperLU/superlu_dist_code_html/pzgssvx___a_bglobal_8c.html" target="_blank">http://crd-legacy.lbl.gov/~<wbr>xiaoye/SuperLU/superlu_dist_<wbr>code_html/pzgssvx___a_bglobal_<wbr>8c.html</a><br></div><div>I see detailed description on using <span style="font-size:12.8px">SamePattern_SameRowPerm, which requires more from user than </span><span style="font-size:12.8px">SamePattern. I guess these flags are used for efficiency. The library sets a default, then have users to switch for their own applications. The default setting should not cause crash. If crash occurs, give a meaningful error message would be help.</span></div></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">Do you have suggestion how should we set default in petsc for this flag?</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">Hong</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">-------------------</span></div><div><div class="m_-4562941754056311974gmail-gE m_-4562941754056311974gmail-iv m_-4562941754056311974gmail-gt" style="font-size:12.8px"><table class="m_-4562941754056311974gmail-cf m_-4562941754056311974gmail-gJ" cellpadding="0"><tbody><tr class="m_-4562941754056311974gmail-acZ"><td class="m_-4562941754056311974gmail-gF m_-4562941754056311974gmail-gK" style="width:368.8px"><table class="m_-4562941754056311974gmail-cf m_-4562941754056311974gmail-ix" style="width:368.8px" cellpadding="0"><tbody><tr><td><h3 class="m_-4562941754056311974gmail-iw"><span name="Hong" class="m_-4562941754056311974gmail-gD" style="font-size:12.8px">Hong</span> <span class="m_-4562941754056311974gmail-go"><span><</span><a href="mailto:hzhang@mcs.anl.gov" target="_blank">hzhang@mcs.anl.gov</a><span>></span></span></h3></td></tr></tbody></table></td><td class="m_-4562941754056311974gmail-gH"><div class="m_-4562941754056311974gmail-gK"><span id="m_-4562941754056311974gmail-:1ue" class="m_-4562941754056311974gmail-g3" title="Mon, Dec 7, 2015 at 12:42 PM" alt="Mon, Dec 7, 2015 at 12:42 PM">12/7/15</span><div class="m_-4562941754056311974gmail-zd" style="outline:0px"><span class="m_-4562941754056311974gmail-T-KT"><img class="m_-4562941754056311974gmail-f m_-4562941754056311974gmail-T-KT-JX" src="https://mail.google.com/mail/u/0/images/cleardot.gif" alt=""></span></div></div></td><td class="m_-4562941754056311974gmail-gH"></td><td class="m_-4562941754056311974gmail-gH m_-4562941754056311974gmail-acX" rowspan="2"><div class="m_-4562941754056311974gmail-T-I m_-4562941754056311974gmail-J-J5-Ji m_-4562941754056311974gmail-T-I-Js-IF m_-4562941754056311974gmail-aaq m_-4562941754056311974gmail-T-I-ax7 m_-4562941754056311974gmail-L3"><img class="m_-4562941754056311974gmail-hB m_-4562941754056311974gmail-T-I-J3" src="https://mail.google.com/mail/u/0/images/cleardot.gif" alt=""></div><div id="m_-4562941754056311974gmail-:1r4" class="m_-4562941754056311974gmail-T-I m_-4562941754056311974gmail-J-J5-Ji m_-4562941754056311974gmail-T-I-Js-Gs m_-4562941754056311974gmail-aap m_-4562941754056311974gmail-T-I-awG m_-4562941754056311974gmail-T-I-ax7 m_-4562941754056311974gmail-L3"><img class="m_-4562941754056311974gmail-hA m_-4562941754056311974gmail-T-I-J3" src="https://mail.google.com/mail/u/0/images/cleardot.gif" alt=""></div></td></tr><tr class="m_-4562941754056311974gmail-acZ m_-4562941754056311974gmail-xD"><td colspan="3"><table class="m_-4562941754056311974gmail-cf m_-4562941754056311974gmail-adz" style="width:448px" cellpadding="0"><tbody><tr><td class="m_-4562941754056311974gmail-ady"><div class="m_-4562941754056311974gmail-iw m_-4562941754056311974gmail-ajw"><span class="m_-4562941754056311974gmail-hb">to <span dir="ltr" name="Danyang" class="m_-4562941754056311974gmail-g2">Danyang</span>, <span dir="ltr" name="petsc-maint" class="m_-4562941754056311974gmail-g2">petsc-maint</span>, <span dir="ltr" name="PETSc" class="m_-4562941754056311974gmail-g2">PETSc</span><wbr>, <span dir="ltr" name="Xiaoye" class="m_-4562941754056311974gmail-g2">Xiaoye</span></span></div><div class="m_-4562941754056311974gmail-ajy"><img class="m_-4562941754056311974gmail-ajz" id="m_-4562941754056311974gmail-:1r3" src="https://mail.google.com/mail/u/0/images/cleardot.gif" alt=""></div></td></tr></tbody></table></td></tr></tbody></table></div><div class="m_-4562941754056311974gmail-qQVYZb"></div><div class="m_-4562941754056311974gmail-utdU2e"></div><div class="m_-4562941754056311974gmail-tx78Ic"></div><div class="m_-4562941754056311974gmail-aHl"></div><div id="m_-4562941754056311974gmail-:1r2"></div><div id="m_-4562941754056311974gmail-:1uc" class="m_-4562941754056311974gmail-ii m_-4562941754056311974gmail-gt m_-4562941754056311974gmail-adP m_-4562941754056311974gmail-adO" style="font-size:12.8px"><div id="m_-4562941754056311974gmail-:1ub" class="m_-4562941754056311974gmail-a3s m_-4562941754056311974gmail-aXjCH m_-4562941754056311974gmail-m1517dc034049d992"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">Danyang :</div><div class="gmail_quote"><br></div><div class="gmail_quote">Adding '-mat_superlu_dist_fact <span class="m_-4562941754056311974gmail-il">SamePa<wbr>ttern</span>' fixed the problem. Below is how I figured it out.</div><div class="gmail_quote"><br></div><div class="gmail_quote"><div class="gmail_quote" style="font-size:12.8px">1. Reading ex52f.F, I see '-superlu_default' = '-pc_factor_mat_solver_package superlu_dist', the later enables runtime options for other packages. I use superlu_dist-4.2 and superlu-4.1 for the tests below.</div><div>...</div><div><div class="gmail_quote" style="font-size:12.8px">5.</div><div class="gmail_quote" style="font-size:12.8px">Using a_flow_check_1.bin, I am able to reproduce the error you reported: all packages give correct results except superlu_dist:</div><div class="gmail_quote" style="font-size:12.8px">./ex52f -f0 matrix_and_rhs_bin/a_flow_chec<wbr>k_1.bin -rhs matrix_and_rhs_bin/b_flow_chec<wbr>k_168.bin -loop_matrices flow_check -loop_folder matrix_and_rhs_bin -pc_type lu -pc_factor_mat_solver_package superlu_dist</div><div class="gmail_quote" style="font-size:12.8px">Norm of error 2.5970E-12 iterations 1</div><span class="m_-4562941754056311974gmail-im" style="font-size:12.8px"><div class="gmail_quote"> -->Test for matrix 168</div></span><div class="gmail_quote" style="font-size:12.8px">Norm of error 1.3936E-01 iterations 34</div><span class="m_-4562941754056311974gmail-im" style="font-size:12.8px"><div class="gmail_quote"> -->Test for matrix 169</div><div class="gmail_quote"><br></div></span><div class="gmail_quote" style="font-size:12.8px">I guess the error might come from reuse of matrix factor. Replacing default</div><div class="gmail_quote" style="font-size:12.8px">-mat_superlu_dist_fact <<span class="m_-4562941754056311974gmail-il">SamePattern_SameRowPerm</span>> with</div><div class="gmail_quote" style="font-size:12.8px">-mat_superlu_dist_fact <span class="m_-4562941754056311974gmail-il">SamePat<wbr>tern</span>, I get</div><div class="gmail_quote" style="font-size:12.8px"><br></div><div class="gmail_quote" style="font-size:12.8px">./ex52f -f0 matrix_and_rhs_bin/a_flow_chec<wbr>k_1.bin -rhs matrix_and_rhs_bin/b_flow_chec<wbr>k_168.bin -loop_matrices flow_check -loop_folder matrix_and_rhs_bin -pc_type lu -pc_factor_mat_solver_package superlu_dist -mat_superlu_dist_fact <span class="m_-4562941754056311974gmail-il">SamePat<wbr>tern</span></div><div class="gmail_quote" style="font-size:12.8px"><br></div><div class="gmail_quote" style="font-size:12.8px">Norm of error 2.5970E-12 iterations 1</div><span class="m_-4562941754056311974gmail-im" style="font-size:12.8px"><div class="gmail_quote"> -->Test for matrix 168</div><div class="gmail_quote">...</div><div class="gmail_quote"><div class="gmail_quote" style="font-size:12.8px">Sherry may tell you why <span class="m_-4562941754056311974gmail-il">SamePattern_SameRowPerm</span> <wbr>cause the difference here.</div><div class="gmail_quote" style="font-size:12.8px">Best on the above experiments, I would set following as default</div><div class="gmail_quote" style="font-size:12.8px">'-mat_superlu_diagpivotthresh 0.0' in petsc/superlu interface.</div><div class="gmail_quote" style="font-size:12.8px">'-mat_superlu_dist_fact <span class="m_-4562941754056311974gmail-il">SamePa<wbr>ttern</span>' in petsc/superlu_dist interface.</div><span class="HOEnZb"><font color="#888888"><div class="gmail_quote" style="font-size:12.8px"><br></div><div class="gmail_quote" style="font-size:12.8px">Hong</div><div class="m_-4562941754056311974gmail-yj6qo m_-4562941754056311974gmail-ajU" style="font-size:12.8px"></div></font></span></div></span></div></div></div></div></div></div></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Oct 25, 2016 at 10:38 AM, Hong <span dir="ltr"><<a href="mailto:hzhang@mcs.anl.gov" target="_blank">hzhang@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Anton,<div>I guess, when you reuse matrix and its symbolic factor with updated numerical values, superlu_dist requires this option. I'm cc'ing Sherry to confirm it.</div><div><br></div><div>I'll check petsc/superlu-dist interface to set this flag for this case.</div><span class="m_-4562941754056311974HOEnZb"><font color="#888888"><div><br></div></font></span><div><span class="m_-4562941754056311974HOEnZb"><font color="#888888">Hong</font></span><div><div class="m_-4562941754056311974h5"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Oct 25, 2016 at 8:20 AM, Anton Popov <span dir="ltr"><<a href="mailto:popov@uni-mainz.de" target="_blank">popov@uni-mainz.de</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<p>Hong,</p>
<p>I get all the problems gone and valgrind-clean output if I
specify this:</p>
<p>-mat_superlu_dist_fact SamePattern_SameRowPerm<br>
</p>
What does SamePattern_SameRowPerm actually mean? <br>
Row permutations are for large diagonal, column permutations are for
sparsity, right?<br>
Will it skip subsequent matrix permutations for large diagonal even
if matrix values change significantly?<br>
<br>
Surprisingly everything works even with:<br>
<br>
-mat_superlu_dist_colperm PARMETIS<br>
-mat_superlu_dist_parsymbfact TRUE<br>
<br>
Thanks,<br>
Anton<br>
<br>
<div class="m_-4562941754056311974m_-2683123818943114666m_-1905162172435090326moz-cite-prefix">On 10/24/2016 09:06 PM, Hong wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">Anton:
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"><span class="m_-4562941754056311974m_-2683123818943114666m_-1905162172435090326gmail-">
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">If replacing
superlu_dist with mumps, does your code work?</div>
</div>
</div>
</blockquote>
</span> yes</div>
</blockquote>
<div> </div>
<div>You may use mumps in your code, or tests different
options for superlu_dist:</div>
<div><br>
</div>
<div>
<div> -mat_superlu_dist_equil: <TRUE> Equilibrate
matrix (None)</div>
<div> -mat_superlu_dist_rowperm <LargeDiag> Row
permutation (choose one of) LargeDiag NATURAL (None)</div>
<div> -mat_superlu_dist_colperm <METIS_AT_PLUS_A>
Column permutation (choose one of) NATURAL MMD_AT_PLUS_A
MMD_ATA METIS_AT_PLUS_A PARMETIS (None)</div>
<div> -mat_superlu_dist_replacetinyp<wbr>ivot: <FALSE>
Replace tiny pivots (None)</div>
<div> -mat_superlu_dist_parsymbfact: <FALSE>
Parallel symbolic factorization (None)</div>
<div> -mat_superlu_dist_fact <SamePattern> Sparsity
pattern for repeated matrix factorization (choose one
of) SamePattern SamePattern_SameRowPerm (None)</div>
</div>
<div><br>
</div>
<div>The options inside <> are defaults. You may try
others. This might help narrow down the bug.</div>
<div><br>
</div>
<div>Hong</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"><span class="m_-4562941754056311974m_-2683123818943114666m_-1905162172435090326gmail-"><br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">Hong
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"><span class="m_-4562941754056311974m_-2683123818943114666m_-1905162172435090326gmail-m_-8985211249997465386gmail-">
<div class="m_-4562941754056311974m_-2683123818943114666m_-1905162172435090326gmail-m_-8985211249997465386gmail-m_4450561754203495905moz-cite-prefix">On
10/24/2016 05:47 PM, Hong wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Barry,
<div>Your change indeed fixed the
error of his testing code.</div>
<div>As Satish tested, on your
branch, ex16 runs smooth.</div>
<div><br>
</div>
<div>I do not understand why on
maint or master branch, ex16
creases inside superlu_dist, but
not with mumps. </div>
<div><br>
</div>
</div>
</blockquote>
<br>
</span> I also confirm that ex16 runs fine
with latest fix, but unfortunately not my
code.<br>
<br>
This is something to be expected, since my
code preallocates once in the beginning.
So there is no way it can be affected by
multiple preallocations. Subsequently I
only do matrix assembly, that makes sure
structure doesn't change (set to get error
otherwise).<br>
<br>
Summary: we don't have a simple test code
to debug superlu issue anymore.<span class="m_-4562941754056311974m_-2683123818943114666m_-1905162172435090326gmail-m_-8985211249997465386gmail-HOEnZb"><font color="#888888"><br>
<br>
Anton</font></span><span class="m_-4562941754056311974m_-2683123818943114666m_-1905162172435090326gmail-m_-8985211249997465386gmail-"><br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div>Hong<br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon,
Oct 24, 2016 at 9:34 AM,
Satish Balay <span dir="ltr"><<a href="mailto:balay@mcs.anl.gov" target="_blank">balay@mcs.anl.gov</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span>On Mon, 24 Oct 2016, Barry
Smith wrote:<br>
<br>
><br>
</span><span>> > [Or
perhaps Hong is using a
different test code and is
observing bugs<br>
> > with
superlu_dist interface..]<br>
><br>
> She states that
her test does a NEW
MatCreate() for each
matrix load (I cut and
pasted it in the email I
just sent). The bug I
fixed was only related to
using the SAME matrix from
one MatLoad() in another
MatLoad().<br>
<br>
</span>Ah - ok.. Sorry -
wasn't thinking clearly :(<br>
<span class="m_-4562941754056311974m_-2683123818943114666m_-1905162172435090326gmail-m_-8985211249997465386gmail-m_4450561754203495905HOEnZb"><font color="#888888"><br>
Satish<br>
</font></span></blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
<br>
</span></div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
</span></div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
</div>
</blockquote></div><br></div></div></div></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>