<div dir="ltr">Sherry,<div><br><div>We set <span style="font-size:12.8px">'-mat_superlu_dist_fact </span><span class="gmail-il" style="background-color:rgb(255,255,255);font-size:12.8px">SamePattern</span><span style="font-size:12.8px">' as default in petsc/superlu_dist on 12/6/15 (see attached email below). </span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">However, Anton must set '</span><span style="font-size:12.8px">SamePattern_SameRowPerm' to avoid crash in his code. Checking </span></div><div><a href="http://crd-legacy.lbl.gov/~xiaoye/SuperLU/superlu_dist_code_html/pzgssvx___a_bglobal_8c.html">http://crd-legacy.lbl.gov/~xiaoye/SuperLU/superlu_dist_code_html/pzgssvx___a_bglobal_8c.html</a><br></div><div>I see detailed description on using <span style="font-size:12.8px">SamePattern_SameRowPerm, which requires more from user than </span><span style="font-size:12.8px">SamePattern. I guess these flags are used for efficiency. The library sets a default, then have users to switch for their own applications. The default setting should not cause crash. If crash occurs, give a meaningful error message would be help.</span></div></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">Do you have suggestion how should we set default in petsc for this flag?</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">Hong</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">-------------------</span></div><div><div class="gmail-gE gmail-iv gmail-gt" style="font-size:12.8px"><table cellpadding="0" class="gmail-cf gmail-gJ"><tbody><tr class="gmail-acZ"><td class="gmail-gF gmail-gK" style="width:368.8px"><table cellpadding="0" class="gmail-cf gmail-ix" style="width:368.8px"><tbody><tr><td><h3 class="gmail-iw"><span name="Hong" class="gmail-gD" style="font-size:12.8px">Hong</span> <span class="gmail-go"><span><</span><a href="mailto:hzhang@mcs.anl.gov">hzhang@mcs.anl.gov</a><span>></span></span></h3></td></tr></tbody></table></td><td class="gmail-gH"><div class="gmail-gK"><span id="gmail-:1ue" class="gmail-g3" title="Mon, Dec 7, 2015 at 12:42 PM" alt="Mon, Dec 7, 2015 at 12:42 PM">12/7/15</span><div class="gmail-zd" tabindex="0" style="outline:0px"><span class="gmail-T-KT"><img class="gmail-f gmail-T-KT-JX" src="https://mail.google.com/mail/u/0/images/cleardot.gif" alt=""></span></div></div></td><td class="gmail-gH"></td><td class="gmail-gH gmail-acX" rowspan="2"><div class="gmail-T-I gmail-J-J5-Ji gmail-T-I-Js-IF gmail-aaq gmail-T-I-ax7 gmail-L3" tabindex="0"><img class="gmail-hB gmail-T-I-J3" src="https://mail.google.com/mail/u/0/images/cleardot.gif" alt=""></div><div id="gmail-:1r4" class="gmail-T-I gmail-J-J5-Ji gmail-T-I-Js-Gs gmail-aap gmail-T-I-awG gmail-T-I-ax7 gmail-L3" tabindex="0"><img class="gmail-hA gmail-T-I-J3" src="https://mail.google.com/mail/u/0/images/cleardot.gif" alt=""></div></td></tr><tr class="gmail-acZ gmail-xD"><td colspan="3"><table cellpadding="0" class="gmail-cf gmail-adz" style="width:448px"><tbody><tr><td class="gmail-ady"><div class="gmail-iw gmail-ajw"><span class="gmail-hb">to <span dir="ltr" name="Danyang" class="gmail-g2">Danyang</span>, <span dir="ltr" name="petsc-maint" class="gmail-g2">petsc-maint</span>, <span dir="ltr" name="PETSc" class="gmail-g2">PETSc</span>, <span dir="ltr" name="Xiaoye" class="gmail-g2">Xiaoye</span></span></div><div class="gmail-ajy"><img class="gmail-ajz" id="gmail-:1r3" src="https://mail.google.com/mail/u/0/images/cleardot.gif" alt=""></div></td></tr></tbody></table></td></tr></tbody></table></div><div class="gmail-qQVYZb"></div><div class="gmail-utdU2e"></div><div class="gmail-tx78Ic"></div><div class="gmail-aHl"></div><div id="gmail-:1r2" tabindex="-1"></div><div id="gmail-:1uc" class="gmail-ii gmail-gt gmail-adP gmail-adO" style="font-size:12.8px"><div id="gmail-:1ub" class="gmail-a3s gmail-aXjCH gmail-m1517dc034049d992"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">Danyang :</div><div class="gmail_quote"><br></div><div class="gmail_quote">Adding '-mat_superlu_dist_fact <span class="gmail-il">SamePattern</span>' fixed the problem. Below is how I figured it out.</div><div class="gmail_quote"><br></div><div class="gmail_quote"><div class="gmail_quote" style="font-size:12.8px">1. Reading ex52f.F, I see '-superlu_default' = '-pc_factor_mat_solver_package superlu_dist', the later enables runtime options for other packages. I use superlu_dist-4.2 and superlu-4.1 for the tests below.</div><div>...</div><div><div class="gmail_quote" style="font-size:12.8px">5.</div><div class="gmail_quote" style="font-size:12.8px">Using a_flow_check_1.bin, I am able to reproduce the error you reported: all packages give correct results except superlu_dist:</div><div class="gmail_quote" style="font-size:12.8px">./ex52f -f0 matrix_and_rhs_bin/a_flow_<wbr>check_1.bin -rhs matrix_and_rhs_bin/b_flow_<wbr>check_168.bin -loop_matrices flow_check -loop_folder matrix_and_rhs_bin -pc_type lu -pc_factor_mat_solver_package superlu_dist</div><div class="gmail_quote" style="font-size:12.8px">Norm of error 2.5970E-12 iterations 1</div><span class="gmail-im" style="font-size:12.8px"><div class="gmail_quote"> -->Test for matrix 168</div></span><div class="gmail_quote" style="font-size:12.8px">Norm of error 1.3936E-01 iterations 34</div><span class="gmail-im" style="font-size:12.8px"><div class="gmail_quote"> -->Test for matrix 169</div><div class="gmail_quote"><br></div></span><div class="gmail_quote" style="font-size:12.8px">I guess the error might come from reuse of matrix factor. Replacing default</div><div class="gmail_quote" style="font-size:12.8px">-mat_superlu_dist_fact <<span class="gmail-il">SamePattern_SameRowPerm</span>> with</div><div class="gmail_quote" style="font-size:12.8px">-mat_superlu_dist_fact <span class="gmail-il">SamePattern</span>, I get</div><div class="gmail_quote" style="font-size:12.8px"><br></div><div class="gmail_quote" style="font-size:12.8px">./ex52f -f0 matrix_and_rhs_bin/a_flow_<wbr>check_1.bin -rhs matrix_and_rhs_bin/b_flow_<wbr>check_168.bin -loop_matrices flow_check -loop_folder matrix_and_rhs_bin -pc_type lu -pc_factor_mat_solver_package superlu_dist -mat_superlu_dist_fact <span class="gmail-il">SamePattern</span></div><div class="gmail_quote" style="font-size:12.8px"><br></div><div class="gmail_quote" style="font-size:12.8px">Norm of error 2.5970E-12 iterations 1</div><span class="gmail-im" style="font-size:12.8px"><div class="gmail_quote"> -->Test for matrix 168</div><div class="gmail_quote">...</div><div class="gmail_quote"><div class="gmail_quote" style="font-size:12.8px">Sherry may tell you why <span class="gmail-il">SamePattern_SameRowPerm</span> cause the difference here.</div><div class="gmail_quote" style="font-size:12.8px">Best on the above experiments, I would set following as default</div><div class="gmail_quote" style="font-size:12.8px">'-mat_superlu_diagpivotthresh 0.0' in petsc/superlu interface.</div><div class="gmail_quote" style="font-size:12.8px">'-mat_superlu_dist_fact <span class="gmail-il">SamePattern</span>' in petsc/superlu_dist interface.</div><div class="gmail_quote" style="font-size:12.8px"><br></div><div class="gmail_quote" style="font-size:12.8px">Hong</div><div class="gmail-yj6qo gmail-ajU" style="font-size:12.8px"></div></div></span></div></div></div></div></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Oct 25, 2016 at 10:38 AM, Hong <span dir="ltr"><<a href="mailto:hzhang@mcs.anl.gov" target="_blank">hzhang@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Anton,<div>I guess, when you reuse matrix and its symbolic factor with updated numerical values, superlu_dist requires this option. I'm cc'ing Sherry to confirm it.</div><div><br></div><div>I'll check petsc/superlu-dist interface to set this flag for this case.</div><span class="HOEnZb"><font color="#888888"><div><br></div></font></span><div><span class="HOEnZb"><font color="#888888">Hong</font></span><div><div class="h5"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Oct 25, 2016 at 8:20 AM, Anton Popov <span dir="ltr"><<a href="mailto:popov@uni-mainz.de" target="_blank">popov@uni-mainz.de</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<p>Hong,</p>
<p>I get all the problems gone and valgrind-clean output if I
specify this:</p>
<p>-mat_superlu_dist_fact SamePattern_SameRowPerm<br>
</p>
What does SamePattern_SameRowPerm actually mean? <br>
Row permutations are for large diagonal, column permutations are for
sparsity, right?<br>
Will it skip subsequent matrix permutations for large diagonal even
if matrix values change significantly?<br>
<br>
Surprisingly everything works even with:<br>
<br>
-mat_superlu_dist_colperm PARMETIS<br>
-mat_superlu_dist_parsymbfact TRUE<br>
<br>
Thanks,<br>
Anton<br>
<br>
<div class="m_-2683123818943114666m_-1905162172435090326moz-cite-prefix">On 10/24/2016 09:06 PM, Hong wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">Anton:
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"><span class="m_-2683123818943114666m_-1905162172435090326gmail-">
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">If replacing
superlu_dist with mumps, does your code work?</div>
</div>
</div>
</blockquote>
</span> yes</div>
</blockquote>
<div> </div>
<div>You may use mumps in your code, or tests different
options for superlu_dist:</div>
<div><br>
</div>
<div>
<div> -mat_superlu_dist_equil: <TRUE> Equilibrate
matrix (None)</div>
<div> -mat_superlu_dist_rowperm <LargeDiag> Row
permutation (choose one of) LargeDiag NATURAL (None)</div>
<div> -mat_superlu_dist_colperm <METIS_AT_PLUS_A>
Column permutation (choose one of) NATURAL MMD_AT_PLUS_A
MMD_ATA METIS_AT_PLUS_A PARMETIS (None)</div>
<div> -mat_superlu_dist_replacetinyp<wbr>ivot: <FALSE>
Replace tiny pivots (None)</div>
<div> -mat_superlu_dist_parsymbfact: <FALSE>
Parallel symbolic factorization (None)</div>
<div> -mat_superlu_dist_fact <SamePattern> Sparsity
pattern for repeated matrix factorization (choose one
of) SamePattern SamePattern_SameRowPerm (None)</div>
</div>
<div><br>
</div>
<div>The options inside <> are defaults. You may try
others. This might help narrow down the bug.</div>
<div><br>
</div>
<div>Hong</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"><span class="m_-2683123818943114666m_-1905162172435090326gmail-"><br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">Hong
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"><span class="m_-2683123818943114666m_-1905162172435090326gmail-m_-8985211249997465386gmail-">
<div class="m_-2683123818943114666m_-1905162172435090326gmail-m_-8985211249997465386gmail-m_4450561754203495905moz-cite-prefix">On
10/24/2016 05:47 PM, Hong wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Barry,
<div>Your change indeed fixed the
error of his testing code.</div>
<div>As Satish tested, on your
branch, ex16 runs smooth.</div>
<div><br>
</div>
<div>I do not understand why on
maint or master branch, ex16
creases inside superlu_dist, but
not with mumps. </div>
<div><br>
</div>
</div>
</blockquote>
<br>
</span> I also confirm that ex16 runs fine
with latest fix, but unfortunately not my
code.<br>
<br>
This is something to be expected, since my
code preallocates once in the beginning.
So there is no way it can be affected by
multiple preallocations. Subsequently I
only do matrix assembly, that makes sure
structure doesn't change (set to get error
otherwise).<br>
<br>
Summary: we don't have a simple test code
to debug superlu issue anymore.<span class="m_-2683123818943114666m_-1905162172435090326gmail-m_-8985211249997465386gmail-HOEnZb"><font color="#888888"><br>
<br>
Anton</font></span><span class="m_-2683123818943114666m_-1905162172435090326gmail-m_-8985211249997465386gmail-"><br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div>Hong<br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon,
Oct 24, 2016 at 9:34 AM,
Satish Balay <span dir="ltr"><<a href="mailto:balay@mcs.anl.gov" target="_blank">balay@mcs.anl.gov</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span>On Mon, 24 Oct 2016, Barry
Smith wrote:<br>
<br>
><br>
</span><span>> > [Or
perhaps Hong is using a
different test code and is
observing bugs<br>
> > with
superlu_dist interface..]<br>
><br>
> She states that
her test does a NEW
MatCreate() for each
matrix load (I cut and
pasted it in the email I
just sent). The bug I
fixed was only related to
using the SAME matrix from
one MatLoad() in another
MatLoad().<br>
<br>
</span>Ah - ok.. Sorry -
wasn't thinking clearly :(<br>
<span class="m_-2683123818943114666m_-1905162172435090326gmail-m_-8985211249997465386gmail-m_4450561754203495905HOEnZb"><font color="#888888"><br>
Satish<br>
</font></span></blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
<br>
</span></div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
</span></div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
</div>
</blockquote></div><br></div></div></div></div></div>
</blockquote></div><br></div>