<div dir="ltr"><br><div class="gmail_extra">On Wed, Dec 26, 2012 at 4:38 PM, Sanjay Govindjee <span dir="ltr"><<a href="mailto:s_g@berkeley.edu" target="_blank">s_g@berkeley.edu</a>></span> wrote:<br><div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I have a macbook pro (Mac OS X 10.7.5)<br>
<br>
% uname -a<br>
Darwin sg-macbook-prolocal.local 11.4.2 Darwin Kernel Version 11.4.2: Thu Aug 23 16:25:48 PDT 2012; root:xnu-1699.32.7~1/RELEASE_<u></u>X86_64 x86_64<br>
<br>
I configured using:<div class="im"><br>
<br>
./configure --with-cc=icc --with-fc=ifort -download-{spooles,parmetis,<u></u>superlu_dist,prometheus,mpich,<u></u>ml,hypre,metis}<br>
<br></div>
so everything was built together.</blockquote><div><br></div><div style>Since</div><div style><br></div><div style> a) you have tried other compilers</div><div style><br></div><div style> b) we cannot reproduce it</div>
<div style><br></div><div style> c) we are building the library during configure</div><div style><br></div><div style>I would guess that some outside library, in your default link path, is contaminating</div><div style>the executable with symbols which override some of those in SuperLU. The SuperLU</div>
<div style>people are not super careful about naming. Could you</div><div style><br></div><div style> 1) Try this same exercise using --with-shared-libraries</div><div style><br></div><div style> 2) Once you do that, use otool -L on the executable so we can see where everything comes from</div>
<div style><br></div><div style> Thanks,</div><div style><br></div><div style> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb">
<div class="h5">
On 12/26/12 1:34 PM, Hong Zhang wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Sanjay:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
hmmm....I guess that is good news -- in that superlu is not broken. However,<br>
for me<br>
not so good news since I seems that there is nasty bug lurking on my<br>
machine.<br>
<br>
Any suggestions on chasing down the error?<br>
</blockquote>
How did you install your supelu_dist with petsc-3.3?<br>
What machine do you use?<br>
<br>
Hong<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
On 12/26/12 1:23 PM, Hong Zhang wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Sanjay:<br>
I get<br>
petsc-3.3/src/ksp/ksp/<u></u>examples/tutorials>mpiexec -n 2 ./ex2<br>
-ksp_monitor_short -ksp_type preonly -pc_type lu<br>
-pc_factor_mat_solver_package superlu_dist -m 500 -n 500<br>
Norm of error 1.92279e-11 iterations 1<br>
<br>
Hong<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I have done some more testing of the problem, continuing with<br>
src/ksp/ksp/examples/<u></u>tutorials/ex2.c.<br>
<br>
The behavior I am seeing is that with smaller problems sizes superlu_dist<br>
is<br>
behaving properly<br>
but with larger problem sizes things seem to go wrong and what goes wrong<br>
is<br>
apparently consistent; the error appears both with my intel build as well<br>
as<br>
with my gcc build.<br>
<br>
I have two run lines:<br>
<br>
runex2superlu:<br>
-@${MPIEXEC} -n 2 ./ex2 -ksp_monitor_short -m 100 -n 100<br>
-ksp_type<br>
preonly -pc_type lu -pc_factor_mat_solver_package superlu_dist<br>
<br>
runex2spooles:<br>
-@${MPIEXEC} -n 2 ./ex2 -ksp_monitor_short -m 100 -n 100<br>
-ksp_type<br>
preonly -pc_type lu -pc_factor_mat_solver_package spooles<br>
<br>
From my intel build, I get<br>
<br>
sg-macbook-prolocal:tutorials sg$ make runex2superlu<br>
Norm of error 7.66145e-13 iterations 1<br>
sg-macbook-prolocal:tutorials sg$ make runex2spooles<br>
Norm of error 2.21422e-12 iterations 1<br>
<br>
From my GCC build, I get<br>
sg-macbook-prolocal:tutorials sg$ make runex2superlu<br>
Norm of error 7.66145e-13 iterations 1<br>
sg-macbook-prolocal:tutorials sg$ make runex2spooles<br>
Norm of error 2.21422e-12 iterations 1<br>
<br>
If I change the -m 100 -n 100 to -m 500 -n 500, I get for my intel build<br>
<br>
sg-macbook-prolocal:tutorials sg$ make runex2superlu<br>
Norm of error 419.953 iterations 1<br>
sg-macbook-prolocal:tutorials sg$ make runex2spooles<br>
Norm of error 2.69468e-10 iterations 1<br>
<br>
From my GCC build with -m 500 -n 500, I get<br>
<br>
sg-macbook-prolocal:tutorials sg$ make runex2superlu<br>
Norm of error 419.953 iterations 1<br>
sg-macbook-prolocal:tutorials sg$ make runex2spooles<br>
Norm of error 2.69468e-10 iterations 1<br>
<br>
<br>
Any suggestions will be greatly appreciated.<br>
<br>
-sanjay<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
On 12/23/12 6:42 PM, Matthew Knepley wrote:<br>
<br>
<br>
On Sun, Dec 23, 2012 at 9:37 PM, Sanjay Govindjee <<a href="mailto:s_g@berkeley.edu" target="_blank">s_g@berkeley.edu</a>><br>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I decided to go with ksp/ksp/exampeles/tutorials/<u></u>ex2.c; I was unsure how<br>
to convert the run lines for snes/examples/ex5.c to work with a direct<br>
solver as I am not versed in SNES options.<br>
<br>
Notwithstanding something strange is happening only on select examples.<br>
With ksp/ksp/exampeles/tutorials/<u></u>ex2.c and the run line:<br>
<br>
-@${MPIEXEC} -n 2 ./ex2 -ksp_monitor_short -m 20 -n 20 -ksp_type preonly<br>
-pc_type lu -pc_factor_mat_solver_package superlu_dist<br>
<br>
I get good results (of the order):<br>
<br>
Norm of error 1.85464e-14 iterations 1<br>
<br>
using both superlu_dist and spooles.<br>
<br>
My BLAS/LAPACK: -llapack -lblas (so native to my machine).<br>
<br>
If you can guide me on a run line for the snes ex5.c I can try that too.<br>
I'll also try to construct a GCC build later to see if that is an issue.<br>
</blockquote>
<br>
Same line on ex5, but ex2 is good enough. However, it will not tell us<br>
anything new. Try another build.<br>
<br>
Matt<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
-sanjay<br>
<br>
<br>
On 12/23/12 5:58 PM, Matthew Knepley wrote:<br>
<br>
On Sun, Dec 23, 2012 at 8:08 PM, Sanjay Govindjee <<a href="mailto:s_g@berkeley.edu" target="_blank">s_g@berkeley.edu</a>><br>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Not sure what you mean by where is your matrix? I am simply running<br>
ex6<br>
in the ksp/examples/tests directory.<br>
<br>
The reason I ran this test is because I was seeing the same behavior<br>
with<br>
my finite element code (on perfectly benign problems).<br>
<br>
Is there a built-in test that you use to check that superlu_dist is<br>
working properly with petsc?<br>
i.e. something you know that works with with petsc 3.3-p5?<br>
</blockquote>
<br>
1) Run it on a SNES ex5 (or KSP ex2), which is a nice Laplacian<br>
<br>
2) Compare with MUMPS<br>
<br>
Matt<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
-sanjay<br>
<br>
<br>
<br>
On 12/23/12 4:56 PM, Jed Brown wrote:<br>
<br>
Where is your matrix? It might be ending up with a very bad pivot. If<br>
the<br>
problem can be reproduced, it should be reported to the SuperLU_DIST<br>
developers to fix. (Note that we do not see this with other matrices.)<br>
You<br>
can also try MUMPS.<br>
<br>
<br>
On Sun, Dec 23, 2012 at 6:48 PM, Sanjay Govindjee <<a href="mailto:s_g@berkeley.edu" target="_blank">s_g@berkeley.edu</a>><br>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I wanted to use SuperLU Dist to perform a direct solve but seem to be<br>
encountering<br>
a problem. I was wonder if this is a know issue and if there is a<br>
solution for it.<br>
<br>
The problem is easily observed using ex6.c in<br>
src/ksp/ksp/examples/tests.<br>
<br>
Out of the box: make runex6 produces a residual error of O(1e-11), all<br>
is well.<br>
<br>
I then changed the run to run on two processors and add the flag<br>
-pc_factor_mat_solver_package spooles this produces a residual error<br>
of<br>
O(1e-11), all is still well.<br>
<br>
I then switch over to -pc_factor_mat_solver_package superlu_dist and<br>
the<br>
residual error comes back as 22.6637! Something seems very wrong.<br>
<br>
My build is perfectly vanilla:<br>
<br>
export PETSC_DIR=/Users/sg/petsc-3.3-<u></u>p5/<br>
export PETSC_ARCH=intel<br>
<br>
./configure --with-cc=icc --with-fc=ifort \<br>
<br>
-download-{spooles,parmetis,<u></u>superlu_dist,prometheus,mpich,<u></u>ml,hypre,metis}<br>
<br>
make PETSC_DIR=/Users/sg/petsc-3.3-<u></u>p5/ PETSC_ARCH=intel all<br>
make PETSC_DIR=/Users/sg/petsc-3.3-<u></u>p5/ PETSC_ARCH=intel test<br>
<br>
-sanjay<br>
</blockquote>
<br>
<br>
--<br>
------------------------------<u></u>-----------------<br>
Sanjay Govindjee, PhD, PE<br>
Professor of Civil Engineering<br>
Vice Chair for Academic Affairs<br>
<br>
779 Davis Hall<br>
Structural Engineering, Mechanics and Materials<br>
Department of Civil Engineering<br>
University of California<br>
Berkeley, CA 94720-1710<br>
<br>
Voice: <a href="tel:%2B1%20510%20642%206060" value="+15106426060" target="_blank">+1 510 642 6060</a><br>
FAX: <a href="tel:%2B1%20510%20643%205264" value="+15106435264" target="_blank">+1 510 643 5264</a><br>
<a href="mailto:s_g@berkeley.edu" target="_blank">s_g@berkeley.edu</a><br>
<a href="http://www.ce.berkeley.edu/~sanjay" target="_blank">http://www.ce.berkeley.edu/~<u></u>sanjay</a><br>
------------------------------<u></u>-----------------<br>
<br>
New Books:<br>
<br>
Engineering Mechanics of Deformable<br>
Solids: A Presentation with Exercises<br>
<br>
<br>
<a href="http://www.oup.com/us/catalog/general/subject/Physics/MaterialsScience/?view=usa&ci=9780199651641" target="_blank">http://www.oup.com/us/catalog/<u></u>general/subject/Physics/<u></u>MaterialsScience/?view=usa&ci=<u></u>9780199651641</a><br>
<a href="http://ukcatalogue.oup.com/product/9780199651641.do" target="_blank">http://ukcatalogue.oup.com/<u></u>product/9780199651641.do</a><br>
<a href="http://amzn.com/0199651647" target="_blank">http://amzn.com/0199651647</a><br>
<br>
<br>
Engineering Mechanics 3 (Dynamics)<br>
<a href="http://www.springer.com/materials/mechanics/book/978-3-642-14018-1" target="_blank">http://www.springer.com/<u></u>materials/mechanics/book/978-<u></u>3-642-14018-1</a><br>
<a href="http://amzn.com/3642140181" target="_blank">http://amzn.com/3642140181</a><br>
<br>
------------------------------<u></u>-----------------<br>
</blockquote>
<br>
<br>
<br>
--<br>
What most experimenters take for granted before they begin their<br>
experiments is infinitely more interesting than any results to which<br>
their<br>
experiments lead.<br>
-- Norbert Wiener<br>
<br>
<br>
</blockquote>
<br>
--<br>
What most experimenters take for granted before they begin their<br>
experiments<br>
is infinitely more interesting than any results to which their<br>
experiments<br>
lead.<br>
-- Norbert Wiener<br>
<br>
<br>
</blockquote></blockquote>
--<br>
------------------------------<u></u>-----------------<br>
Sanjay Govindjee, PhD, PE<br>
Professor of Civil Engineering<br>
Vice Chair for Academic Affairs<br>
<br>
779 Davis Hall<br>
Structural Engineering, Mechanics and Materials<br>
Department of Civil Engineering<br>
University of California<br>
Berkeley, CA 94720-1710<br>
<br>
Voice: <a href="tel:%2B1%20510%20642%206060" value="+15106426060" target="_blank">+1 510 642 6060</a><br>
FAX: <a href="tel:%2B1%20510%20643%205264" value="+15106435264" target="_blank">+1 510 643 5264</a><br>
<a href="mailto:s_g@berkeley.edu" target="_blank">s_g@berkeley.edu</a><br>
<a href="http://www.ce.berkeley.edu/~sanjay" target="_blank">http://www.ce.berkeley.edu/~<u></u>sanjay</a><br>
------------------------------<u></u>-----------------<br>
<br>
New Books:<br>
<br>
Engineering Mechanics of Deformable<br>
Solids: A Presentation with Exercises<br>
<a href="http://www.oup.com/us/catalog/general/subject/Physics/MaterialsScience/?view=usa&ci=9780199651641" target="_blank">http://www.oup.com/us/catalog/<u></u>general/subject/Physics/<u></u>MaterialsScience/?view=usa&ci=<u></u>9780199651641</a><br>
<a href="http://ukcatalogue.oup.com/product/9780199651641.do" target="_blank">http://ukcatalogue.oup.com/<u></u>product/9780199651641.do</a><br>
<a href="http://amzn.com/0199651647" target="_blank">http://amzn.com/0199651647</a><br>
<br>
<br>
Engineering Mechanics 3 (Dynamics)<br>
<a href="http://www.springer.com/materials/mechanics/book/978-3-642-14018-1" target="_blank">http://www.springer.com/<u></u>materials/mechanics/book/978-<u></u>3-642-14018-1</a><br>
<a href="http://amzn.com/3642140181" target="_blank">http://amzn.com/3642140181</a><br>
<br>
------------------------------<u></u>-----------------<br>
<br>
</blockquote></blockquote>
<br>
-- <br>
------------------------------<u></u>-----------------<br>
Sanjay Govindjee, PhD, PE<br>
Professor of Civil Engineering<br>
Vice Chair for Academic Affairs<br>
<br>
779 Davis Hall<br>
Structural Engineering, Mechanics and Materials<br>
Department of Civil Engineering<br>
University of California<br>
Berkeley, CA 94720-1710<br>
<br>
Voice: <a href="tel:%2B1%20510%20642%206060" value="+15106426060" target="_blank">+1 510 642 6060</a><br>
FAX: <a href="tel:%2B1%20510%20643%205264" value="+15106435264" target="_blank">+1 510 643 5264</a><br>
<a href="mailto:s_g@berkeley.edu" target="_blank">s_g@berkeley.edu</a><br>
<a href="http://www.ce.berkeley.edu/~sanjay" target="_blank">http://www.ce.berkeley.edu/~<u></u>sanjay</a><br>
------------------------------<u></u>-----------------<br>
<br>
New Books:<br>
<br>
Engineering Mechanics of Deformable<br>
Solids: A Presentation with Exercises<br>
<a href="http://www.oup.com/us/catalog/general/subject/Physics/MaterialsScience/?view=usa&ci=9780199651641" target="_blank">http://www.oup.com/us/catalog/<u></u>general/subject/Physics/<u></u>MaterialsScience/?view=usa&ci=<u></u>9780199651641</a><br>
<a href="http://ukcatalogue.oup.com/product/9780199651641.do" target="_blank">http://ukcatalogue.oup.com/<u></u>product/9780199651641.do</a><br>
<a href="http://amzn.com/0199651647" target="_blank">http://amzn.com/0199651647</a><br>
<br>
<br>
Engineering Mechanics 3 (Dynamics)<br>
<a href="http://www.springer.com/materials/mechanics/book/978-3-642-14018-1" target="_blank">http://www.springer.com/<u></u>materials/mechanics/book/978-<u></u>3-642-14018-1</a><br>
<a href="http://amzn.com/3642140181" target="_blank">http://amzn.com/3642140181</a><br>
<br>
------------------------------<u></u>-----------------<br>
<br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener
</div></div>