[petsc-users] Using superlu_dist in a direct solve
Matthew Knepley
knepley at gmail.com
Wed Dec 26 20:46:59 CST 2012
On Wed, Dec 26, 2012 at 8:34 PM, Sanjay Govindjee <s_g at berkeley.edu> wrote:
> For what it is worth. I ran the problems with valgrind (before I built
> the --with-shared-libraries version).
> With spooles the run is essentially clean. With superlu I see lots of
> errors of the type:
>
This looks like a well-known MPICH problem with valgrind reporting.
However, these
stacks look strange. You should have source line numbers if this is
compiled with debugging
and you should have the whole stack for MPICH.
Also, why is libquadmath being linked?
Matt
> ==91099== Syscall param writev(vector[...]) points to uninitialised byte(s)
> ==91099== at 0x1245FF2: writev (in
> /usr/lib/system/libsystem_kernel.dylib)
> ==91099== by 0x101209846: MPIDU_Sock_writev (in ./ex2)
> ==91099== by 0x101A2BA23: ???
> ==91099== by 0x1FFFFFFFB: ???
> ==91099== by 0x101A2BA0F: ???
> ==91099== by 0x10852053F: ???
> ==91099== by 0x101A24907: ???
> ==91099== by 0x7FFF5FBFE2DF: ???
> ==91099== by 0x1: ???
> ==91099== by 0x10120AF13: MPIDI_CH3_iSendv (in ./ex2)
> ==91099== Address 0x10712d0c8 is 136 bytes inside a block of size
> 1,661,792 alloc'd
> ==91099== at 0xC713: malloc (vg_replace_malloc.c:271)
> ==91099== by 0x100D5C6DF: superlu_malloc_dist (in ./ex2)
> ==91099== by 0x100D23375: doubleMalloc_dist (in ./ex2)
> ==91099== by 0x100D415C1: pdgstrs (in ./ex2)
> ==91099== by 0x100D3F852: pdgssvx (in ./ex2)
> ==91099== by 0x1007E5D38: MatSolve_SuperLU_DIST (in ./ex2)
> ==91099== by 0x1002BDA1E: MatSolve (in ./ex2)
> ==91099== by 0x1009EAF55: PCApply_LU (in ./ex2)
> ==91099== by 0x100AAE053: PCApply (in ./ex2)
> ==91099== by 0x100B1BCEE: KSPSolve_PREONLY (in ./ex2)
> ==91099== by 0x100B54F55: KSPSolve (in ./ex2)
> ==91099== by 0x1000022FC: main (in ./ex2)
>
>
>
> On 12/26/12 5:24 PM, Sanjay Govindjee wrote:
>
> I have re-configured/built using:
>
> ./configure PETSC_ARCH=gnu_shared
> -download-{spooles,parmetis,superlu_dist,prometheus,mpich,ml,hypre,metis}
> --with-shared-libraries
>
> make PETSC_ARCH=gnu_shared all
>
> make PETSC_ARCH=gnu_shared test
>
>
> Using the same test problem (src/ksp/ksp/examples/tutorials/ex2.c), on the
> 100x100 case I get:
>
> sg-macbook-prolocal:tutorials sg$ make PETSC_ARCH=gnu_shared runex2spooles
> Norm of error 2.21422e-12 iterations 1
> sg-macbook-prolocal:tutorials sg$ make PETSC_ARCH=gnu_shared runex2superlu
> Norm of error 7.66145e-13 iterations 1
>
> One the 500x500 case I get:
>
> sg-macbook-prolocal:tutorials sg$ make PETSC_ARCH=gnu_shared runex2spooles
> Norm of error 2.69468e-10 iterations 1
> sg-macbook-prolocal:tutorials sg$ make PETSC_ARCH=gnu_shared runex2superlu
> Norm of error 419.953 iterations 1
>
> otool shows:
>
> sg-macbook-prolocal:tutorials sg$ otool -L ex2
> ex2:
> /Users/sg/petsc-3.3-p5/gnu_shared/lib/libpetsc.dylib (compatibility
> version 0.0.0, current version 0.0.0)
> /usr/X11/lib/libX11.6.dylib (compatibility version 10.0.0, current
> version 10.0.0)
> /Users/sg/petsc-3.3-p5/gnu_shared/lib/libmpichcxx.dylib (compatibility
> version 0.0.0, current version 3.0.0)
> /usr/local/lib/libstdc++.6.dylib (compatibility version 7.0.0, current
> version 7.17.0)
> /Users/sg/petsc-3.3-p5/gnu_shared/lib/libparmetis.dylib (compatibility
> version 0.0.0, current version 0.0.0)
> /Users/sg/petsc-3.3-p5/gnu_shared/lib/libmetis.dylib (compatibility
> version 0.0.0, current version 0.0.0)
>
> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLAPACK.dylib
> (compatibility version 1.0.0, current version 1.0.0)
>
> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
> (compatibility version 1.0.0, current version 1.0.0)
> /Users/sg/petsc-3.3-p5/gnu_shared/lib/libmpichf90.dylib (compatibility
> version 0.0.0, current version 3.0.0)
> /usr/local/lib/libgfortran.3.dylib (compatibility version 4.0.0,
> current version 4.0.0)
> /usr/local/lib/libquadmath.0.dylib (compatibility version 1.0.0,
> current version 1.0.0)
> /Users/sg/petsc-3.3-p5/gnu_shared/lib/libpmpich.dylib (compatibility
> version 0.0.0, current version 3.0.0)
> /Users/sg/petsc-3.3-p5/gnu_shared/lib/libmpich.dylib (compatibility
> version 0.0.0, current version 3.0.0)
> /Users/sg/petsc-3.3-p5/gnu_shared/lib/libopa.1.dylib (compatibility
> version 2.0.0, current version 2.0.0)
> /Users/sg/petsc-3.3-p5/gnu_shared/lib/libmpl.1.dylib (compatibility
> version 3.0.0, current version 3.0.0)
> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
> version 159.1.0)
> /usr/local/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current
> version 1.0.0)
>
>
>
>
> On 12/26/12 3:08 PM, Matthew Knepley wrote:
>
>
> On Wed, Dec 26, 2012 at 4:38 PM, Sanjay Govindjee <s_g at berkeley.edu>wrote:
>
>> I have a macbook pro (Mac OS X 10.7.5)
>>
>> % uname -a
>> Darwin sg-macbook-prolocal.local 11.4.2 Darwin Kernel Version 11.4.2: Thu
>> Aug 23 16:25:48 PDT 2012; root:xnu-1699.32.7~1/RELEASE_X86_64 x86_64
>>
>> I configured using:
>>
>>
>> ./configure --with-cc=icc --with-fc=ifort
>> -download-{spooles,parmetis,superlu_dist,prometheus,mpich,ml,hypre,metis}
>>
>> so everything was built together.
>
>
> Since
>
> a) you have tried other compilers
>
> b) we cannot reproduce it
>
> c) we are building the library during configure
>
> I would guess that some outside library, in your default link path, is
> contaminating
> the executable with symbols which override some of those in SuperLU. The
> SuperLU
> people are not super careful about naming. Could you
>
> 1) Try this same exercise using --with-shared-libraries
>
> 2) Once you do that, use otool -L on the executable so we can see
> where everything comes from
>
> Thanks,
>
> Matt
>
>
>> On 12/26/12 1:34 PM, Hong Zhang wrote:
>>
>>> Sanjay:
>>>
>>>> hmmm....I guess that is good news -- in that superlu is not broken.
>>>> However,
>>>> for me
>>>> not so good news since I seems that there is nasty bug lurking on my
>>>> machine.
>>>>
>>>> Any suggestions on chasing down the error?
>>>>
>>> How did you install your supelu_dist with petsc-3.3?
>>> What machine do you use?
>>>
>>> Hong
>>>
>>>>
>>>> On 12/26/12 1:23 PM, Hong Zhang wrote:
>>>>
>>>>> Sanjay:
>>>>> I get
>>>>> petsc-3.3/src/ksp/ksp/examples/tutorials>mpiexec -n 2 ./ex2
>>>>> -ksp_monitor_short -ksp_type preonly -pc_type lu
>>>>> -pc_factor_mat_solver_package superlu_dist -m 500 -n 500
>>>>> Norm of error 1.92279e-11 iterations 1
>>>>>
>>>>> Hong
>>>>>
>>>>> I have done some more testing of the problem, continuing with
>>>>>> src/ksp/ksp/examples/tutorials/ex2.c.
>>>>>>
>>>>>> The behavior I am seeing is that with smaller problems sizes
>>>>>> superlu_dist
>>>>>> is
>>>>>> behaving properly
>>>>>> but with larger problem sizes things seem to go wrong and what goes
>>>>>> wrong
>>>>>> is
>>>>>> apparently consistent; the error appears both with my intel build as
>>>>>> well
>>>>>> as
>>>>>> with my gcc build.
>>>>>>
>>>>>> I have two run lines:
>>>>>>
>>>>>> runex2superlu:
>>>>>> -@${MPIEXEC} -n 2 ./ex2 -ksp_monitor_short -m 100 -n 100
>>>>>> -ksp_type
>>>>>> preonly -pc_type lu -pc_factor_mat_solver_package superlu_dist
>>>>>>
>>>>>> runex2spooles:
>>>>>> -@${MPIEXEC} -n 2 ./ex2 -ksp_monitor_short -m 100 -n 100
>>>>>> -ksp_type
>>>>>> preonly -pc_type lu -pc_factor_mat_solver_package spooles
>>>>>>
>>>>>> From my intel build, I get
>>>>>>
>>>>>> sg-macbook-prolocal:tutorials sg$ make runex2superlu
>>>>>> Norm of error 7.66145e-13 iterations 1
>>>>>> sg-macbook-prolocal:tutorials sg$ make runex2spooles
>>>>>> Norm of error 2.21422e-12 iterations 1
>>>>>>
>>>>>> From my GCC build, I get
>>>>>> sg-macbook-prolocal:tutorials sg$ make runex2superlu
>>>>>> Norm of error 7.66145e-13 iterations 1
>>>>>> sg-macbook-prolocal:tutorials sg$ make runex2spooles
>>>>>> Norm of error 2.21422e-12 iterations 1
>>>>>>
>>>>>> If I change the -m 100 -n 100 to -m 500 -n 500, I get for my intel
>>>>>> build
>>>>>>
>>>>>> sg-macbook-prolocal:tutorials sg$ make runex2superlu
>>>>>> Norm of error 419.953 iterations 1
>>>>>> sg-macbook-prolocal:tutorials sg$ make runex2spooles
>>>>>> Norm of error 2.69468e-10 iterations 1
>>>>>>
>>>>>> From my GCC build with -m 500 -n 500, I get
>>>>>>
>>>>>> sg-macbook-prolocal:tutorials sg$ make runex2superlu
>>>>>> Norm of error 419.953 iterations 1
>>>>>> sg-macbook-prolocal:tutorials sg$ make runex2spooles
>>>>>> Norm of error 2.69468e-10 iterations 1
>>>>>>
>>>>>>
>>>>>> Any suggestions will be greatly appreciated.
>>>>>>
>>>>>> -sanjay
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 12/23/12 6:42 PM, Matthew Knepley wrote:
>>>>>>
>>>>>>
>>>>>> On Sun, Dec 23, 2012 at 9:37 PM, Sanjay Govindjee <s_g at berkeley.edu>
>>>>>> wrote:
>>>>>>
>>>>>>> I decided to go with ksp/ksp/exampeles/tutorials/ex2.c; I was unsure
>>>>>>> how
>>>>>>> to convert the run lines for snes/examples/ex5.c to work with a
>>>>>>> direct
>>>>>>> solver as I am not versed in SNES options.
>>>>>>>
>>>>>>> Notwithstanding something strange is happening only on select
>>>>>>> examples.
>>>>>>> With ksp/ksp/exampeles/tutorials/ex2.c and the run line:
>>>>>>>
>>>>>>> -@${MPIEXEC} -n 2 ./ex2 -ksp_monitor_short -m 20 -n 20 -ksp_type
>>>>>>> preonly
>>>>>>> -pc_type lu -pc_factor_mat_solver_package superlu_dist
>>>>>>>
>>>>>>> I get good results (of the order):
>>>>>>>
>>>>>>> Norm of error 1.85464e-14 iterations 1
>>>>>>>
>>>>>>> using both superlu_dist and spooles.
>>>>>>>
>>>>>>> My BLAS/LAPACK: -llapack -lblas (so native to my machine).
>>>>>>>
>>>>>>> If you can guide me on a run line for the snes ex5.c I can try that
>>>>>>> too.
>>>>>>> I'll also try to construct a GCC build later to see if that is an
>>>>>>> issue.
>>>>>>>
>>>>>>
>>>>>> Same line on ex5, but ex2 is good enough. However, it will not tell us
>>>>>> anything new. Try another build.
>>>>>>
>>>>>> Matt
>>>>>>
>>>>>> -sanjay
>>>>>>>
>>>>>>>
>>>>>>> On 12/23/12 5:58 PM, Matthew Knepley wrote:
>>>>>>>
>>>>>>> On Sun, Dec 23, 2012 at 8:08 PM, Sanjay Govindjee <s_g at berkeley.edu>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Not sure what you mean by where is your matrix? I am simply running
>>>>>>>> ex6
>>>>>>>> in the ksp/examples/tests directory.
>>>>>>>>
>>>>>>>> The reason I ran this test is because I was seeing the same behavior
>>>>>>>> with
>>>>>>>> my finite element code (on perfectly benign problems).
>>>>>>>>
>>>>>>>> Is there a built-in test that you use to check that superlu_dist is
>>>>>>>> working properly with petsc?
>>>>>>>> i.e. something you know that works with with petsc 3.3-p5?
>>>>>>>>
>>>>>>>
>>>>>>> 1) Run it on a SNES ex5 (or KSP ex2), which is a nice Laplacian
>>>>>>>
>>>>>>> 2) Compare with MUMPS
>>>>>>>
>>>>>>> Matt
>>>>>>>
>>>>>>> -sanjay
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 12/23/12 4:56 PM, Jed Brown wrote:
>>>>>>>>
>>>>>>>> Where is your matrix? It might be ending up with a very bad pivot.
>>>>>>>> If
>>>>>>>> the
>>>>>>>> problem can be reproduced, it should be reported to the SuperLU_DIST
>>>>>>>> developers to fix. (Note that we do not see this with other
>>>>>>>> matrices.)
>>>>>>>> You
>>>>>>>> can also try MUMPS.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sun, Dec 23, 2012 at 6:48 PM, Sanjay Govindjee <s_g at berkeley.edu
>>>>>>>> >
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I wanted to use SuperLU Dist to perform a direct solve but seem to
>>>>>>>>> be
>>>>>>>>> encountering
>>>>>>>>> a problem. I was wonder if this is a know issue and if there is a
>>>>>>>>> solution for it.
>>>>>>>>>
>>>>>>>>> The problem is easily observed using ex6.c in
>>>>>>>>> src/ksp/ksp/examples/tests.
>>>>>>>>>
>>>>>>>>> Out of the box: make runex6 produces a residual error of O(1e-11),
>>>>>>>>> all
>>>>>>>>> is well.
>>>>>>>>>
>>>>>>>>> I then changed the run to run on two processors and add the flag
>>>>>>>>> -pc_factor_mat_solver_package spooles this produces a residual
>>>>>>>>> error
>>>>>>>>> of
>>>>>>>>> O(1e-11), all is still well.
>>>>>>>>>
>>>>>>>>> I then switch over to -pc_factor_mat_solver_package superlu_dist
>>>>>>>>> and
>>>>>>>>> the
>>>>>>>>> residual error comes back as 22.6637! Something seems very wrong.
>>>>>>>>>
>>>>>>>>> My build is perfectly vanilla:
>>>>>>>>>
>>>>>>>>> export PETSC_DIR=/Users/sg/petsc-3.3-p5/
>>>>>>>>> export PETSC_ARCH=intel
>>>>>>>>>
>>>>>>>>> ./configure --with-cc=icc --with-fc=ifort \
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -download-{spooles,parmetis,superlu_dist,prometheus,mpich,ml,hypre,metis}
>>>>>>>>>
>>>>>>>>> make PETSC_DIR=/Users/sg/petsc-3.3-p5/ PETSC_ARCH=intel all
>>>>>>>>> make PETSC_DIR=/Users/sg/petsc-3.3-p5/ PETSC_ARCH=intel test
>>>>>>>>>
>>>>>>>>> -sanjay
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> -----------------------------------------------
>>>>>>>> Sanjay Govindjee, PhD, PE
>>>>>>>> Professor of Civil Engineering
>>>>>>>> Vice Chair for Academic Affairs
>>>>>>>>
>>>>>>>> 779 Davis Hall
>>>>>>>> Structural Engineering, Mechanics and Materials
>>>>>>>> Department of Civil Engineering
>>>>>>>> University of California
>>>>>>>> Berkeley, CA 94720-1710
>>>>>>>>
>>>>>>>> Voice: +1 510 642 6060 <%2B1%20510%20642%206060>
>>>>>>>> FAX: +1 510 643 5264 <%2B1%20510%20643%205264>
>>>>>>>> s_g at berkeley.edu
>>>>>>>> http://www.ce.berkeley.edu/~sanjay
>>>>>>>> -----------------------------------------------
>>>>>>>>
>>>>>>>> New Books:
>>>>>>>>
>>>>>>>> Engineering Mechanics of Deformable
>>>>>>>> Solids: A Presentation with Exercises
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> http://www.oup.com/us/catalog/general/subject/Physics/MaterialsScience/?view=usa&ci=9780199651641
>>>>>>>> http://ukcatalogue.oup.com/product/9780199651641.do
>>>>>>>> http://amzn.com/0199651647
>>>>>>>>
>>>>>>>>
>>>>>>>> Engineering Mechanics 3 (Dynamics)
>>>>>>>> http://www.springer.com/materials/mechanics/book/978-3-642-14018-1
>>>>>>>> http://amzn.com/3642140181
>>>>>>>>
>>>>>>>> -----------------------------------------------
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> What most experimenters take for granted before they begin their
>>>>>>> experiments is infinitely more interesting than any results to which
>>>>>>> their
>>>>>>> experiments lead.
>>>>>>> -- Norbert Wiener
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>>>>> experiments
>>>>>> is infinitely more interesting than any results to which their
>>>>>> experiments
>>>>>> lead.
>>>>>> -- Norbert Wiener
>>>>>>
>>>>>>
>>>>>> --
>>>> -----------------------------------------------
>>>> Sanjay Govindjee, PhD, PE
>>>> Professor of Civil Engineering
>>>> Vice Chair for Academic Affairs
>>>>
>>>> 779 Davis Hall
>>>> Structural Engineering, Mechanics and Materials
>>>> Department of Civil Engineering
>>>> University of California
>>>> Berkeley, CA 94720-1710
>>>>
>>>> Voice: +1 510 642 6060 <%2B1%20510%20642%206060>
>>>> FAX: +1 510 643 5264 <%2B1%20510%20643%205264>
>>>> s_g at berkeley.edu
>>>> http://www.ce.berkeley.edu/~sanjay
>>>> -----------------------------------------------
>>>>
>>>> New Books:
>>>>
>>>> Engineering Mechanics of Deformable
>>>> Solids: A Presentation with Exercises
>>>>
>>>> http://www.oup.com/us/catalog/general/subject/Physics/MaterialsScience/?view=usa&ci=9780199651641
>>>> http://ukcatalogue.oup.com/product/9780199651641.do
>>>> http://amzn.com/0199651647
>>>>
>>>>
>>>> Engineering Mechanics 3 (Dynamics)
>>>> http://www.springer.com/materials/mechanics/book/978-3-642-14018-1
>>>> http://amzn.com/3642140181
>>>>
>>>> -----------------------------------------------
>>>>
>>>>
>> --
>> -----------------------------------------------
>> Sanjay Govindjee, PhD, PE
>> Professor of Civil Engineering
>> Vice Chair for Academic Affairs
>>
>> 779 Davis Hall
>> Structural Engineering, Mechanics and Materials
>> Department of Civil Engineering
>> University of California
>> Berkeley, CA 94720-1710
>>
>> Voice: +1 510 642 6060 <%2B1%20510%20642%206060>
>> FAX: +1 510 643 5264 <%2B1%20510%20643%205264>
>> s_g at berkeley.edu
>> http://www.ce.berkeley.edu/~sanjay
>> -----------------------------------------------
>>
>> New Books:
>>
>> Engineering Mechanics of Deformable
>> Solids: A Presentation with Exercises
>>
>> http://www.oup.com/us/catalog/general/subject/Physics/MaterialsScience/?view=usa&ci=9780199651641
>> http://ukcatalogue.oup.com/product/9780199651641.do
>> http://amzn.com/0199651647
>>
>>
>> Engineering Mechanics 3 (Dynamics)
>> http://www.springer.com/materials/mechanics/book/978-3-642-14018-1
>> http://amzn.com/3642140181
>>
>> -----------------------------------------------
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
> --
> -----------------------------------------------
> Sanjay Govindjee, PhD, PE
> Professor of Civil Engineering
> Vice Chair for Academic Affairs
>
> 779 Davis Hall
> Structural Engineering, Mechanics and Materials
> Department of Civil Engineering
> University of California
> Berkeley, CA 94720-1710
>
> Voice: +1 510 642 6060
> FAX: +1 510 643 5264s_g at berkeley.eduhttp://www.ce.berkeley.edu/~sanjay
> -----------------------------------------------
>
> New Books:
>
> Engineering Mechanics of Deformable
> Solids: A Presentation with Exerciseshttp://www.oup.com/us/catalog/general/subject/Physics/MaterialsScience/?view=usa&ci=9780199651641http://ukcatalogue.oup.com/product/9780199651641.dohttp://amzn.com/0199651647
>
>
> Engineering Mechanics 3 (Dynamics)http://www.springer.com/materials/mechanics/book/978-3-642-14018-1http://amzn.com/3642140181
>
> -----------------------------------------------
>
>
> --
> -----------------------------------------------
> Sanjay Govindjee, PhD, PE
> Professor of Civil Engineering
> Vice Chair for Academic Affairs
>
> 779 Davis Hall
> Structural Engineering, Mechanics and Materials
> Department of Civil Engineering
> University of California
> Berkeley, CA 94720-1710
>
> Voice: +1 510 642 6060
> FAX: +1 510 643 5264s_g at berkeley.eduhttp://www.ce.berkeley.edu/~sanjay
> -----------------------------------------------
>
> New Books:
>
> Engineering Mechanics of Deformable
> Solids: A Presentation with Exerciseshttp://www.oup.com/us/catalog/general/subject/Physics/MaterialsScience/?view=usa&ci=9780199651641http://ukcatalogue.oup.com/product/9780199651641.dohttp://amzn.com/0199651647
>
>
> Engineering Mechanics 3 (Dynamics)http://www.springer.com/materials/mechanics/book/978-3-642-14018-1http://amzn.com/3642140181
>
> -----------------------------------------------
>
>
--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20121226/ab3a1aed/attachment-0001.html>
More information about the petsc-users
mailing list