[petsc-users] Using superlu_dist in a direct solve

Wed Dec 26 15:23:33 CST 2012

Sanjay:
I get
petsc-3.3/src/ksp/ksp/examples/tutorials>mpiexec -n 2 ./ex2
-ksp_monitor_short  -ksp_type preonly -pc_type lu
-pc_factor_mat_solver_package superlu_dist -m 500 -n 500
Norm of error 1.92279e-11 iterations 1

Hong

> I have done some more testing of the problem, continuing with
> src/ksp/ksp/examples/tutorials/ex2.c.
>
> The behavior I am seeing is that with smaller problems sizes superlu_dist is
> behaving properly
> but with larger problem sizes things seem to go wrong and what goes wrong is
> apparently consistent; the error appears both with my intel build as well as
> with my gcc build.
>
> I have two run lines:
>
> runex2superlu:
>         -@${MPIEXEC} -n 2 ./ex2 -ksp_monitor_short -m 100 -n 100 -ksp_type
> preonly -pc_type lu -pc_factor_mat_solver_package superlu_dist
>
> runex2spooles:
>         -@${MPIEXEC} -n 2 ./ex2 -ksp_monitor_short -m 100 -n 100 -ksp_type
> preonly -pc_type lu -pc_factor_mat_solver_package spooles
>
> From my intel build, I get
>
> sg-macbook-prolocal:tutorials sg$ make runex2superlu
> Norm of error 7.66145e-13 iterations 1
> sg-macbook-prolocal:tutorials sg$ make runex2spooles
> Norm of error 2.21422e-12 iterations 1
>
> From my GCC build, I get
> sg-macbook-prolocal:tutorials sg$ make runex2superlu
> Norm of error 7.66145e-13 iterations 1
> sg-macbook-prolocal:tutorials sg$ make runex2spooles
> Norm of error 2.21422e-12 iterations 1
>
> If I change the -m 100 -n 100 to -m 500 -n 500, I get for my intel build
>
> sg-macbook-prolocal:tutorials sg$ make runex2superlu
> Norm of error 419.953 iterations 1
> sg-macbook-prolocal:tutorials sg$ make runex2spooles
> Norm of error 2.69468e-10 iterations 1
>
> From my GCC build with -m 500 -n 500, I get
>
> sg-macbook-prolocal:tutorials sg$ make runex2superlu
> Norm of error 419.953 iterations 1
> sg-macbook-prolocal:tutorials sg$ make runex2spooles
> Norm of error 2.69468e-10 iterations 1
>
>
> Any suggestions will be greatly appreciated.
>
> -sanjay
>
>
>
>
>
>
>
> On 12/23/12 6:42 PM, Matthew Knepley wrote:
>
>
> On Sun, Dec 23, 2012 at 9:37 PM, Sanjay Govindjee <s_g at berkeley.edu> wrote:
>>
>> I decided to go with ksp/ksp/exampeles/tutorials/ex2.c; I was unsure how
>> to convert the run lines for snes/examples/ex5.c to work with a direct
>> solver as I am not versed in SNES options.
>>
>> Notwithstanding something strange is happening only on select examples.
>> With ksp/ksp/exampeles/tutorials/ex2.c and the run line:
>>
>> -@${MPIEXEC} -n 2 ./ex2 -ksp_monitor_short -m 20 -n 20 -ksp_type preonly
>> -pc_type lu -pc_factor_mat_solver_package superlu_dist
>>
>> I get good results (of the order):
>>
>> Norm of error 1.85464e-14 iterations 1
>>
>> using both superlu_dist and spooles.
>>
>> My BLAS/LAPACK: -llapack -lblas (so native to my machine).
>>
>> If you can guide me on a run line for the snes ex5.c I can try that too.
>> I'll also try to construct a GCC build later to see if that is an issue.
>
>
> Same line on ex5, but ex2 is good enough. However, it will not tell us
> anything new. Try another build.
>
>    Matt
>
>>
>> -sanjay
>>
>>
>> On 12/23/12 5:58 PM, Matthew Knepley wrote:
>>
>> On Sun, Dec 23, 2012 at 8:08 PM, Sanjay Govindjee <s_g at berkeley.edu>
>> wrote:
>>>
>>> Not sure what you mean by where is your matrix?  I am simply running ex6
>>> in the ksp/examples/tests directory.
>>>
>>> The reason I ran this test is because I was seeing the same behavior with
>>> my finite element code (on perfectly benign problems).
>>>
>>> Is there a built-in test that you use to check that superlu_dist is
>>> working properly with petsc?
>>> i.e. something you know that works with with petsc 3.3-p5?
>>
>>
>> 1) Run it on a SNES ex5 (or KSP ex2), which is a nice Laplacian
>>
>> 2) Compare with MUMPS
>>
>>    Matt
>>
>>>
>>> -sanjay
>>>
>>>
>>>
>>> On 12/23/12 4:56 PM, Jed Brown wrote:
>>>
>>> Where is your matrix? It might be ending up with a very bad pivot. If the
>>> problem can be reproduced, it should be reported to the SuperLU_DIST
>>> developers to fix. (Note that we do not see this with other matrices.) You
>>> can also try MUMPS.
>>>
>>>
>>> On Sun, Dec 23, 2012 at 6:48 PM, Sanjay Govindjee <s_g at berkeley.edu>
>>> wrote:
>>>>
>>>> I wanted to use SuperLU Dist to perform a direct solve but seem to be
>>>> encountering
>>>> a problem.  I was wonder if this is a know issue and if there is a
>>>> solution for it.
>>>>
>>>> The problem is easily observed using ex6.c in
>>>> src/ksp/ksp/examples/tests.
>>>>
>>>> Out of the box: make runex6 produces a residual error of O(1e-11), all
>>>> is well.
>>>>
>>>> I then changed the run to run on two processors and add the flag
>>>> -pc_factor_mat_solver_package spooles  this produces a residual error of
>>>> O(1e-11), all is still well.
>>>>
>>>> I then switch over to -pc_factor_mat_solver_package superlu_dist and the
>>>> residual error comes back as 22.6637!  Something seems very wrong.
>>>>
>>>> My build is perfectly vanilla:
>>>>
>>>> export PETSC_DIR=/Users/sg/petsc-3.3-p5/
>>>> export PETSC_ARCH=intel
>>>>
>>>> ./configure --with-cc=icc --with-fc=ifort  \
>>>> -download-{spooles,parmetis,superlu_dist,prometheus,mpich,ml,hypre,metis}
>>>>
>>>> make PETSC_DIR=/Users/sg/petsc-3.3-p5/ PETSC_ARCH=intel all
>>>> make PETSC_DIR=/Users/sg/petsc-3.3-p5/ PETSC_ARCH=intel test
>>>>
>>>> -sanjay
>>>
>>>
>>>
>>> --
>>> -----------------------------------------------
>>> Sanjay Govindjee, PhD, PE
>>> Professor of Civil Engineering
>>> Vice Chair for Academic Affairs
>>>
>>> 779 Davis Hall
>>> Structural Engineering, Mechanics and Materials
>>> Department of Civil Engineering
>>> University of California
>>> Berkeley, CA 94720-1710
>>>
>>> Voice:  +1 510 642 6060
>>> FAX:    +1 510 643 5264
>>> s_g at berkeley.edu
>>> http://www.ce.berkeley.edu/~sanjay
>>> -----------------------------------------------
>>>
>>> New Books:
>>>
>>> Engineering Mechanics of Deformable
>>> Solids: A Presentation with Exercises
>>>
>>> http://www.oup.com/us/catalog/general/subject/Physics/MaterialsScience/?view=usa&ci=9780199651641
>>> http://ukcatalogue.oup.com/product/9780199651641.do
>>> http://amzn.com/0199651647
>>>
>>>
>>> Engineering Mechanics 3 (Dynamics)
>>> http://www.springer.com/materials/mechanics/book/978-3-642-14018-1
>>> http://amzn.com/3642140181
>>>
>>> -----------------------------------------------
>>
>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>>
>
>
>
> --
> What most experimenters take for granted before they begin their experiments
> is infinitely more interesting than any results to which their experiments
> lead.
> -- Norbert Wiener
>
>