Direct LU solver
Amit.Itagi at seagate.com
Amit.Itagi at seagate.com
Mon Mar 3 13:08:36 CST 2008
Hong,
Your comments were helpful. Now I can do the LU solves multiple times.
However within the program, if I destroy the matrix and the KSP and create
them again (for a different matrix), I get an error in superlu.
Here is the log from a successful run.
[1] 0.249931 Event begin: KSPSetup
[1] 0.24994 Event end: KSPSetup
[1] 0.249948 Event begin: PCSetUp
[1] 0.249963 Event begin: MatGetOrdering
[0] 0.151691 Event begin: KSPSetup
[0] 0.151699 Event end: KSPSetup
[0] PCSetUp(): Setting up new PC
[0] 0.151713 Event begin: PCSetUp
[0] 0.151722 Event begin: MatGetOrdering
[1] PetscCommDuplicate(): returning tag 2147483493
[0] PetscCommDuplicate(): returning tag 2147483493
[1] PetscCommDuplicate(): returning tag 2147483492
[0] PetscCommDuplicate(): returning tag 2147483492
[1] 0.250169 Event end: MatGetOrdering
[1] 0.250182 Event begin: MatLUFactorSym
[0] 0.151885 Event end: MatGetOrdering
[0] 0.151896 Event begin: MatLUFactorSym
[1] PetscCommDuplicate(): returning tag 2147483491
[0] PetscCommDuplicate(): returning tag 2147483491
[1] MatConvert_AIJ_SuperLU_DIST(): Using SuperLU_DIST for SeqAIJ LU factorization and solves.
[0] MatConvert_AIJ_SuperLU_DIST(): Using SuperLU_DIST for SeqAIJ LU factorization and solves.
[1] PetscCommDuplicate(): Using internal PETSc communicator 1083412768 143685848
[1] PetscCommDuplicate(): returning tag 2147483627
[1] PetscCommDuplicate(): Using internal PETSc communicator 1083412768 143685848
[1] PetscCommDuplicate(): returning tag 2147483626
[0] PetscCommDuplicate(): Using internal PETSc communicator 1083412768 144562608
[0] PetscCommDuplicate(): returning tag 2147483627
[0] PetscCommDuplicate(): Using internal PETSc communicator 1083412768 144562608
[0] PetscCommDuplicate(): returning tag 2147483626
[1] 0.252072 Event end: MatLUFactorSym
[1] 0.252098 Event begin: MatLUFactorNum
[0] 0.153821 Event end: MatLUFactorSym
[0] 0.153835 Event begin: MatLUFactorNum
Nonzeros in L 171877
Nonzeros in U 171877
nonzeros in L+U-I 342195
nonzeros in LSUB 52137
Mat conversion(PETSc->SuperLU_DIST) time (max/min/avg):
0.000296116 / 0.00028491 / 0.000290513
[1] 0.623389 Event end: MatLUFactorNum
[1] 0.623425 Event end: PCSetUp
EQUIL time 0.00
ROWPERM time 0.01
COLPERM time 0.01
SYMBFACT time 0.01
DISTRIBUTE time 0.02
FACTOR time 0.29
Factor flops 2.652935e+08 Mflops 924.82
SOLVE time 0.00
[0] 0.525149 Event end: MatLUFactorNum
[0] 0.525167 Event end: PCSetUp
Here is the log from a crash.
[0] 1.41665 Event begin: KSPSetup
[0] 1.41666 Event end: KSPSetup
[0] PCSetUp(): Setting up new PC
[0] 1.41668 Event begin: PCSetUp
[0] 1.41669 Event begin: MatGetOrdering
[1] 1.51505 Event begin: KSPSetup
[1] 1.51506 Event end: KSPSetup
[1] 1.51507 Event begin: PCSetUp
[1] 1.51507 Event begin: MatGetOrdering
[0] PetscCommDuplicate(): returning tag 2147483226
[1] PetscCommDuplicate(): returning tag 2147483226
[0] PetscCommDuplicate(): returning tag 2147483225
[1] PetscCommDuplicate(): returning tag 2147483225
[0] 1.4169 Event end: MatGetOrdering
[0] 1.41692 Event begin: MatLUFactorSym
[1] 1.51526 Event end: MatGetOrdering
[1] 1.51526 Event begin: MatLUFactorSym
[0] PetscCommDuplicate(): returning tag 2147483224
[1] PetscCommDuplicate(): returning tag 2147483224
[0] MatConvert_AIJ_SuperLU_DIST(): Using SuperLU_DIST for SeqAIJ LU factorization and solves.
[1] MatConvert_AIJ_SuperLU_DIST(): Using SuperLU_DIST for SeqAIJ LU factorization and solves.
[0] PetscCommDuplicate(): Using internal PETSc communicator 1083412768 144562608
[0] PetscCommDuplicate(): returning tag 2147483627
[0] PetscCommDuplicate(): Using internal PETSc communicator 1083412768 144562608
[0] PetscCommDuplicate(): returning tag 2147483626
[1] PetscCommDuplicate(): Using internal PETSc communicator 1083412768 143685848
[1] PetscCommDuplicate(): returning tag 2147483627
[1] PetscCommDuplicate(): Using internal PETSc communicator 1083412768 143685848
[1] PetscCommDuplicate(): returning tag 2147483626
[0] 1.4187 Event end: MatLUFactorSym
[0] 1.41872 Event begin: MatLUFactorNum
[1] 1.51706 Event end: MatLUFactorSym
[1] 1.51707 Event begin: MatLUFactorNum
MPI_Alltoallv: invalid datatype argument: Invalid argument (rank 0, comm 5)
Rank (0, MPI_COMM_WORLD): Call stack within LAM:
Rank (0, MPI_COMM_WORLD): - MPI_Alltoallv()
Rank (0, MPI_COMM_WORLD): - main()
If I trace the place of the error in the debugger, the trace gives
KSPSolve() -> KSPSetUp() -> PCSetUp() -> PCSetUp_LU() -> MatLuFactorNumeric() -> MatLuFactorNumeric_SuperLU_Dist()
-> pzgssvx() -> pzCompRow_loc_to_CompCol_global() -> MPI_Alltoallv
I could not make much head-way by looking at the errors. Could you give me some tips on what might be causing this error, and what to look for ?
Thanks
Rgds,
Amit
owner-petsc-users at mcs.anl.gov wrote on 03/03/2008 10:55:21 AM:
>
> Amit,
>
> > Thanks for your response. A couple of follow-up questions -
> >
> > I go over the steps
> >
> > MatCreate()
> > MatSetType(MPIAIJ)
> > MatMPIAIJSetPreallocation()
> > MatSetType(MATSUPERLU_DIST).
> >
> > Even though I want to do the direct LU solves repeatedly (with the same
> > matrix), I don't want the program to do LU factorization repeatedly. I
hope
> > I can get that functionality by using the "SAME_PRECONDITIONER" option
> > (along with -ksppreonly), while defining the KSP. When the program does
the
> > factorization, does it do it in-place or does it allocate new storage ?
>
> allocate new storage unless you call
> MatLUFactor() explicitly.
>
> It is done by
> KSPSetUp()
> which calls PCSetUp() -> PCSetUp_LU()
> -> MatLUFactorSymbolic() & MatLUFactorNumeric(
>
> You can do following for reusing factored L and U:
> KSPCreate()
> KSPSetOperators()
> KSPSetFromOptions()
> KSPSetUp()
> while ( num_rhs-- ) {
> KSPSolve(ksp,b,x);
> }
>
> See src/ksp/ksp/examples/tutorials/ex10.c
>
> > After doing the LU factorization, are the other operations such as
> > MatMult() preserved ?
>
> Yes.
>
> Hong
>
> >
> > Thanks
> >
> > Rgds,
> > Amit
> >
> >
More information about the petsc-users
mailing list