[petsc-users] superlu_dist and MatSolveTranspose

Hong hzhang at mcs.anl.gov
Fri Aug 29 20:13:57 CDT 2014 

We can add MatSolveTranspose() to the petsc interface with superlu_dist.

Jed,
Are you working on it? If not, I can work on it.

Hong

On Fri, Aug 29, 2014 at 6:14 PM, Gaetan Kenway <gaetank at gmail.com> wrote:
> Hi Antoine
>
> We are also using PETSc for solving adjoint systems resulting from CFD. To
> get around the matSolveTranspose issue we just assemble the transpose matrix
> directly and then call KSPSolve(). If this is possible in your application I
> think it is probably the best approach
>
> Gaetan
>
>
> On Fri, Aug 29, 2014 at 3:58 PM, Antoine De Blois
> <antoine.deblois at aero.bombardier.com> wrote:
>>
>> Hello Jed,
>>
>> Thank you for your quick response. So I spent some time to dig deeper into
>> my problem. I coded a shell script that passes through a bunch of ksp_type,
>> pc_type and sub_pc_type. So please disregard the comment about the "does not
>> converge properly for transpose". I had taken that conclusion from my own
>> code (and not from the ex10 and extracted matrix), and a KSPSetFromOptions
>> was missing. Apologies for that.
>>
>> What remains is the performance issue. The MatSolveTranspose takes a very
>> long time to converge. For a matrix of 3 million rows, MatSolveTranspose
>> takes roughly 5 minutes on 64 cpus, whereas the MatSolve is almost
>> instantaneous!. When I gdb my code, petsc seems to be stalled in the
>> MatLUFactorNumeric_SeqAIJ_Inode () for a long time. I also did a top on the
>> compute node to check the RAM usage. It was hovering over 2 gig, so memory
>> usage does not seem to be an issue here.
>>
>> #0  0x00002afe8dfebd08 in MatLUFactorNumeric_SeqAIJ_Inode ()
>>    from
>> /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.5.1/lib/libpetsc.so.3.5
>> #1  0x00002afe8e07f15c in MatLUFactorNumeric ()
>>    from
>> /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.5.1/lib/libpetsc.so.3.5
>> #2  0x00002afe8e2afa99 in PCSetUp_ILU ()
>>    from
>> /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.5.1/lib/libpetsc.so.3.5
>> #3  0x00002afe8e337c0d in PCSetUp ()
>>    from
>> /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.5.1/lib/libpetsc.so.3.5
>> #4  0x00002afe8e39d643 in KSPSetUp ()
>>    from
>> /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.5.1/lib/libpetsc.so.3.5
>> #5  0x00002afe8e39e3ee in KSPSolveTranspose ()
>>    from
>> /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.5.1/lib/libpetsc.so.3.5
>> #6  0x00002afe8e300f8c in PCApplyTranspose_ASM ()
>>    from
>> /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.5.1/lib/libpetsc.so.3.5
>> #7  0x00002afe8e338c13 in PCApplyTranspose ()
>>    from
>> /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.5.1/lib/libpetsc.so.3.5
>> #8  0x00002afe8e3a8a84 in KSPInitialResidual ()
>>    from
>> /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.5.1/lib/libpetsc.so.3.5
>> #9  0x00002afe8e376c32 in KSPSolve_GMRES ()
>>    from
>> /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.5.1/lib/libpetsc.so.3.5
>> #10 0x00002afe8e39e425 in KSPSolveTranspose ()
>>
>> For that particular application, I was using:
>> ksp_type:                       gmres
>> pc_type:                        asm
>> sub_pc_type:                    ilu
>> adj_sub_pc_factor_levels        1
>>
>> For small matrices, the MatSolveTranspose computing time is very similar
>> to the simple MatSolve.
>>
>> And if I want to revert to a MatTranspose followed by the MatSolve, then
>> the MatTranspose takes forever to finish... For a matrix of 3 million rows,
>> MatTranspose takes 30 minutes on 64 cpus!!
>>
>> So thank you for implementing the transpose solve in superlu_dist. It
>> would also be nice to have it with hypre.
>> Let me know what you think and ideas on how to improve my computational
>> time,
>> Regards,
>> Antoine
>>
>> -----Message d'origine-----
>> De : Jed Brown [mailto:jed at jedbrown.org]
>> Envoyé : Thursday, August 28, 2014 5:01 PM
>> À : Antoine De Blois; 'petsc-users at mcs.anl.gov'
>> Objet : Re: [petsc-users] superlu_dist and MatSolveTranspose
>>
>> Antoine De Blois <antoine.deblois at aero.bombardier.com> writes:
>>
>> > Hello everyone,
>> >
>> > I am trying to solve a A^T x = b system. For my applications, I had
>> > realized that the MatSolveTranspose does not converge properly.
>>
>> What do you mean "does not converge properly"?  Can you send a test case
>> where the transpose solve should be equivalent, but is not?  We have only a
>> few tests for transpose solve and not all preconditioners support it, but
>> where it is supported, we want to ensure that it is correct.
>>
>> > Therefore, I had implemented a MatTranspose followed by a MatSolve.
>> > This proved to converge perfectly (which is strange since the
>> > transposed matrix has the same eigenvalues as the untransposed...).
>> > The problem is that for bigger matrices, the MatTranspose is very
>> > costly and thus cannot be used.
>>
>> Costly in terms of memory?  (I want you to be able to use
>> KSPSolveTranspose, but I'm curious what you're experiencing.)
>>
>> > I tried using the superlu_dist package. Although it the package works
>> > perfectly for the MatSolve, I get the an "No support for this operation for
>> > this object type" error  with MatSolveTransopse. I reproduced the error
>> > using the MatView an ex10 tutorial. I can provide the matrix and rhs upon
>> > request. My command line was:
>> >
>> > ex10 -f0 A_and_rhs.bin -pc_type lu -pc_factor_mat_solver_package
>> > superlu_dist -trans
>> >
>> > So it there an additional parameter I need to use for the transposed
>> > solve?
>> >
>> > [0]PETSC ERROR: --------------------- Error Message
>> > --------------------------------------------------------------
>> > [0]PETSC ERROR: No support for this operation for this object type
>> > [0]PETSC ERROR: Matrix type mpiaij
>>
>> This is easy to add.  I'll do it now.
>>
>> > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> > for trouble shooting.
>> > [0]PETSC ERROR: Petsc Release Version 3.5.1, unknown [0]PETSC ERROR:
>> > /gpfs/fs1/aero/SOFTWARE/TOOLS/PROGRAMMING/petsc/src/ksp/ksp/examples/t
>> > utorials/ex10 on a ARGUS_impi_opt named hpc-user11 by ad007804 Thu Aug
>> > 28 16:41:15 2014 [0]PETSC ERROR: Configure options --CFLAGS="-xHost
>> > -axAVX" --download-hypre --download-metis --download-ml
>> > --download-parmetis --download-scalapack --download-superlu_dist
>> > --download-mumps --with-c2html=0 --with-cc=mpiicc --with-fc=mpiifort
>> > --with-cxx=mpiicpc --with-debugging=yes
>> > --prefix=/gpfs/fs1/aero/SOFTWARE/TOOLS/PROGRAMMING/petsc/petsc-3.5.1
>> > --with-cmake=/gpfs/fs1/aero/SOFTWARE/TOOLS/CMAKE/cmake-2.8.7/bin/cmake
>> > --with-valgrind=/gpfs/fs1/aero/SOFTWARE/TOOLS/PROGRAMMING/valgrind-3.9
>> > .0/bin/valgrind --with-shared-libraries=0 [0]PETSC ERROR: #1
>> > MatSolveTranspose() line 3473 in
>> > /gpfs/fs2/aero/SOFTWARE/TOOLS/PROGRAMMING/petsc/src/mat/interface/matr
>> > ix.c [0]PETSC ERROR: #2 PCApplyTranspose_LU() line 214 in
>> > /gpfs/fs2/aero/SOFTWARE/TOOLS/PROGRAMMING/petsc/src/ksp/pc/impls/facto
>> > r/lu/lu.c [0]PETSC ERROR: #3 PCApplyTranspose() line 573 in
>> > /gpfs/fs2/aero/SOFTWARE/TOOLS/PROGRAMMING/petsc/src/ksp/pc/interface/p
>> > recon.c [0]PETSC ERROR: #4 KSP_PCApply() line 233 in
>> > /gpfs/fs2/aero/SOFTWARE/TOOLS/PROGRAMMING/petsc/include/petsc-private/
>> > kspimpl.h [0]PETSC ERROR: #5 KSPInitialResidual() line 63 in
>> > /gpfs/fs2/aero/SOFTWARE/TOOLS/PROGRAMMING/petsc/src/ksp/ksp/interface/
>> > itres.c [0]PETSC ERROR: #6 KSPSolve_GMRES() line 234 in
>> > /gpfs/fs2/aero/SOFTWARE/TOOLS/PROGRAMMING/petsc/src/ksp/ksp/impls/gmre
>> > s/gmres.c [0]PETSC ERROR: #7 KSPSolveTranspose() line 704 in
>> > /gpfs/fs2/aero/SOFTWARE/TOOLS/PROGRAMMING/petsc/src/ksp/ksp/interface/
>> > itfunc.c [0]PETSC ERROR: #8 main() line 324 in
>> > /gpfs/fs1/aero/SOFTWARE/TOOLS/PROGRAMMING/petsc/src/ksp/ksp/examples/t
>> > utorials/ex10.c
>> >
>> > FYI, the transpose solve is a typical application for adjoint
>> > optimization. There should be a big adjoint community of developers that try
>> > to solve the transposed matrix.
>> >
>> > Any help is much appreciated,
>> > Best,
>> > Antoine
>> >
>> >
>> > Antoine DeBlois
>> > Specialiste ingenierie, MDO lead / Engineering Specialist, MDO lead
>> > Aéronautique / Aerospace 514-855-5001, x 50862
>> > antoine.deblois at aero.bombardier.com<mailto:antoine.deblois at aero.bombar
>> > dier.com>
>> >
>> > 2351 Blvd Alfred-Nobel
>> > Montreal, Qc
>> > H4S 1A9
>> >
>> > [Description : Description :
>> > http://signatures.ca.aero.bombardier.net/eom_logo_164x39_fr.jpg]
>> > CONFIDENTIALITY NOTICE - This communication may contain privileged or
>> > confidential information.
>> > If you are not the intended recipient or received this communication
>> > by error, please notify the sender and delete the message without
>> > copying
>
>

More information about the petsc-users mailing list