<div dir="ltr">Hi Antoine<div><br></div><div>We are also using PETSc for solving adjoint systems resulting from CFD. To get around the matSolveTranspose issue we just assemble the transpose matrix directly and then call KSPSolve(). If this is possible in your application I think it is probably the best approach</div>
<div><br></div><div>Gaetan</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Aug 29, 2014 at 3:58 PM, Antoine De Blois <span dir="ltr"><<a href="mailto:antoine.deblois@aero.bombardier.com" target="_blank">antoine.deblois@aero.bombardier.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello Jed,<br>
<br>
Thank you for your quick response. So I spent some time to dig deeper into my problem. I coded a shell script that passes through a bunch of ksp_type, pc_type and sub_pc_type. So please disregard the comment about the "does not converge properly for transpose". I had taken that conclusion from my own code (and not from the ex10 and extracted matrix), and a KSPSetFromOptions was missing. Apologies for that.<br>
<br>
What remains is the performance issue. The MatSolveTranspose takes a very long time to converge. For a matrix of 3 million rows, MatSolveTranspose takes roughly 5 minutes on 64 cpus, whereas the MatSolve is almost instantaneous!. When I gdb my code, petsc seems to be stalled in the MatLUFactorNumeric_SeqAIJ_Inode () for a long time. I also did a top on the compute node to check the RAM usage. It was hovering over 2 gig, so memory usage does not seem to be an issue here.<br>
<br>
#0 0x00002afe8dfebd08 in MatLUFactorNumeric_SeqAIJ_Inode ()<br>
from /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.5.1/lib/libpetsc.so.3.5<br>
#1 0x00002afe8e07f15c in MatLUFactorNumeric ()<br>
from /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.5.1/lib/libpetsc.so.3.5<br>
#2 0x00002afe8e2afa99 in PCSetUp_ILU ()<br>
from /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.5.1/lib/libpetsc.so.3.5<br>
#3 0x00002afe8e337c0d in PCSetUp ()<br>
from /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.5.1/lib/libpetsc.so.3.5<br>
#4 0x00002afe8e39d643 in KSPSetUp ()<br>
from /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.5.1/lib/libpetsc.so.3.5<br>
#5 0x00002afe8e39e3ee in KSPSolveTranspose ()<br>
from /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.5.1/lib/libpetsc.so.3.5<br>
#6 0x00002afe8e300f8c in PCApplyTranspose_ASM ()<br>
from /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.5.1/lib/libpetsc.so.3.5<br>
#7 0x00002afe8e338c13 in PCApplyTranspose ()<br>
from /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.5.1/lib/libpetsc.so.3.5<br>
#8 0x00002afe8e3a8a84 in KSPInitialResidual ()<br>
from /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.5.1/lib/libpetsc.so.3.5<br>
#9 0x00002afe8e376c32 in KSPSolve_GMRES ()<br>
from /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.5.1/lib/libpetsc.so.3.5<br>
#10 0x00002afe8e39e425 in KSPSolveTranspose ()<br>
<br>
For that particular application, I was using:<br>
ksp_type: gmres<br>
pc_type: asm<br>
sub_pc_type: ilu<br>
adj_sub_pc_factor_levels 1<br>
<br>
For small matrices, the MatSolveTranspose computing time is very similar to the simple MatSolve.<br>
<br>
And if I want to revert to a MatTranspose followed by the MatSolve, then the MatTranspose takes forever to finish... For a matrix of 3 million rows, MatTranspose takes 30 minutes on 64 cpus!!<br>
<br>
So thank you for implementing the transpose solve in superlu_dist. It would also be nice to have it with hypre.<br>
Let me know what you think and ideas on how to improve my computational time,<br>
Regards,<br>
Antoine<br>
<br>
-----Message d'origine-----<br>
De : Jed Brown [mailto:<a href="mailto:jed@jedbrown.org">jed@jedbrown.org</a>]<br>
Envoyé : Thursday, August 28, 2014 5:01 PM<br>
À : Antoine De Blois; '<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>'<br>
Objet : Re: [petsc-users] superlu_dist and MatSolveTranspose<br>
<div><div class="h5"><br>
Antoine De Blois <<a href="mailto:antoine.deblois@aero.bombardier.com">antoine.deblois@aero.bombardier.com</a>> writes:<br>
<br>
> Hello everyone,<br>
><br>
> I am trying to solve a A^T x = b system. For my applications, I had<br>
> realized that the MatSolveTranspose does not converge properly.<br>
<br>
What do you mean "does not converge properly"? Can you send a test case where the transpose solve should be equivalent, but is not? We have only a few tests for transpose solve and not all preconditioners support it, but where it is supported, we want to ensure that it is correct.<br>
<br>
> Therefore, I had implemented a MatTranspose followed by a MatSolve.<br>
> This proved to converge perfectly (which is strange since the<br>
> transposed matrix has the same eigenvalues as the untransposed...).<br>
> The problem is that for bigger matrices, the MatTranspose is very<br>
> costly and thus cannot be used.<br>
<br>
Costly in terms of memory? (I want you to be able to use KSPSolveTranspose, but I'm curious what you're experiencing.)<br>
<br>
> I tried using the superlu_dist package. Although it the package works perfectly for the MatSolve, I get the an "No support for this operation for this object type" error with MatSolveTransopse. I reproduced the error using the MatView an ex10 tutorial. I can provide the matrix and rhs upon request. My command line was:<br>
><br>
> ex10 -f0 A_and_rhs.bin -pc_type lu -pc_factor_mat_solver_package<br>
> superlu_dist -trans<br>
><br>
> So it there an additional parameter I need to use for the transposed solve?<br>
><br>
> [0]PETSC ERROR: --------------------- Error Message<br>
> --------------------------------------------------------------<br>
> [0]PETSC ERROR: No support for this operation for this object type<br>
> [0]PETSC ERROR: Matrix type mpiaij<br>
<br>
This is easy to add. I'll do it now.<br>
<br>
> [0]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.<br>
> [0]PETSC ERROR: Petsc Release Version 3.5.1, unknown [0]PETSC ERROR:<br>
> /gpfs/fs1/aero/SOFTWARE/TOOLS/PROGRAMMING/petsc/src/ksp/ksp/examples/t<br>
> utorials/ex10 on a ARGUS_impi_opt named hpc-user11 by ad007804 Thu Aug<br>
> 28 16:41:15 2014 [0]PETSC ERROR: Configure options --CFLAGS="-xHost<br>
> -axAVX" --download-hypre --download-metis --download-ml<br>
> --download-parmetis --download-scalapack --download-superlu_dist<br>
> --download-mumps --with-c2html=0 --with-cc=mpiicc --with-fc=mpiifort<br>
> --with-cxx=mpiicpc --with-debugging=yes<br>
> --prefix=/gpfs/fs1/aero/SOFTWARE/TOOLS/PROGRAMMING/petsc/petsc-3.5.1<br>
> --with-cmake=/gpfs/fs1/aero/SOFTWARE/TOOLS/CMAKE/cmake-2.8.7/bin/cmake<br>
> --with-valgrind=/gpfs/fs1/aero/SOFTWARE/TOOLS/PROGRAMMING/valgrind-3.9<br>
> .0/bin/valgrind --with-shared-libraries=0 [0]PETSC ERROR: #1<br>
> MatSolveTranspose() line 3473 in<br>
> /gpfs/fs2/aero/SOFTWARE/TOOLS/PROGRAMMING/petsc/src/mat/interface/matr<br>
</div></div>> ix.c [0]PETSC ERROR: #2 PCApplyTranspose_LU() line 214 in<br>
> /gpfs/fs2/aero/SOFTWARE/TOOLS/PROGRAMMING/petsc/src/ksp/pc/impls/facto<br>
> r/lu/lu.c [0]PETSC ERROR: #3 PCApplyTranspose() line 573 in<br>
> /gpfs/fs2/aero/SOFTWARE/TOOLS/PROGRAMMING/petsc/src/ksp/pc/interface/p<br>
> recon.c [0]PETSC ERROR: #4 KSP_PCApply() line 233 in<br>
> /gpfs/fs2/aero/SOFTWARE/TOOLS/PROGRAMMING/petsc/include/petsc-private/<br>
> kspimpl.h [0]PETSC ERROR: #5 KSPInitialResidual() line 63 in<br>
> /gpfs/fs2/aero/SOFTWARE/TOOLS/PROGRAMMING/petsc/src/ksp/ksp/interface/<br>
> itres.c [0]PETSC ERROR: #6 KSPSolve_GMRES() line 234 in<br>
> /gpfs/fs2/aero/SOFTWARE/TOOLS/PROGRAMMING/petsc/src/ksp/ksp/impls/gmre<br>
> s/gmres.c [0]PETSC ERROR: #7 KSPSolveTranspose() line 704 in<br>
> /gpfs/fs2/aero/SOFTWARE/TOOLS/PROGRAMMING/petsc/src/ksp/ksp/interface/<br>
> itfunc.c [0]PETSC ERROR: #8 main() line 324 in<br>
<div class="HOEnZb"><div class="h5">> /gpfs/fs1/aero/SOFTWARE/TOOLS/PROGRAMMING/petsc/src/ksp/ksp/examples/t<br>
> utorials/ex10.c<br>
><br>
> FYI, the transpose solve is a typical application for adjoint optimization. There should be a big adjoint community of developers that try to solve the transposed matrix.<br>
><br>
> Any help is much appreciated,<br>
> Best,<br>
> Antoine<br>
><br>
><br>
> Antoine DeBlois<br>
> Specialiste ingenierie, MDO lead / Engineering Specialist, MDO lead<br>
> Aéronautique / Aerospace <a href="tel:514-855-5001%2C%20x%2050862" value="+15148555001">514-855-5001, x 50862</a><br>
> <a href="mailto:antoine.deblois@aero.bombardier.com">antoine.deblois@aero.bombardier.com</a><mailto:<a href="mailto:antoine.deblois@aero.bombar">antoine.deblois@aero.bombar</a><br>
> <a href="http://dier.com" target="_blank">dier.com</a>><br>
><br>
> 2351 Blvd Alfred-Nobel<br>
> Montreal, Qc<br>
> H4S 1A9<br>
><br>
> [Description : Description :<br>
> <a href="http://signatures.ca.aero.bombardier.net/eom_logo_164x39_fr.jpg" target="_blank">http://signatures.ca.aero.bombardier.net/eom_logo_164x39_fr.jpg</a>]<br>
> CONFIDENTIALITY NOTICE - This communication may contain privileged or confidential information.<br>
> If you are not the intended recipient or received this communication<br>
> by error, please notify the sender and delete the message without<br>
> copying<br>
</div></div></blockquote></div><br></div>