<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Ok, Thank you.<div>I didn't know about MatCreateNormal.</div><div><br></div><div>In terms of computer performance, what is best to solve Ax=b with A rectangular?</div><div>Is it to keep A rectangular and use KSPLSQR along with PCNONE or </div><div>to convert to normal equations using MatCreateNormal and use another ksp type with another pc type? </div><div><br></div><div>In the future, our A will be very sparse and will be something like up to 100 millions lines and 10 millions columns in size.</div><div><br></div><div><div>I will study all that.<br></div></div><div><br></div><div>Fuji</div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Sep 26, 2022 at 11:45 AM Pierre Jolivet <<a href="mailto:pierre@joliv.et">pierre@joliv.et</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;">I’m sorry, solving overdetermined systems, alongside (overlapping) domain decomposition preconditioners and solving systems with multiple right-hand sides, is one of the topic for which I need to stop pushing new features and update the users manual instead…<div>The very basic documentation of PCQR is here: <a href="https://petsc.org/main/docs/manualpages/PC/PCQR" target="_blank">https://petsc.org/main/docs/manualpages/PC/PCQR</a> (I’m guessing you are using the release documentation in which it’s indeed not present).</div><div>Some comments about your problem of solving Ax=b with a rectangular matrix A.</div><div>1) if you switch to KSPLSQR, it is wrong to use KSPSetOperators(ksp, A, A).</div><div>You can get away with murder if your PCType is PCNONE, but otherwise, you should always supply the normal equations as the Pmat (you will get runtime errors otherwise).</div><div>To get the normal equations, you can use <a href="https://petsc.org/main/docs/manualpages/Mat/MatCreateNormal/" target="_blank">https://petsc.org/main/docs/manualpages/Mat/MatCreateNormal/</a></div><div>The following two points only applies if your Pmat is sparse (or sparse with some dense rows).</div><div>2) there are a couple of PC that handle MATNORMAL: PCNONE, PCQR, PCJACOBI, PCBJACOBI, PCASM, PCHPDDM</div><div>Currently, PCQR needs SuiteSparse, and thus runs only if the Pmat is distributed on a single MPI process (if your Pmat is distributed on multiple processes, you should first use PCREDUNDANT).</div><div>3) if you intend to make your problem scale in sizes, most of these preconditioners will not be very robust.</div><div>In that case, if your problem does not have any dense rows, you should either use PCHPDDM or MatConvert(Pmat, MATAIJ, PmatAIJ) and then use PCCHOLESKY, PCHYPRE or PCGAMG (you can have a look at <a href="https://epubs.siam.org/doi/abs/10.1137/21M1434891" target="_blank">https://epubs.siam.org/doi/abs/10.1137/21M1434891</a> for a comparison)</div><div>If your problem has dense rows, I have somewhere the code to recast it into an augmented system then solved using PCFIELDSPLIT (see <a href="https://link.springer.com/article/10.1007/s11075-018-0478-2" target="_blank">https://link.springer.com/article/10.1007/s11075-018-0478-2</a>). I can send it to you if needed.</div><div>Let me know if I can be of further assistance or if something is not clear to you.</div><div><br></div><div>Thanks,</div><div>Pierre</div><div><div><br><blockquote type="cite"><div>On 26 Sep 2022, at 10:56 AM, fujisan <<a href="mailto:fujisan43@gmail.com" target="_blank">fujisan43@gmail.com</a>> wrote:</div><br><div><div dir="ltr">OK thank you.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Sep 26, 2022 at 10:53 AM Jose E. Roman <<a href="mailto:jroman@dsic.upv.es" target="_blank">jroman@dsic.upv.es</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">The QR factorization from SuiteSparse is sequential only, cannot be used in parallel.<br>
In parallel you can try PCBJACOBI with a PCQR local preconditioner.<br>
Pierre may have additional suggestions.<br>
<br>
Jose<br>
<br>
<br>
> El 26 sept 2022, a las 10:47, fujisan <<a href="mailto:fujisan43@gmail.com" target="_blank">fujisan43@gmail.com</a>> escribió:<br>
> <br>
> I did configure Petsc with the option --download-suitesparse.<br>
> <br>
> The error is more like this:<br>
> PETSC ERROR: Could not locate a solver type for factorization type QR and matrix type mpiaij.<br>
> <br>
> Fuji<br>
> <br>
> On Mon, Sep 26, 2022 at 10:25 AM Jose E. Roman <<a href="mailto:jroman@dsic.upv.es" target="_blank">jroman@dsic.upv.es</a>> wrote:<br>
> If the error message is "Could not locate a solver type for factorization type QR" then you should configure PETSc with --download-suitesparse<br>
> <br>
> Jose<br>
> <br>
> <br>
> > El 26 sept 2022, a las 9:06, fujisan <<a href="mailto:fujisan43@gmail.com" target="_blank">fujisan43@gmail.com</a>> escribió:<br>
> > <br>
> > Thank you Pierre,<br>
> > <br>
> > I used PCNONE along with KSPLSQR and it worked.<br>
> > But as for PCQR, it cannot be found. There is nothing about it in the documentation as well.<br>
> > <br>
> > Fuji<br>
> > <br>
> > On Wed, Sep 21, 2022 at 12:20 PM Pierre Jolivet <<a href="mailto:pierre@joliv.et" target="_blank">pierre@joliv.et</a>> wrote:<br>
> > Yes, but you need to use a KSP that handles rectangular Mat, such as KSPLSQR (-ksp_type lsqr).<br>
> > PCLU does not handle rectangular Pmat. The only PC that handle rectangular Pmat are PCQR, PCNONE.<br>
> > If you supply the normal equations as the Pmat for LSQR, then you can use “standard” PC.<br>
> > You can have a look at <a href="https://petsc.org/main/src/ksp/ksp/tutorials/ex27.c.html" rel="noreferrer" target="_blank">https://petsc.org/main/src/ksp/ksp/tutorials/ex27.c.html</a> that covers most of these cases.<br>
> > <br>
> > Thanks,<br>
> > Pierre<br>
> > <br>
> > (sorry for the earlier answer sent wrongfully to petsc-maint, please discard the previous email)<br>
> > <br>
> >> On 21 Sep 2022, at 10:03 AM, fujisan <<a href="mailto:fujisan43@gmail.com" target="_blank">fujisan43@gmail.com</a>> wrote:<br>
> >> <br>
> >> I'm trying to solve Ax=b with a sparse rectangular matrix A (of size 33x17 in my test) using<br>
> >> options '-ksp_type stcg -pc_type lu' on 1 or 2 cpus.<br>
> >> <br>
> >> And I always get an error saying "Incompatible vector local lengths" (see below).<br>
> >> <br>
> >> Here is the relevant lines of my code:<br>
> >> <br>
> >> program test<br>
> >> ...<br>
> >> ! Variable declarations<br>
> >> <br>
> >> PetscCallA(PetscInitialize(PETSC_NULL_CHARACTER,ierr))<br>
> >> <br>
> >> PetscCall(MatCreate(PETSC_COMM_WORLD,A,ierr))<br>
> >> PetscCall(MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,m,n,ierr))<br>
> >> PetscCall(MatSetType(A,MATMPIAIJ,ierr))<br>
> >> PetscCall(MatSetFromOptions(A,ierr))<br>
> >> PetscCall(MatSetUp(A,ierr))<br>
> >> PetscCall(MatGetOwnershipRange(A,istart,iend,ierr))<br>
> >> <br>
> >> do irow=istart,iend-1<br>
> >> ... Reading from file ...<br>
> >> PetscCall(MatSetValues(A,1,irow,nzv,col,val,ADD_VALUES,ierr))<br>
> >> ...<br>
> >> enddo<br>
> >> <br>
> >> PetscCall(MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr))<br>
> >> PetscCall(MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr))<br>
> >> <br>
> >> ! Creating vectors x and b<br>
> >> PetscCallA(MatCreateVecs(A,x,b,ierr))<br>
> >> <br>
> >> ! Duplicating x in u.<br>
> >> PetscCallA(VecDuplicate(x,u,ierr))<br>
> >> <br>
> >> ! u is used to calculate b<br>
> >> PetscCallA(VecSet(u,1.0,ierr))<br>
> >> <br>
> >> PetscCallA(VecAssemblyBegin(u,ierr))<br>
> >> PetscCallA(VecAssemblyEnd(u,ierr))<br>
> >> <br>
> >> ! Calculating Au = b<br>
> >> PetscCallA(MatMult(A,u,b,ierr)) ! A.u = b<br>
> >> <br>
> >> PetscCallA(KSPSetType(ksp,KSPCG,ierr))<br>
> >> <br>
> >> PetscCallA(KSPSetOperators(ksp,A,A,ierr))<br>
> >> <br>
> >> PetscCallA(KSPSetFromOptions(ksp,ierr))<br>
> >> <br>
> >> ! Solving Ax = b, x unknown<br>
> >> PetscCallA(KSPSolve(ksp,b,x,ierr))<br>
> >> <br>
> >> PetscCallA(VecDestroy(x,ierr))<br>
> >> PetscCallA(VecDestroy(u,ierr))<br>
> >> PetscCallA(VecDestroy(b,ierr))<br>
> >> PetscCallA(MatDestroy(A,ierr))<br>
> >> PetscCallA(KSPDestroy(ksp,ierr))<br>
> >> <br>
> >> call PetscFinalize(ierr)<br>
> >> end program<br>
> >> <br>
> >> The code reads a sparse matrix from a binary file.<br>
> >> I also output the sizes of matrix A and vectors b, x, u.<br>
> >> They all seem consistent.<br>
> >> <br>
> >> What am I doing wrong?<br>
> >> Is it possible to solve Ax=b with A rectangular?<br>
> >> <br>
> >> Thank you in advance for your help.<br>
> >> Have a nice day.<br>
> >> <br>
> >> Fuji<br>
> >> <br>
> >> Matrix size : m= 33 n= 17 cpu size: 1<br>
> >> Size of matrix A : 33 17<br>
> >> Size of vector b : 33<br>
> >> Size of vector x : 17<br>
> >> Size of vector u : 17<br>
> >> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<br>
> >> [0]PETSC ERROR: Arguments are incompatible<br>
> >> [0]PETSC ERROR: Incompatible vector local lengths parameter # 1 local size 33 != parameter # 2 local size 17<br>
> >> [0]PETSC ERROR: See <a href="https://petsc.org/release/faq/" rel="noreferrer" target="_blank">https://petsc.org/release/faq/</a> for trouble shooting.<br>
> >> [0]PETSC ERROR: Petsc Development GIT revision: v3.17.4-1341-g91b2b62a00 GIT Date: 2022-09-15 19:26:07 +0000<br>
> >> [0]PETSC ERROR: ./bin/solve on a x86_64 named master by fujisan Tue Sep 20 16:56:37 2022<br>
> >> [0]PETSC ERROR: Configure options --with-petsc-arch=x86_64 --COPTFLAGS="-g -O3" --FOPTFLAGS="-g -O3" --CXXOPTFLAGS="-g -O3" --with-debugging=0 --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-single-library=1 --with-mpiexec=mpiexec --with-precision=double --with-fortran-interfaces=1 --with-make=1 --with-mpi=1 --with-mpi-compilers=1 --download-fblaslapack=0 --download-hypre=1 --download-cmake=0 --with-cmake=1 --download-metis=1 --download-parmetis=1 --download-ptscotch=0 --download-suitesparse=1 --download-triangle=1 --download-superlu=1 --download-superlu_dist=1 --download-scalapack=1 --download-mumps=1 --download-elemental=1 --download-spai=0 --download-parms=1 --download-moab=1 --download-chaco=0 --download-fftw=1 --with-petsc4py=1 --download-mpi4py=1 --download-saws --download-concurrencykit=1 --download-revolve=1 --download-cams=1 --download-p4est=0 --with-zlib=1 --download-mfem=1 --download-glvis=0 --with-opengl=0 --download-libpng=1 --download-libjpeg=1 --download-slepc=1 --download-hpddm=1 --download-bamg=1 --download-mmg=0 --download-parmmg=0 --download-htool=1 --download-egads=0 --download-opencascade=0 PETSC_ARCH=x86_64<br>
> >> [0]PETSC ERROR: #1 VecCopy() at /data/softs/petsc/src/vec/vec/interface/vector.c:1607<br>
> >> [0]PETSC ERROR: #2 KSPSolve_BiCG() at /data/softs/petsc/src/ksp/ksp/impls/bicg/bicg.c:40<br>
> >> [0]PETSC ERROR: #3 KSPSolve_Private() at /data/softs/petsc/src/ksp/ksp/interface/itfunc.c:877<br>
> >> [0]PETSC ERROR: #4 KSPSolve() at /data/softs/petsc/src/ksp/ksp/interface/itfunc.c:1048<br>
> >> [0]PETSC ERROR: #5 solve.F90:218<br>
> >> Abort(75) on node 0 (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 75) - process 0<br>
> >> <br>
> > <br>
> <br>
<br>
</blockquote></div>
</div></blockquote></div><br></div></div></blockquote></div>