<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><br id="lineBreakAtBeginningOfMessage"><div><br><blockquote type="cite"><div>On 4 Oct 2024, at 8:50 AM, Edoardo alinovi <edoardo.alinovi@gmail.com> wrote:</div><br class="Apple-interchange-newline"><div><div dir="ltr"><div dir="ltr"><div><div dir="ltr" class="gmail_signature"><div dir="ltr"><div dir="ltr"><div>Jed, Junchao,</div><div><br></div><div>Thanks for getting me back! </div><div> </div><div>Junchao, I cannot provide you with an input file as I am plugging just part of that code into mine. I am building blocks of the full matrix on my own, not reading from the file unfortunately. I can share a branch of my code with a 3x3 cavity toy case, but that would require compiling it and it might be a bit inconvenient for you. </div><div><br></div><div>Jed, I am using fieldsplit indeed, it's just the matrix that is assembled in a 2x2 block form. Here is the full backtrace:</div><div><br></div></div></div></div></div></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: No support for this operation for this object type</i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: No method getinfo for Mat of type nest</i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc!</i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: Option left: name:-UPeqn_fieldsplit_p_pc_type value: hypre source: command line</i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: Option left: name:-UPeqn_fieldsplit_u_pc_type value: bjacobi source: command line</i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: See <a href="https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!a6ejiT9ML4BpeAAcqVuIMX-J7sA75OiyYcHeUd6-INXK8sR-gcQPA3ndqsgpzphK4tIBnE13dbzycfmUkwST7aXF6oxXipY$">https://petsc.org/release/faq/</a> for trouble shooting.</i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: Petsc Release Version 3.20.4, unknown </i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: flubio_coupled on a linux_x86_64 named localhost.localdomain by edo Thu Oct 3 14:17:20 2024</i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: Configure options PETSC_ARCH=linux_x86_64 FOPTFLAGS=-O3 COPTFLAGS=-O3 CXXOPTFLAGS=-O3 -with-debugging=no -download-fblaslapack=1 -download-superlu_dist -download-mumps -download-hypre -download-metis -download-parmetis -download-scalapack -download-ml -download-slepc -download-hpddm -download-cmake -with-mpi-dir=/home/edo/software_repo/openmpi-5.0.1/build/</i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: #1 MatGetInfo() at /home/edo/software_repo/petsc/src/mat/interface/matrix.c:3005</i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: #2 PCSetUp_GAMG() at /home/edo/software_repo/petsc/src/ksp/pc/impls/gamg/gamg.c:619</i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: #3 PCSetUp() at /home/edo/software_repo/petsc/src/ksp/pc/interface/precon.c:1080</i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: It appears a new error in the code was triggered after a previous error, possibly because:</i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: - The first error was not properly handled via (for example) the use of</i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: PetscCall(TheFunctionThatErrors()); or</i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: - The second error was triggered while handling the first error.</i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: Above is the traceback for the previous unhandled error, below the traceback for the next error</i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: ALL ERRORS in the PETSc libraries are fatal, you should add the appropriate error checking to the code</i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: No support for this operation for this object type</i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: No method getinfo for Mat of type nest</i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc!</i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: Option left: name:-UPeqn_fieldsplit_p_pc_type value: hypre source: command line</i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: Option left: name:-UPeqn_fieldsplit_u_pc_type value: bjacobi source: command line</i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: See <a href="https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!a6ejiT9ML4BpeAAcqVuIMX-J7sA75OiyYcHeUd6-INXK8sR-gcQPA3ndqsgpzphK4tIBnE13dbzycfmUkwST7aXF6oxXipY$">https://petsc.org/release/faq/</a> for trouble shooting.</i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: Petsc Release Version 3.20.4, unknown </i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: flubio_coupled on a linux_x86_64 named localhost.localdomain by edo Thu Oct 3 14:17:20 2024</i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: Configure options PETSC_ARCH=linux_x86_64 FOPTFLAGS=-O3 COPTFLAGS=-O3 CXXOPTFLAGS=-O3 -with-debugging=no -download-fblaslapack=1 -download-superlu_dist -download-mumps -download-hypre -download-metis -download-parmetis -download-scalapack -download-ml -download-slepc -download-hpddm -download-cmake -with-mpi-dir=/home/edo/software_repo/openmpi-5.0.1/build/</i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: #1 MatGetInfo() at /home/edo/software_repo/petsc/src/mat/interface/matrix.c:3005</i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: #2 PCSetUp_GAMG() at /home/edo/software_repo/petsc/src/ksp/pc/impls/gamg/gamg.c:619</i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: #3 PCSetUp() at /home/edo/software_repo/petsc/src/ksp/pc/interface/precon.c:1080</i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: #4 KSPSetUp() at /home/edo/software_repo/petsc/src/ksp/ksp/interface/itfunc.c:415</i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: It appears a new error in the code was triggered after a previous error, possibly because:</i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: - The first error was not properly handled via (for example) the use of</i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: PetscCall(TheFunctionThatErrors()); or</i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: - The second error was triggered while handling the first error.</i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: Above is the traceback for the previous unhandled error, below the traceback for the next error</i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: ALL ERRORS in the PETSc libraries are fatal, you should add the appropriate error checking to the code</i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: No support for this operation for this object type</i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: No method getinfo for Mat of type nest</i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc!</i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: Option left: name:-UPeqn_fieldsplit_p_pc_type value: hypre source: command line</i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: Option left: name:-UPeqn_fieldsplit_u_pc_type value: bjacobi source: command line</i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: See <a href="https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!a6ejiT9ML4BpeAAcqVuIMX-J7sA75OiyYcHeUd6-INXK8sR-gcQPA3ndqsgpzphK4tIBnE13dbzycfmUkwST7aXF6oxXipY$">https://petsc.org/release/faq/</a> for trouble shooting.</i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: Petsc Release Version 3.20.4, unknown </i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: flubio_coupled on a linux_x86_64 named localhost.localdomain by edo Thu Oct 3 14:17:20 2024</i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: Configure options PETSC_ARCH=linux_x86_64 FOPTFLAGS=-O3 COPTFLAGS=-O3 CXXOPTFLAGS=-O3 -with-debugging=no -download-fblaslapack=1 -download-superlu_dist -download-mumps -download-hypre -download-metis -download-parmetis -download-scalapack -download-ml -download-slepc -download-hpddm -download-cmake -with-mpi-dir=/home/edo/software_repo/openmpi-5.0.1/build/</i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: #1 MatGetInfo() at /home/edo/software_repo/petsc/src/mat/interface/matrix.c:3005</i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: #2 PCSetUp_GAMG() at /home/edo/software_repo/petsc/src/ksp/pc/impls/gamg/gamg.c:619</i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: #3 PCSetUp() at /home/edo/software_repo/petsc/src/ksp/pc/interface/precon.c:1080</i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: #4 KSPSetUp() at /home/edo/software_repo/petsc/src/ksp/ksp/interface/itfunc.c:415</i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: #5 KSPSolve_Private() at /home/edo/software_repo/petsc/src/ksp/ksp/interface/itfunc.c:832</i></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><i>[0]PETSC ERROR: #6 KSPSolve() at /home/edo/software_repo/petsc/src/ksp/ksp/interface/itfunc.c:1079</i></div></div></div></div></div></div></blockquote><div dir="ltr"><br></div><div>This the code I am doing right now:</div><div><br></div><div><i> M = this%setMomentumBlockMatrix()<br> C = this%setPressureBlockMatrix()<br> G = this%setPressureGradBlockMatrix()<br> D = this%setDivBlockMatrix()<br> bhat = this%setPressureMomentumRhs()<br><br> !------------------!<br> ! Create Ahat !<br> !-----------------!<br><br> ! Bottom left block of Ahat<br> Mat_array(4) = M <br><br> ! Extraction of the diagonal of M<br> call MatCreateVecs(M, PETSC_NULL_VEC, v, ierr) !v has the parallel distribution of M<br> call MatGetDiagonal(M, v, ierr)<br><br> call MatCreateDiagonal(v, diag_2M, ierr)<br> call MatScale(diag_2M, 2.0, ierr) ! store 2*diagonal part of M<br> call MatConvert(diag_2M, MATAIJ, MAT_INPLACE_MATRIX, diag_2M, ierr)<br> call VecReciprocal(v, ierr)<br><br> ! Creation of D_M_inv_G = D_M_inv*G<br> call MatDuplicate(G,MAT_COPY_VALUES, D_M_inv_G, ierr) ! D_M_inv_G contains G<br> call MatCreateVecs(D_M_inv_G, PETSC_NULL_VEC, v_redistributed, ierr) ! v_redistributed has the parallel distribution of D_M_inv_G<br><br> call VecGetOwnershipRange(v, col_min, col_max, ierr) <br> call ISCreateStride(PETSC_COMM_WORLD, col_max-col_min, col_min, 1, is_from, ierr)<br> call VecGetOwnershipRange(v_redistributed, col_min, col_max, ierr)<br> call ISCreateStride(PETSC_COMM_WORLD, col_max-col_min, col_min, 1, is_to, ierr)<br><br> call VecScatterCreate(v,is_from,v_redistributed,is_to, scat, ierr)<br> call VecScatterBegin(scat, v, v_redistributed,INSERT_VALUES,SCATTER_FORWARD, ierr)<br> call VecScatterEnd( scat, v, v_redistributed,INSERT_VALUES,SCATTER_FORWARD, ierr)<br> call MatDiagonalScale( D_M_inv_G, v_redistributed, PETSC_NULL_VEC, ierr) ! D_M_inv_G contains D_M_inv*G<br><br> call ISDestroy(is_to, ierr)<br> call ISDestroy(is_from, ierr)<br><br> ! Creation of Chat (PETSC_DEFAULT_REAL -> PETSC_DETERMINE)<br> call MatMatMult(D, D_M_inv_G, MAT_INITIAL_MATRIX, PETSC_DEFAULT_REAL, Chat, ierr) ! Chat contains D*D_M_inv*G<br> call MatAXPY( Chat, 1.0, C, SUBSET_NONZERO_PATTERN, ierr) ! Chat contains C + D*D_M_inv*G<br> Mat_array(1) = Chat ! Top left block of Ahat<br> <br> ! Creation of Ghat<br> call MatMatMult(M, D_M_inv_G, MAT_INITIAL_MATRIX, PETSC_DEFAULT_REAL, Ghat, ierr) ! Ghat contains M*D_M_inv*G<br> call MatAYPX( Ghat, -1.0, G, UNKNOWN_NONZERO_PATTERN, ierr) ! Ghat contains G - M*D_M_inv*G<br> Mat_array(3) = Ghat ! Bottom left block of Ahat<br> <br> ! Creation of -D<br> call MatScale(D,-1.0, ierr)<br> Mat_array(2) = D ! Top right block of Ahat<br> <br> ! Creation of Ahat = transformed+ reordered A_input<br> call MatCreateNest(PETSC_COMM_WORLD, 2, PETSC_NULL_IS, 2, PETSC_NULL_IS, Mat_array, Ahat, ierr)<br> <br> ! Creation of Pmat<br> Mat_array(4) = diag_2M<br> Mat_array(2) = PETSC_NULL_MAT ! Cancel top right block<br> call MatCreateNest(PETSC_COMM_WORLD, 2, PETSC_NULL_IS, 2, PETSC_NULL_IS, Mat_array, Pmat, ierr);<br> <br> ! Finalisation of the preconditioner<br> call ISCreateStride(PETSC_COMM_WORLD, numberOfElements, 0, 1, is_P_hat, ierr); <br> call ISCreateStride(PETSC_COMM_WORLD, (3-bdim)*numberOfElements, numberOfElements, 1, is_U_hat, ierr);<br><br> !----------------!<br> ! KSP PART !<br> !----------------!<br><br> call KSPSetOperators(this%ksp, Ahat, Pmat, ierr)<br> call KSPGetPC(this%ksp, pc, ierr)<br><br> call PCSetFromOptions(pc, ierr)<br> call PCSetUp(pc, ierr)<br> call KSPSetFromOptions(this%ksp, ierr)<br> call KSPSetUp(this%ksp, ierr)<br><br> call VecDuplicate(bhat, xhat, ierr)<br><br> ! I think it makes sense to move this part where fieldsplit preconditioner is defined<br> call PCFieldSplitSetIS(pc, "phat", is_P_hat, ierr)<br> call PCFieldSplitSetIS(pc, "uhat", is_U_hat, ierr)<br><br> call KSPSolve(this%ksp, bhat, xhat, ierr)</i><br></div><div><br></div><div>KSP id FGMRES + fieldsplit with default settings.</div></div></div></blockquote><div><br></div><div>Apparently not, otherwise GAMG would not be called.</div><div>In particular, it looks like the code is currently trying to call GAMG on the global outer MatNest, whereas you likely want to call it on just one of the inner block.</div><div>Please send the complete code, complete options file, and complete command line arguments you are using, because with just the snippet you sent and the stack trace, the code is indeed not expected to work, e.g., I see no occurence of -pc_type fieldsplit or PCSetType(pc, PCFIELDSPLIT).</div><div><br></div><div>Thanks,</div><div>Pierre</div><br><blockquote type="cite"><div><div dir="ltr"><div>Many thanks!</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Il giorno gio 3 ott 2024 alle ore 19:01 Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com">junchao.zhang@gmail.com</a>> ha scritto:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Could you also provide input files to run your code?<div><br clear="all"><div><div dir="ltr" class="gmail_signature"><div dir="ltr">--Junchao Zhang</div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Oct 3, 2024 at 5:14 AM Edoardo alinovi <<a href="mailto:edoardo.alinovi@gmail.com" target="_blank">edoardo.alinovi@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr" class="gmail_signature"><div dir="ltr"><div dir="ltr"><div>Hi Barry,</div><div><br></div><div>I am trying to port this preconditioner I found presented at the last PETSc conference into may code:</div><div><a href="https://urldefense.us/v3/__https://github.com/ndjinga/SOLVERLAB/blob/master/CoreFlows/examples/C/SaddlePointPreconditioner/SaddlePointLinearSolver.c__;!!G_uCfscf7eWS!a-JnrBsta75n_WDtxUNd7Xb8_m7tg1pWSxwWNTmUBSfpBwISq4IxJkbM6QgADd-x23Q_2s-d6d6bOWCKGRMxMzHyc9DDm7A$" target="_blank">https://github.com/ndjinga/SOLVERLAB/blob/master/CoreFlows/examples/C/SaddlePointPreconditioner/SaddlePointLinearSolver.c</a></div><div><br></div><div>So far so good, I am just translating C to Fortran, however when I try to solve my linear system with: </div><div><br></div></div></div></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div class="gmail_signature"><div><div><div>call KSPSetOperators(ksp, Ahat, Pmat, ierr)</div></div></div></div><div class="gmail_signature"><div><div><div>call KSPSolve(ksp, bhat, xhat, ierr)</div></div></div><div><br></div></div></blockquote><div dir="ltr" class="gmail_signature"><div dir="ltr"><div dir="ltr"><div>with Ahat and Pmat being indeed two nest MatNest, created from an array of Mat as:</div><div><br></div></div></div></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div class="gmail_signature"><div><div><div>call MatCreateNest(PETSC_COMM_WORLD, 2, PETSC_NULL_IS, 2, PETSC_NULL_IS, Mat_array, Ahat, ierr)</div></div></div></div><div class="gmail_signature"><div><div><div>call MatCreateNest(PETSC_COMM_WORLD, 2, PETSC_NULL_IS, 2, PETSC_NULL_IS, Mat_array, Pmat, ierr)</div></div></div></div></blockquote><div dir="ltr" class="gmail_signature"><div dir="ltr"><div dir="ltr"><div><br></div><div>However I am getting this error on getinfo:</div><div><br></div><div>from C to Fortran</div><div><br></div><div>[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<br>[0]PETSC ERROR: No support for this operation for this object type<br>[0]PETSC ERROR: No method getinfo for Mat of type nest<br>[0]PETSC ERROR: See <a href="https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!a-JnrBsta75n_WDtxUNd7Xb8_m7tg1pWSxwWNTmUBSfpBwISq4IxJkbM6QgADd-x23Q_2s-d6d6bOWCKGRMxMzHyRUMnyvE$" target="_blank">https://petsc.org/release/faq/</a> for trouble shooting.<br>[0]PETSC ERROR: Petsc Release Version 3.20.4, unknown <br></div><div><br></div><div>How can I get rid of this issue?</div><div><br></div><div>Many thanks!</div><div><br></div></div></div></div></div>
</blockquote></div>
</blockquote></div></div>
</div></blockquote></div><br></body></html>