From rpoli at utexas.edu Fri Aug 1 21:52:28 2025 From: rpoli at utexas.edu (Renato E Poli) Date: Fri, 1 Aug 2025 21:52:28 -0500 Subject: [petsc-users] Error compiling with cuda Message-ID: Hi, I see the error below when compiling with --with-cuda. Any suggestion? CC arch-linux-c-opt/obj/src/mat/impls/aij/seq/inode2.o CC arch-linux-c-opt/obj/src/mat/impls/aij/seq/aijperm/aijperm.o CC arch-linux-c-opt/obj/src/mat/impls/aij/seq/mattransposematmult.o CC arch-linux-c-opt/obj/src/mat/impls/aij/seq/matptap.o CC arch-linux-c-opt/obj/src/mat/impls/aij/seq/matrart.o FC arch-linux-c-opt/obj/src/sys/ftn-mod/petscsysmod.o NVFORTRAN-S-0034-Syntax error at or near end of line (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: 6) NVFORTRAN-S-0034-Syntax error at or near end of line (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: 18) NVFORTRAN-S-0034-Syntax error at or near end of line (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: 30) NVFORTRAN-S-0034-Syntax error at or near end of line (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: 42) NVFORTRAN-S-0034-Syntax error at or near end of line (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: 54) NVFORTRAN-S-0034-Syntax error at or near end of line (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: 65) NVFORTRAN-S-0034-Syntax error at or near end of line (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: 76) NVFORTRAN-S-0034-Syntax error at or near end of line (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: 88) NVFORTRAN-S-0034-Syntax error at or near end of line (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: 100) NVFORTRAN-S-0034-Syntax error at or near end of line (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: 112) NVFORTRAN-S-0034-Syntax error at or near end of line (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: 124) NVFORTRAN-S-0034-Syntax error at or near end of line (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: 136) NVFORTRAN-S-0034-Syntax error at or near end of line (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: 148) NVFORTRAN-S-0034-Syntax error at or near end of line (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: 160) NVFORTRAN-S-0034-Syntax error at or near end of line (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscviewer.h: 6) NVFORTRAN-S-0034-Syntax error at or near end of line (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscviewer.h: 17) NVFORTRAN-S-0034-Syntax error at or near end of line (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscoptions.h: 6) NVFORTRAN-S-0034-Syntax error at or near end of line (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscoptions.h: 18) NVFORTRAN-S-0034-Syntax error at or near end of line (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscoptions.h: 30) NVFORTRAN-S-0034-Syntax error at or near end of line (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petsclog.h: 5) NVFORTRAN-S-0034-Syntax error at or near end of line (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petsclog.h: 17) NVFORTRAN-S-0034-Syntax error at or near end of line (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petsclog.h: 29) NVFORTRAN-S-0034-Syntax error at or near end of line (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscdevice.h: 6) NVFORTRAN-S-0034-Syntax error at or near end of line (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscdevice.h: 17) 0 inform, 0 warnings, 24 severes, 0 fatal for petscsysdef -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay.anl at fastmail.org Sat Aug 2 00:28:36 2025 From: balay.anl at fastmail.org (Satish Balay) Date: Sat, 2 Aug 2025 00:28:36 -0500 (CDT) Subject: [petsc-users] Error compiling with cuda In-Reply-To: References: Message-ID: <69cf94ee-442f-6d8d-2d7a-9368106474a9@fastmail.org> NVFORTRAN (based on flang-17) does not support some F2008 features that are required here (flang-19+ should work) Suggest using gfortran instead. Satish On Fri, 1 Aug 2025, Renato E Poli wrote: > Hi, > > I see the error below when compiling with --with-cuda. > > Any suggestion? > > > CC arch-linux-c-opt/obj/src/mat/impls/aij/seq/inode2.o > CC arch-linux-c-opt/obj/src/mat/impls/aij/seq/aijperm/aijperm.o > CC arch-linux-c-opt/obj/src/mat/impls/aij/seq/mattransposematmult.o > CC arch-linux-c-opt/obj/src/mat/impls/aij/seq/matptap.o > CC arch-linux-c-opt/obj/src/mat/impls/aij/seq/matrart.o > FC arch-linux-c-opt/obj/src/sys/ftn-mod/petscsysmod.o > NVFORTRAN-S-0034-Syntax error at or near end of line > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: 6) > NVFORTRAN-S-0034-Syntax error at or near end of line > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: 18) > NVFORTRAN-S-0034-Syntax error at or near end of line > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: 30) > NVFORTRAN-S-0034-Syntax error at or near end of line > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: 42) > NVFORTRAN-S-0034-Syntax error at or near end of line > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: 54) > NVFORTRAN-S-0034-Syntax error at or near end of line > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: 65) > NVFORTRAN-S-0034-Syntax error at or near end of line > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: 76) > NVFORTRAN-S-0034-Syntax error at or near end of line > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: 88) > NVFORTRAN-S-0034-Syntax error at or near end of line > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: 100) > NVFORTRAN-S-0034-Syntax error at or near end of line > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: 112) > NVFORTRAN-S-0034-Syntax error at or near end of line > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: 124) > NVFORTRAN-S-0034-Syntax error at or near end of line > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: 136) > NVFORTRAN-S-0034-Syntax error at or near end of line > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: 148) > NVFORTRAN-S-0034-Syntax error at or near end of line > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: 160) > NVFORTRAN-S-0034-Syntax error at or near end of line > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscviewer.h: 6) > NVFORTRAN-S-0034-Syntax error at or near end of line > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscviewer.h: > 17) > NVFORTRAN-S-0034-Syntax error at or near end of line > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscoptions.h: > 6) > NVFORTRAN-S-0034-Syntax error at or near end of line > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscoptions.h: > 18) > NVFORTRAN-S-0034-Syntax error at or near end of line > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscoptions.h: > 30) > NVFORTRAN-S-0034-Syntax error at or near end of line > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petsclog.h: 5) > NVFORTRAN-S-0034-Syntax error at or near end of line > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petsclog.h: 17) > NVFORTRAN-S-0034-Syntax error at or near end of line > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petsclog.h: 29) > NVFORTRAN-S-0034-Syntax error at or near end of line > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscdevice.h: 6) > NVFORTRAN-S-0034-Syntax error at or near end of line > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscdevice.h: > 17) > 0 inform, 0 warnings, 24 severes, 0 fatal for petscsysdef > From rpoli at utexas.edu Sat Aug 2 09:48:34 2025 From: rpoli at utexas.edu (Renato E Poli) Date: Sat, 2 Aug 2025 09:48:34 -0500 Subject: [petsc-users] Error compiling with cuda In-Reply-To: <69cf94ee-442f-6d8d-2d7a-9368106474a9@fastmail.org> References: <69cf94ee-442f-6d8d-2d7a-9368106474a9@fastmail.org> Message-ID: It seems that the installer is the one choosing to use nvfortran instead of gfortran. I am building a docker container (clean environment). I simply install CUDA SDK from https://urldefense.us/v3/__https://developer.download.nvidia.com/hpc-sdk/25.5/nvhpc_2025_255_Linux_x86_64_cuda_12.9.tar.gz__;!!G_uCfscf7eWS!cBTymZpXcpFZ7BHpCGxn0ooq_STD1f9CioiFgRypIYNWg9lpwnmLpddrWk0-J-wQoeTw9Ipu2FPm037mQg_C2w$ , set paths and call petsc configure. RUN cd petsc-3.23.5 &&\ ./configure --prefix=/usr/local/petsc --with-debugging=0 --with-mpi --download-hdf5 --download-hypre --with-blaslapack --with-cuda && \ make all && make install Do you see any reason for using the wrong compiler? Renato On Sat, Aug 2, 2025 at 12:28?AM Satish Balay wrote: > NVFORTRAN (based on flang-17) does not support some F2008 features that > are required here (flang-19+ should work) > > Suggest using gfortran instead. > > Satish > > On Fri, 1 Aug 2025, Renato E Poli wrote: > > > Hi, > > > > I see the error below when compiling with --with-cuda. > > > > Any suggestion? > > > > > > CC arch-linux-c-opt/obj/src/mat/impls/aij/seq/inode2.o > > CC arch-linux-c-opt/obj/src/mat/impls/aij/seq/aijperm/aijperm.o > > CC > arch-linux-c-opt/obj/src/mat/impls/aij/seq/mattransposematmult.o > > CC arch-linux-c-opt/obj/src/mat/impls/aij/seq/matptap.o > > CC arch-linux-c-opt/obj/src/mat/impls/aij/seq/matrart.o > > FC arch-linux-c-opt/obj/src/sys/ftn-mod/petscsysmod.o > > NVFORTRAN-S-0034-Syntax error at or near end of line > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: 6) > > NVFORTRAN-S-0034-Syntax error at or near end of line > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > 18) > > NVFORTRAN-S-0034-Syntax error at or near end of line > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > 30) > > NVFORTRAN-S-0034-Syntax error at or near end of line > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > 42) > > NVFORTRAN-S-0034-Syntax error at or near end of line > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > 54) > > NVFORTRAN-S-0034-Syntax error at or near end of line > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > 65) > > NVFORTRAN-S-0034-Syntax error at or near end of line > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > 76) > > NVFORTRAN-S-0034-Syntax error at or near end of line > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > 88) > > NVFORTRAN-S-0034-Syntax error at or near end of line > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > 100) > > NVFORTRAN-S-0034-Syntax error at or near end of line > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > 112) > > NVFORTRAN-S-0034-Syntax error at or near end of line > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > 124) > > NVFORTRAN-S-0034-Syntax error at or near end of line > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > 136) > > NVFORTRAN-S-0034-Syntax error at or near end of line > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > 148) > > NVFORTRAN-S-0034-Syntax error at or near end of line > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > 160) > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscviewer.h: 6) > > NVFORTRAN-S-0034-Syntax error at or near end of line > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscviewer.h: > > 17) > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscoptions.h: > > 6) > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscoptions.h: > > 18) > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscoptions.h: > > 30) > > NVFORTRAN-S-0034-Syntax error at or near end of line > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petsclog.h: 5) > > NVFORTRAN-S-0034-Syntax error at or near end of line > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petsclog.h: > 17) > > NVFORTRAN-S-0034-Syntax error at or near end of line > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petsclog.h: > 29) > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscdevice.h: 6) > > NVFORTRAN-S-0034-Syntax error at or near end of line > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscdevice.h: > > 17) > > 0 inform, 0 warnings, 24 severes, 0 fatal for petscsysdef > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kiranhegde75 at gmail.com Sat Aug 2 10:21:59 2025 From: kiranhegde75 at gmail.com (kiran hegde) Date: Sat, 2 Aug 2025 20:51:59 +0530 Subject: [petsc-users] missing binary operator before token Message-ID: Hi , I am compiling fortran code using petsc include & lib, but getting below error: 298 | #if PetscHasAttribute(format) && !defined(PETSC_SKIP_ATTRIBUTE_FORMAT) Error: missing binary operator before token '(' /home/kiranhegde/MyLinuxApplications/Installations/petsc/include/petscmacros.h:330:0: 330 | #if PetscHasAttribute(__cold__) Error: missing binary operator before token '(' /home/kiranhegde/MyLinuxApplications/Installations/petsc/include/petscmacros.h:332:0: 332 | #elif PetscHasAttribute(cold) /* some implementations (old gcc) use no underscores */ Error: missing binary operator before token '(' /home/kiranhegde/MyLinuxApplications/Installations/petsc/include/petscmacros.h:355:0: 298 | #if PetscHasAttribute(format) && !defined(PETSC_SKIP_ATTRIBUTE_FORMAT) Error: missing binary operator before token '(' /home/kiranhegde/MyLinuxApplications/Installations/petsc/include/petscmacros.h:330:0: 330 | #if PetscHasAttribute(__cold__) Error: missing binary operator before token '(' /home/kiranhegde/MyLinuxApplications/Installations/petsc/include/petscmacros.h:332:0: 332 | #elif PetscHasAttribute(cold) /* some implementations (old gcc) use no underscores */ Error: missing binary operator before token '(' /home/kiranhegde/MyLinuxApplications/Installations/petsc/include/petscmacros.h:355:0: Same errors repeat for many lines and header files I am using PETSC at 3.23.4 version with gcc at 15.1.0 -- Regards, Kiran Hegde Mobile:+91 9972600235 -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay.anl at fastmail.org Sat Aug 2 11:04:17 2025 From: balay.anl at fastmail.org (Satish Balay) Date: Sat, 2 Aug 2025 11:04:17 -0500 (CDT) Subject: [petsc-users] missing binary operator before token In-Reply-To: References: Message-ID: Do you get these errors with 'make check' on a petsc install - or when compiling a petsc fortran example (with the corresponding makefile)? Satish On Sat, 2 Aug 2025, kiran hegde wrote: > Hi , > > I am compiling fortran code using petsc include & lib, but getting > below error: > > 298 | #if PetscHasAttribute(format) && !defined(PETSC_SKIP_ATTRIBUTE_FORMAT) > Error: missing binary operator before token '(' > /home/kiranhegde/MyLinuxApplications/Installations/petsc/include/petscmacros.h:330:0: > > 330 | #if PetscHasAttribute(__cold__) > Error: missing binary operator before token '(' > /home/kiranhegde/MyLinuxApplications/Installations/petsc/include/petscmacros.h:332:0: > > 332 | #elif PetscHasAttribute(cold) /* some implementations (old gcc) use > no underscores */ > Error: missing binary operator before token '(' > /home/kiranhegde/MyLinuxApplications/Installations/petsc/include/petscmacros.h:355:0: > > 298 | #if PetscHasAttribute(format) && !defined(PETSC_SKIP_ATTRIBUTE_FORMAT) > Error: missing binary operator before token '(' > /home/kiranhegde/MyLinuxApplications/Installations/petsc/include/petscmacros.h:330:0: > > 330 | #if PetscHasAttribute(__cold__) > Error: missing binary operator before token '(' > /home/kiranhegde/MyLinuxApplications/Installations/petsc/include/petscmacros.h:332:0: > > 332 | #elif PetscHasAttribute(cold) /* some implementations (old gcc) use > no underscores */ > Error: missing binary operator before token '(' > /home/kiranhegde/MyLinuxApplications/Installations/petsc/include/petscmacros.h:355:0: > > Same errors repeat for many lines and header files > I am using PETSC at 3.23.4 version with gcc at 15.1.0 > > > > From balay.anl at fastmail.org Sat Aug 2 11:11:48 2025 From: balay.anl at fastmail.org (Satish Balay) Date: Sat, 2 Aug 2025 11:11:48 -0500 (CDT) Subject: [petsc-users] Error compiling with cuda In-Reply-To: References: <69cf94ee-442f-6d8d-2d7a-9368106474a9@fastmail.org> Message-ID: Please send configure.log presumably you are using a pre-installed mpi from hpc-sdk - that uses NV compilers. [and PETSc configure is defaulting to mpicc/mpif90 in PATH] If you do not have MPI with gcc/gfortran - you can do something like: ./configure --with-cuda=1 --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich [other options] When no options are specified - configure guesses - and picks up what it can find - and sometimes that might not be what you intend to use.. Satish On Sat, 2 Aug 2025, Renato E Poli wrote: > It seems that the installer is the one choosing to use nvfortran instead of > gfortran. > > I am building a docker container (clean environment). > I simply install CUDA SDK from > https://urldefense.us/v3/__https://developer.download.nvidia.com/hpc-sdk/25.5/nvhpc_2025_255_Linux_x86_64_cuda_12.9.tar.gz__;!!G_uCfscf7eWS!cBTymZpXcpFZ7BHpCGxn0ooq_STD1f9CioiFgRypIYNWg9lpwnmLpddrWk0-J-wQoeTw9Ipu2FPm037mQg_C2w$ > , set paths and call petsc configure. > > RUN cd petsc-3.23.5 &&\ > ./configure --prefix=/usr/local/petsc --with-debugging=0 --with-mpi > --download-hdf5 --download-hypre --with-blaslapack --with-cuda && \ > make all && make install > > Do you see any reason for using the wrong compiler? > > Renato > > On Sat, Aug 2, 2025 at 12:28?AM Satish Balay wrote: > > > NVFORTRAN (based on flang-17) does not support some F2008 features that > > are required here (flang-19+ should work) > > > > Suggest using gfortran instead. > > > > Satish > > > > On Fri, 1 Aug 2025, Renato E Poli wrote: > > > > > Hi, > > > > > > I see the error below when compiling with --with-cuda. > > > > > > Any suggestion? > > > > > > > > > CC arch-linux-c-opt/obj/src/mat/impls/aij/seq/inode2.o > > > CC arch-linux-c-opt/obj/src/mat/impls/aij/seq/aijperm/aijperm.o > > > CC > > arch-linux-c-opt/obj/src/mat/impls/aij/seq/mattransposematmult.o > > > CC arch-linux-c-opt/obj/src/mat/impls/aij/seq/matptap.o > > > CC arch-linux-c-opt/obj/src/mat/impls/aij/seq/matrart.o > > > FC arch-linux-c-opt/obj/src/sys/ftn-mod/petscsysmod.o > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: 6) > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > 18) > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > 30) > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > 42) > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > 54) > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > 65) > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > 76) > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > 88) > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > 100) > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > 112) > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > 124) > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > 136) > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > 148) > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > 160) > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscviewer.h: 6) > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscviewer.h: > > > 17) > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscoptions.h: > > > 6) > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscoptions.h: > > > 18) > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscoptions.h: > > > 30) > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petsclog.h: 5) > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petsclog.h: > > 17) > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petsclog.h: > > 29) > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscdevice.h: 6) > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscdevice.h: > > > 17) > > > 0 inform, 0 warnings, 24 severes, 0 fatal for petscsysdef > > > > > > > > From rpoli at utexas.edu Sat Aug 2 12:12:21 2025 From: rpoli at utexas.edu (Renato E Poli) Date: Sat, 2 Aug 2025 12:12:21 -0500 Subject: [petsc-users] Error compiling with cuda In-Reply-To: References: <69cf94ee-442f-6d8d-2d7a-9368106474a9@fastmail.org> Message-ID: Understood. I believe I am running in circles. I managed to create the container, but it ended up with multiple MPI implementations. It seems that CUDA SDK has its own MPI, and polutes the PATH. Is there any docker container PETSC+CUDA you would recommend? This is a snippet of my Dockerfile, but I am not 100% happy with it. <<<<<< FROM docker.io/ubuntu:24.04 # Install necessary build tools and dependencies RUN apt-get update && apt-get install -y libmpich-dev gfortran libomp-dev (...) (...) ## NVIDIA HPC SDK RUN wget https://urldefense.us/v3/__https://developer.download.nvidia.com/hpc-sdk/25.5/nvhpc_2025_255_Linux_x86_64_cuda_12.9.tar.gz__;!!G_uCfscf7eWS!ft2AAfoKdKf-j-6E6nTqtzXHqQEmnbsH7as3hylOIpz7OiONASpD-qNuRNcnDapAFok1YeTP2lLQfH7ne1cFSw$ && tar xpzfv nvhpc_2025_255_Linux_x86_64_cuda_12.9.tar.gz && nvhpc_2025_255_Linux_x86_64_cuda_12.9/install --silent --accept-license --instal-dir /opt/nvidia/hpc_sdk RUN cd petsc-3.23.5 &&\ ./configure --prefix=/usr/local/petsc --PETSC_ARCH=arch-linux-c-opt --with-debugging=0 \ --with-mpi --download-hypre --with-blaslapack --with-cuda --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/25.5/cuda && \ make all install && \ cd .. (...) ENV PATH=${NVHPC_ROOT}/compilers/bin:${NVHPC_ROOT}/comm_libs/mpi/bin:${PATH} ENV MANPATH=${NVHPC_ROOT}/compilers/man:${MANPATH} ENV LD_LIBRARY_PATH=${NVHPC_ROOT}/compilers/lib:${LD_LIBRARY_PATH} >>>>>> Renato On Sat, Aug 2, 2025 at 11:11?AM Satish Balay wrote: > Please send configure.log > > presumably you are using a pre-installed mpi from hpc-sdk - that uses NV > compilers. [and PETSc configure is defaulting to mpicc/mpif90 in PATH] > > If you do not have MPI with gcc/gfortran - you can do something like: > > ./configure --with-cuda=1 --with-cc=gcc --with-cxx=g++ --with-fc=gfortran > --download-mpich [other options] > > When no options are specified - configure guesses - and picks up what it > can find - and sometimes that might not be what you intend to use.. > > Satish > > On Sat, 2 Aug 2025, Renato E Poli wrote: > > > It seems that the installer is the one choosing to use nvfortran instead > of > > gfortran. > > > > I am building a docker container (clean environment). > > I simply install CUDA SDK from > > > https://urldefense.us/v3/__https://developer.download.nvidia.com/hpc-sdk/25.5/nvhpc_2025_255_Linux_x86_64_cuda_12.9.tar.gz__;!!G_uCfscf7eWS!cBTymZpXcpFZ7BHpCGxn0ooq_STD1f9CioiFgRypIYNWg9lpwnmLpddrWk0-J-wQoeTw9Ipu2FPm037mQg_C2w$ > > , set paths and call petsc configure. > > > > RUN cd petsc-3.23.5 &&\ > > ./configure --prefix=/usr/local/petsc --with-debugging=0 --with-mpi > > --download-hdf5 --download-hypre --with-blaslapack --with-cuda && \ > > make all && make install > > > > Do you see any reason for using the wrong compiler? > > > > Renato > > > > On Sat, Aug 2, 2025 at 12:28?AM Satish Balay > wrote: > > > > > NVFORTRAN (based on flang-17) does not support some F2008 features that > > > are required here (flang-19+ should work) > > > > > > Suggest using gfortran instead. > > > > > > Satish > > > > > > On Fri, 1 Aug 2025, Renato E Poli wrote: > > > > > > > Hi, > > > > > > > > I see the error below when compiling with --with-cuda. > > > > > > > > Any suggestion? > > > > > > > > > > > > CC arch-linux-c-opt/obj/src/mat/impls/aij/seq/inode2.o > > > > CC > arch-linux-c-opt/obj/src/mat/impls/aij/seq/aijperm/aijperm.o > > > > CC > > > arch-linux-c-opt/obj/src/mat/impls/aij/seq/mattransposematmult.o > > > > CC arch-linux-c-opt/obj/src/mat/impls/aij/seq/matptap.o > > > > CC arch-linux-c-opt/obj/src/mat/impls/aij/seq/matrart.o > > > > FC arch-linux-c-opt/obj/src/sys/ftn-mod/petscsysmod.o > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: 6) > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > 18) > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > 30) > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > 42) > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > 54) > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > 65) > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > 76) > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > 88) > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > 100) > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > 112) > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > 124) > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > 136) > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > 148) > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > 160) > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscviewer.h: 6) > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscviewer.h: > > > > 17) > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscoptions.h: > > > > 6) > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscoptions.h: > > > > 18) > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscoptions.h: > > > > 30) > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petsclog.h: 5) > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petsclog.h: > > > 17) > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petsclog.h: > > > 29) > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscdevice.h: 6) > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscdevice.h: > > > > 17) > > > > 0 inform, 0 warnings, 24 severes, 0 fatal for petscsysdef > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay.anl at fastmail.org Sat Aug 2 12:20:49 2025 From: balay.anl at fastmail.org (Satish Balay) Date: Sat, 2 Aug 2025 12:20:49 -0500 (CDT) Subject: [petsc-users] Error compiling with cuda In-Reply-To: References: <69cf94ee-442f-6d8d-2d7a-9368106474a9@fastmail.org> Message-ID: <7133e686-7e9d-fb90-92e8-637c86e1d405@fastmail.org> You can first fix on what MPI to use - and build everyting else with it. (and avoid mixing in multiple MPIs) I've had issues with Ubuntu built MPI - but if it works for you - thats great! You can make sure /usr/bin is ahead in your PATH [before nvhpc-sdk so that this MPI gets picked up] - or specify explicitly to PETSc with: Or use 'cuda toolkit' instead of 'nvhpc-sdk' ./configure --with-cc=/usr/bin/mpicc --with-cxx=/usr/bin/mpicxx --with-fc=/usr/bin/mpif90 --download-hypre etc.. BTW: PETSc configure attempts to run some binaries to detect compiler/library features, and such run tests (when linked in with cuda libraries) can fail if a cuda build is done on a box (container?) without nvidia driver installed. Satish On Sat, 2 Aug 2025, Renato E Poli wrote: > Understood. > > I believe I am running in circles. > > I managed to create the container, but it ended up with multiple MPI > implementations. > It seems that CUDA SDK has its own MPI, and polutes the PATH. > Is there any docker container PETSC+CUDA you would recommend? > This is a snippet of my Dockerfile, but I am not 100% happy with it. > > <<<<<< > FROM docker.io/ubuntu:24.04 > # Install necessary build tools and dependencies > RUN apt-get update && apt-get install -y libmpich-dev gfortran > libomp-dev (...) > (...) > > ## NVIDIA HPC SDK > RUN wget > https://urldefense.us/v3/__https://developer.download.nvidia.com/hpc-sdk/25.5/nvhpc_2025_255_Linux_x86_64_cuda_12.9.tar.gz__;!!G_uCfscf7eWS!ft2AAfoKdKf-j-6E6nTqtzXHqQEmnbsH7as3hylOIpz7OiONASpD-qNuRNcnDapAFok1YeTP2lLQfH7ne1cFSw$ > && > tar xpzfv nvhpc_2025_255_Linux_x86_64_cuda_12.9.tar.gz && > nvhpc_2025_255_Linux_x86_64_cuda_12.9/install --silent --accept-license > --instal-dir /opt/nvidia/hpc_sdk > > RUN cd petsc-3.23.5 &&\ > ./configure --prefix=/usr/local/petsc --PETSC_ARCH=arch-linux-c-opt > --with-debugging=0 \ > --with-mpi --download-hypre --with-blaslapack --with-cuda > --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/25.5/cuda && \ > make all install && \ > cd .. > > (...) > ENV PATH=${NVHPC_ROOT}/compilers/bin:${NVHPC_ROOT}/comm_libs/mpi/bin:${PATH} > ENV MANPATH=${NVHPC_ROOT}/compilers/man:${MANPATH} > ENV LD_LIBRARY_PATH=${NVHPC_ROOT}/compilers/lib:${LD_LIBRARY_PATH} > >>>>>> > > Renato > > On Sat, Aug 2, 2025 at 11:11?AM Satish Balay wrote: > > > Please send configure.log > > > > presumably you are using a pre-installed mpi from hpc-sdk - that uses NV > > compilers. [and PETSc configure is defaulting to mpicc/mpif90 in PATH] > > > > If you do not have MPI with gcc/gfortran - you can do something like: > > > > ./configure --with-cuda=1 --with-cc=gcc --with-cxx=g++ --with-fc=gfortran > > --download-mpich [other options] > > > > When no options are specified - configure guesses - and picks up what it > > can find - and sometimes that might not be what you intend to use.. > > > > Satish > > > > On Sat, 2 Aug 2025, Renato E Poli wrote: > > > > > It seems that the installer is the one choosing to use nvfortran instead > > of > > > gfortran. > > > > > > I am building a docker container (clean environment). > > > I simply install CUDA SDK from > > > > > https://urldefense.us/v3/__https://developer.download.nvidia.com/hpc-sdk/25.5/nvhpc_2025_255_Linux_x86_64_cuda_12.9.tar.gz__;!!G_uCfscf7eWS!cBTymZpXcpFZ7BHpCGxn0ooq_STD1f9CioiFgRypIYNWg9lpwnmLpddrWk0-J-wQoeTw9Ipu2FPm037mQg_C2w$ > > > , set paths and call petsc configure. > > > > > > RUN cd petsc-3.23.5 &&\ > > > ./configure --prefix=/usr/local/petsc --with-debugging=0 --with-mpi > > > --download-hdf5 --download-hypre --with-blaslapack --with-cuda && \ > > > make all && make install > > > > > > Do you see any reason for using the wrong compiler? > > > > > > Renato > > > > > > On Sat, Aug 2, 2025 at 12:28?AM Satish Balay > > wrote: > > > > > > > NVFORTRAN (based on flang-17) does not support some F2008 features that > > > > are required here (flang-19+ should work) > > > > > > > > Suggest using gfortran instead. > > > > > > > > Satish > > > > > > > > On Fri, 1 Aug 2025, Renato E Poli wrote: > > > > > > > > > Hi, > > > > > > > > > > I see the error below when compiling with --with-cuda. > > > > > > > > > > Any suggestion? > > > > > > > > > > > > > > > CC arch-linux-c-opt/obj/src/mat/impls/aij/seq/inode2.o > > > > > CC > > arch-linux-c-opt/obj/src/mat/impls/aij/seq/aijperm/aijperm.o > > > > > CC > > > > arch-linux-c-opt/obj/src/mat/impls/aij/seq/mattransposematmult.o > > > > > CC arch-linux-c-opt/obj/src/mat/impls/aij/seq/matptap.o > > > > > CC arch-linux-c-opt/obj/src/mat/impls/aij/seq/matrart.o > > > > > FC arch-linux-c-opt/obj/src/sys/ftn-mod/petscsysmod.o > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: 6) > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > > 18) > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > > 30) > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > > 42) > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > > 54) > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > > 65) > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > > 76) > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > > 88) > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > > 100) > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > > 112) > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > > 124) > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > > 136) > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > > 148) > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > > 160) > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscviewer.h: 6) > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscviewer.h: > > > > > 17) > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscoptions.h: > > > > > 6) > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscoptions.h: > > > > > 18) > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscoptions.h: > > > > > 30) > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petsclog.h: 5) > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petsclog.h: > > > > 17) > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petsclog.h: > > > > 29) > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscdevice.h: 6) > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscdevice.h: > > > > > 17) > > > > > 0 inform, 0 warnings, 24 severes, 0 fatal for petscsysdef > > > > > > > > > > > > > > > > > > > From balay.anl at fastmail.org Sat Aug 2 12:25:40 2025 From: balay.anl at fastmail.org (Satish Balay) Date: Sat, 2 Aug 2025 12:25:40 -0500 (CDT) Subject: [petsc-users] Error compiling with cuda In-Reply-To: References: <69cf94ee-442f-6d8d-2d7a-9368106474a9@fastmail.org> Message-ID: On Sat, 2 Aug 2025, Renato E Poli wrote: > Is there any docker container PETSC+CUDA you would recommend? E4S has containers with PETSc+CUDA installed (via Spack) - I guess you can give it a try https://urldefense.us/v3/__https://e4s.io/download.html__;!!G_uCfscf7eWS!bkz1sbFlbqamdE76pGef9lk7FcXLMP0QAqrmHirWWpmTecelDXAOmImJKzyrVy9pP6eN7ABfWSZNON0GOVJHEO00ync$ Satish From stefano.zampini at gmail.com Sat Aug 2 12:26:34 2025 From: stefano.zampini at gmail.com (Stefano Zampini) Date: Sat, 2 Aug 2025 19:26:34 +0200 Subject: [petsc-users] Error compiling with cuda In-Reply-To: References: <69cf94ee-442f-6d8d-2d7a-9368106474a9@fastmail.org> Message-ID: NVIDIA ships OpenMPI. You can select the FORTRAN compiler by using the OMPI_FC environment variable, see https://urldefense.us/v3/__https://docs.open-mpi.org/en/v5.0.x/building-apps/customizing-wrappers.html__;!!G_uCfscf7eWS!dVGG_CfOVumpO5kAMWVbjzj4d6nRX1gwuaGnbv41cwClu5AMW_lX90wMUOZkeZwnbrCat53pFti0ckjLxs7nQJzi0CZkLW4$ Il giorno sab 2 ago 2025 alle ore 19:12 Renato E Poli ha scritto: > Understood. > > I believe I am running in circles. > > I managed to create the container, but it ended up with multiple MPI > implementations. > It seems that CUDA SDK has its own MPI, and polutes the PATH. > Is there any docker container PETSC+CUDA you would recommend? > This is a snippet of my Dockerfile, but I am not 100% happy with it. > > <<<<<< > FROM docker.io/ubuntu:24.04 > > # Install necessary build tools and dependencies > RUN apt-get update && apt-get install -y libmpich-dev gfortran > libomp-dev (...) > (...) > > ## NVIDIA HPC SDK > RUN wget > https://urldefense.us/v3/__https://developer.download.nvidia.com/hpc-sdk/25.5/nvhpc_2025_255_Linux_x86_64_cuda_12.9.tar.gz__;!!G_uCfscf7eWS!dVGG_CfOVumpO5kAMWVbjzj4d6nRX1gwuaGnbv41cwClu5AMW_lX90wMUOZkeZwnbrCat53pFti0ckjLxs7nQJziY4A4vmo$ > > && > tar xpzfv nvhpc_2025_255_Linux_x86_64_cuda_12.9.tar.gz && > nvhpc_2025_255_Linux_x86_64_cuda_12.9/install --silent > --accept-license --instal-dir /opt/nvidia/hpc_sdk > > RUN cd petsc-3.23.5 &&\ > ./configure --prefix=/usr/local/petsc --PETSC_ARCH=arch-linux-c-opt > --with-debugging=0 \ > --with-mpi --download-hypre --with-blaslapack --with-cuda > --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/25.5/cuda && \ > make all install && \ > cd .. > > (...) > ENV > PATH=${NVHPC_ROOT}/compilers/bin:${NVHPC_ROOT}/comm_libs/mpi/bin:${PATH} > ENV MANPATH=${NVHPC_ROOT}/compilers/man:${MANPATH} > ENV LD_LIBRARY_PATH=${NVHPC_ROOT}/compilers/lib:${LD_LIBRARY_PATH} > >>>>>> > > Renato > > On Sat, Aug 2, 2025 at 11:11?AM Satish Balay > wrote: > >> Please send configure.log >> >> presumably you are using a pre-installed mpi from hpc-sdk - that uses NV >> compilers. [and PETSc configure is defaulting to mpicc/mpif90 in PATH] >> >> If you do not have MPI with gcc/gfortran - you can do something like: >> >> ./configure --with-cuda=1 --with-cc=gcc --with-cxx=g++ --with-fc=gfortran >> --download-mpich [other options] >> >> When no options are specified - configure guesses - and picks up what it >> can find - and sometimes that might not be what you intend to use.. >> >> Satish >> >> On Sat, 2 Aug 2025, Renato E Poli wrote: >> >> > It seems that the installer is the one choosing to use nvfortran >> instead of >> > gfortran. >> > >> > I am building a docker container (clean environment). >> > I simply install CUDA SDK from >> > >> https://urldefense.us/v3/__https://developer.download.nvidia.com/hpc-sdk/25.5/nvhpc_2025_255_Linux_x86_64_cuda_12.9.tar.gz__;!!G_uCfscf7eWS!cBTymZpXcpFZ7BHpCGxn0ooq_STD1f9CioiFgRypIYNWg9lpwnmLpddrWk0-J-wQoeTw9Ipu2FPm037mQg_C2w$ >> > , set paths and call petsc configure. >> > >> > RUN cd petsc-3.23.5 &&\ >> > ./configure --prefix=/usr/local/petsc --with-debugging=0 >> --with-mpi >> > --download-hdf5 --download-hypre --with-blaslapack --with-cuda && \ >> > make all && make install >> > >> > Do you see any reason for using the wrong compiler? >> > >> > Renato >> > >> > On Sat, Aug 2, 2025 at 12:28?AM Satish Balay >> wrote: >> > >> > > NVFORTRAN (based on flang-17) does not support some F2008 features >> that >> > > are required here (flang-19+ should work) >> > > >> > > Suggest using gfortran instead. >> > > >> > > Satish >> > > >> > > On Fri, 1 Aug 2025, Renato E Poli wrote: >> > > >> > > > Hi, >> > > > >> > > > I see the error below when compiling with --with-cuda. >> > > > >> > > > Any suggestion? >> > > > >> > > > >> > > > CC arch-linux-c-opt/obj/src/mat/impls/aij/seq/inode2.o >> > > > CC >> arch-linux-c-opt/obj/src/mat/impls/aij/seq/aijperm/aijperm.o >> > > > CC >> > > arch-linux-c-opt/obj/src/mat/impls/aij/seq/mattransposematmult.o >> > > > CC arch-linux-c-opt/obj/src/mat/impls/aij/seq/matptap.o >> > > > CC arch-linux-c-opt/obj/src/mat/impls/aij/seq/matrart.o >> > > > FC arch-linux-c-opt/obj/src/sys/ftn-mod/petscsysmod.o >> > > > NVFORTRAN-S-0034-Syntax error at or near end of line >> > > > >> (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: 6) >> > > > NVFORTRAN-S-0034-Syntax error at or near end of line >> > > > >> (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: >> > > 18) >> > > > NVFORTRAN-S-0034-Syntax error at or near end of line >> > > > >> (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: >> > > 30) >> > > > NVFORTRAN-S-0034-Syntax error at or near end of line >> > > > >> (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: >> > > 42) >> > > > NVFORTRAN-S-0034-Syntax error at or near end of line >> > > > >> (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: >> > > 54) >> > > > NVFORTRAN-S-0034-Syntax error at or near end of line >> > > > >> (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: >> > > 65) >> > > > NVFORTRAN-S-0034-Syntax error at or near end of line >> > > > >> (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: >> > > 76) >> > > > NVFORTRAN-S-0034-Syntax error at or near end of line >> > > > >> (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: >> > > 88) >> > > > NVFORTRAN-S-0034-Syntax error at or near end of line >> > > > >> (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: >> > > 100) >> > > > NVFORTRAN-S-0034-Syntax error at or near end of line >> > > > >> (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: >> > > 112) >> > > > NVFORTRAN-S-0034-Syntax error at or near end of line >> > > > >> (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: >> > > 124) >> > > > NVFORTRAN-S-0034-Syntax error at or near end of line >> > > > >> (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: >> > > 136) >> > > > NVFORTRAN-S-0034-Syntax error at or near end of line >> > > > >> (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: >> > > 148) >> > > > NVFORTRAN-S-0034-Syntax error at or near end of line >> > > > >> (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: >> > > 160) >> > > > NVFORTRAN-S-0034-Syntax error at or near end of line >> > > > >> > > >> (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscviewer.h: 6) >> > > > NVFORTRAN-S-0034-Syntax error at or near end of line >> > > > >> (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscviewer.h: >> > > > 17) >> > > > NVFORTRAN-S-0034-Syntax error at or near end of line >> > > > >> > > >> (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscoptions.h: >> > > > 6) >> > > > NVFORTRAN-S-0034-Syntax error at or near end of line >> > > > >> > > >> (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscoptions.h: >> > > > 18) >> > > > NVFORTRAN-S-0034-Syntax error at or near end of line >> > > > >> > > >> (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscoptions.h: >> > > > 30) >> > > > NVFORTRAN-S-0034-Syntax error at or near end of line >> > > > >> (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petsclog.h: 5) >> > > > NVFORTRAN-S-0034-Syntax error at or near end of line >> > > > >> (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petsclog.h: >> > > 17) >> > > > NVFORTRAN-S-0034-Syntax error at or near end of line >> > > > >> (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petsclog.h: >> > > 29) >> > > > NVFORTRAN-S-0034-Syntax error at or near end of line >> > > > >> > > >> (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscdevice.h: 6) >> > > > NVFORTRAN-S-0034-Syntax error at or near end of line >> > > > >> (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscdevice.h: >> > > > 17) >> > > > 0 inform, 0 warnings, 24 severes, 0 fatal for petscsysdef >> > > > >> > > >> > > >> > >> > -- Stefano -------------- next part -------------- An HTML attachment was scrubbed... URL: From rpoli at utexas.edu Sat Aug 2 12:53:35 2025 From: rpoli at utexas.edu (Renato E Poli) Date: Sat, 2 Aug 2025 12:53:35 -0500 Subject: [petsc-users] Error compiling with cuda In-Reply-To: <7133e686-7e9d-fb90-92e8-637c86e1d405@fastmail.org> References: <69cf94ee-442f-6d8d-2d7a-9368106474a9@fastmail.org> <7133e686-7e9d-fb90-92e8-637c86e1d405@fastmail.org> Message-ID: *>> You can first fix on what MPI to use - and build everyting else with it. (and avoid mixing in multiple MPIs) * It sounds wise to make PETSC point to CUDA MPI. I changed PETSC configure to: $ ./configure --prefix=/usr/local/petsc --PETSC_ARCH=arch-linux-c-opt --with-debugging=0 --with-mpi *--with-mpi-dir=${NVHPC_ROOT}/comm_libs/12.9/openmpi4/latest* --download-hypre --with-blaslapack --with-cuda --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/25.5/cuda && make all && make install And now it seems hypre is not happy:"Cannot use hypre with 64-bit BLAS/LAPACK indices". Any idea on that one? * >> Or use 'cuda toolkit' instead of 'nvhpc-sdk'* We are using nvfortran, which is shipped with nvhpc-sdk AFAIK *>> E4S has containers with PETSc+CUDA installed (via Spack) - I guess you can give it a try * I'll take a look. Thanks! On Sat, Aug 2, 2025 at 12:20?PM Satish Balay wrote: > You can first fix on what MPI to use - and build everyting else with it. > (and avoid mixing in multiple MPIs) > > I've had issues with Ubuntu built MPI - but if it works for you - thats > great! > > You can make sure /usr/bin is ahead in your PATH [before nvhpc-sdk so that > this MPI gets picked up] - or specify explicitly to PETSc with: > > Or use 'cuda toolkit' instead of 'nvhpc-sdk' > > ./configure --with-cc=/usr/bin/mpicc --with-cxx=/usr/bin/mpicxx > --with-fc=/usr/bin/mpif90 --download-hypre > > etc.. > > BTW: PETSc configure attempts to run some binaries to detect > compiler/library features, and such run tests (when linked in with cuda > libraries) can fail if a cuda build is done on a box (container?) without > nvidia driver installed. > > Satish > > > On Sat, 2 Aug 2025, Renato E Poli wrote: > > > Understood. > > > > I believe I am running in circles. > > > > I managed to create the container, but it ended up with multiple MPI > > implementations. > > It seems that CUDA SDK has its own MPI, and polutes the PATH. > > Is there any docker container PETSC+CUDA you would recommend? > > This is a snippet of my Dockerfile, but I am not 100% happy with it. > > > > <<<<<< > > FROM docker.io/ubuntu:24.04 > > # Install necessary build tools and dependencies > > RUN apt-get update && apt-get install -y libmpich-dev gfortran > > libomp-dev (...) > > (...) > > > > ## NVIDIA HPC SDK > > RUN wget > > > https://urldefense.us/v3/__https://developer.download.nvidia.com/hpc-sdk/25.5/nvhpc_2025_255_Linux_x86_64_cuda_12.9.tar.gz__;!!G_uCfscf7eWS!ft2AAfoKdKf-j-6E6nTqtzXHqQEmnbsH7as3hylOIpz7OiONASpD-qNuRNcnDapAFok1YeTP2lLQfH7ne1cFSw$ > > && > > tar xpzfv nvhpc_2025_255_Linux_x86_64_cuda_12.9.tar.gz && > > nvhpc_2025_255_Linux_x86_64_cuda_12.9/install --silent > --accept-license > > --instal-dir /opt/nvidia/hpc_sdk > > > > RUN cd petsc-3.23.5 &&\ > > ./configure --prefix=/usr/local/petsc --PETSC_ARCH=arch-linux-c-opt > > --with-debugging=0 \ > > --with-mpi --download-hypre --with-blaslapack --with-cuda > > --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/25.5/cuda && \ > > make all install && \ > > cd .. > > > > (...) > > ENV > PATH=${NVHPC_ROOT}/compilers/bin:${NVHPC_ROOT}/comm_libs/mpi/bin:${PATH} > > ENV MANPATH=${NVHPC_ROOT}/compilers/man:${MANPATH} > > ENV LD_LIBRARY_PATH=${NVHPC_ROOT}/compilers/lib:${LD_LIBRARY_PATH} > > >>>>>> > > > > Renato > > > > On Sat, Aug 2, 2025 at 11:11?AM Satish Balay > wrote: > > > > > Please send configure.log > > > > > > presumably you are using a pre-installed mpi from hpc-sdk - that uses > NV > > > compilers. [and PETSc configure is defaulting to mpicc/mpif90 in PATH] > > > > > > If you do not have MPI with gcc/gfortran - you can do something like: > > > > > > ./configure --with-cuda=1 --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran > > > --download-mpich [other options] > > > > > > When no options are specified - configure guesses - and picks up what > it > > > can find - and sometimes that might not be what you intend to use.. > > > > > > Satish > > > > > > On Sat, 2 Aug 2025, Renato E Poli wrote: > > > > > > > It seems that the installer is the one choosing to use nvfortran > instead > > > of > > > > gfortran. > > > > > > > > I am building a docker container (clean environment). > > > > I simply install CUDA SDK from > > > > > > > > https://urldefense.us/v3/__https://developer.download.nvidia.com/hpc-sdk/25.5/nvhpc_2025_255_Linux_x86_64_cuda_12.9.tar.gz__;!!G_uCfscf7eWS!cBTymZpXcpFZ7BHpCGxn0ooq_STD1f9CioiFgRypIYNWg9lpwnmLpddrWk0-J-wQoeTw9Ipu2FPm037mQg_C2w$ > > > > , set paths and call petsc configure. > > > > > > > > RUN cd petsc-3.23.5 &&\ > > > > ./configure --prefix=/usr/local/petsc --with-debugging=0 > --with-mpi > > > > --download-hdf5 --download-hypre --with-blaslapack --with-cuda > && \ > > > > make all && make install > > > > > > > > Do you see any reason for using the wrong compiler? > > > > > > > > Renato > > > > > > > > On Sat, Aug 2, 2025 at 12:28?AM Satish Balay > > > > wrote: > > > > > > > > > NVFORTRAN (based on flang-17) does not support some F2008 features > that > > > > > are required here (flang-19+ should work) > > > > > > > > > > Suggest using gfortran instead. > > > > > > > > > > Satish > > > > > > > > > > On Fri, 1 Aug 2025, Renato E Poli wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > I see the error below when compiling with --with-cuda. > > > > > > > > > > > > Any suggestion? > > > > > > > > > > > > > > > > > > CC arch-linux-c-opt/obj/src/mat/impls/aij/seq/inode2.o > > > > > > CC > > > arch-linux-c-opt/obj/src/mat/impls/aij/seq/aijperm/aijperm.o > > > > > > CC > > > > > arch-linux-c-opt/obj/src/mat/impls/aij/seq/mattransposematmult.o > > > > > > CC arch-linux-c-opt/obj/src/mat/impls/aij/seq/matptap.o > > > > > > CC arch-linux-c-opt/obj/src/mat/impls/aij/seq/matrart.o > > > > > > FC arch-linux-c-opt/obj/src/sys/ftn-mod/petscsysmod.o > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > 6) > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > > > 18) > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > > > 30) > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > > > 42) > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > > > 54) > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > > > 65) > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > > > 76) > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > > > 88) > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > > > 100) > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > > > 112) > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > > > 124) > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > > > 136) > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > > > 148) > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > > > 160) > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscviewer.h: 6) > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscviewer.h: > > > > > > 17) > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscoptions.h: > > > > > > 6) > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscoptions.h: > > > > > > 18) > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscoptions.h: > > > > > > 30) > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petsclog.h: > 5) > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petsclog.h: > > > > > 17) > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petsclog.h: > > > > > 29) > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscdevice.h: 6) > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscdevice.h: > > > > > > 17) > > > > > > 0 inform, 0 warnings, 24 severes, 0 fatal for petscsysdef > > > > > > > > > > > > > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay.anl at fastmail.org Sat Aug 2 13:00:23 2025 From: balay.anl at fastmail.org (Satish Balay) Date: Sat, 2 Aug 2025 13:00:23 -0500 (CDT) Subject: [petsc-users] Error compiling with cuda In-Reply-To: References: <69cf94ee-442f-6d8d-2d7a-9368106474a9@fastmail.org> <7133e686-7e9d-fb90-92e8-637c86e1d405@fastmail.org> Message-ID: <3d7315f1-b692-d34a-97c5-22d32acae6f4@fastmail.org> On Sat, 2 Aug 2025, Renato E Poli wrote: > *>> You can first fix on what MPI to use - and build everyting else with > it. (and avoid mixing in multiple MPIs) * > It sounds wise to make PETSC point to CUDA MPI. > I changed PETSC configure to: > > $ ./configure --prefix=/usr/local/petsc --PETSC_ARCH=arch-linux-c-opt > --with-debugging=0 --with-mpi > *--with-mpi-dir=${NVHPC_ROOT}/comm_libs/12.9/openmpi4/latest* > --download-hypre --with-blaslapack --with-cuda > --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/25.5/cuda && make all && > make install > > And now it seems hypre is not happy:"Cannot use hypre with 64-bit > BLAS/LAPACK indices". > Any idea on that one? This is due to linking with 'cuda libraries' on a box without nvidia gpu/driver gives run-time errors. This breaks configure. You can try using: --with-batch=1 > > * >> Or use 'cuda toolkit' instead of 'nvhpc-sdk'* > We are using nvfortran, which is shipped with nvhpc-sdk AFAIK This e-mail thread started with: >>> NVFORTRAN-S-0034-Syntax error at or near end of line (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: 6) <<< And a recommendation to use gfortran instead of nvfortran Satish > > *>> E4S has containers with PETSc+CUDA installed (via Spack) - I guess you > can give it a try * > I'll take a look. Thanks! > > On Sat, Aug 2, 2025 at 12:20?PM Satish Balay wrote: > > > You can first fix on what MPI to use - and build everyting else with it. > > (and avoid mixing in multiple MPIs) > > > > I've had issues with Ubuntu built MPI - but if it works for you - thats > > great! > > > > You can make sure /usr/bin is ahead in your PATH [before nvhpc-sdk so that > > this MPI gets picked up] - or specify explicitly to PETSc with: > > > > Or use 'cuda toolkit' instead of 'nvhpc-sdk' > > > > ./configure --with-cc=/usr/bin/mpicc --with-cxx=/usr/bin/mpicxx > > --with-fc=/usr/bin/mpif90 --download-hypre > > > > etc.. > > > > BTW: PETSc configure attempts to run some binaries to detect > > compiler/library features, and such run tests (when linked in with cuda > > libraries) can fail if a cuda build is done on a box (container?) without > > nvidia driver installed. > > > > Satish > > > > > > On Sat, 2 Aug 2025, Renato E Poli wrote: > > > > > Understood. > > > > > > I believe I am running in circles. > > > > > > I managed to create the container, but it ended up with multiple MPI > > > implementations. > > > It seems that CUDA SDK has its own MPI, and polutes the PATH. > > > Is there any docker container PETSC+CUDA you would recommend? > > > This is a snippet of my Dockerfile, but I am not 100% happy with it. > > > > > > <<<<<< > > > FROM docker.io/ubuntu:24.04 > > > # Install necessary build tools and dependencies > > > RUN apt-get update && apt-get install -y libmpich-dev gfortran > > > libomp-dev (...) > > > (...) > > > > > > ## NVIDIA HPC SDK > > > RUN wget > > > > > https://urldefense.us/v3/__https://developer.download.nvidia.com/hpc-sdk/25.5/nvhpc_2025_255_Linux_x86_64_cuda_12.9.tar.gz__;!!G_uCfscf7eWS!ft2AAfoKdKf-j-6E6nTqtzXHqQEmnbsH7as3hylOIpz7OiONASpD-qNuRNcnDapAFok1YeTP2lLQfH7ne1cFSw$ > > > && > > > tar xpzfv nvhpc_2025_255_Linux_x86_64_cuda_12.9.tar.gz && > > > nvhpc_2025_255_Linux_x86_64_cuda_12.9/install --silent > > --accept-license > > > --instal-dir /opt/nvidia/hpc_sdk > > > > > > RUN cd petsc-3.23.5 &&\ > > > ./configure --prefix=/usr/local/petsc --PETSC_ARCH=arch-linux-c-opt > > > --with-debugging=0 \ > > > --with-mpi --download-hypre --with-blaslapack --with-cuda > > > --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/25.5/cuda && \ > > > make all install && \ > > > cd .. > > > > > > (...) > > > ENV > > PATH=${NVHPC_ROOT}/compilers/bin:${NVHPC_ROOT}/comm_libs/mpi/bin:${PATH} > > > ENV MANPATH=${NVHPC_ROOT}/compilers/man:${MANPATH} > > > ENV LD_LIBRARY_PATH=${NVHPC_ROOT}/compilers/lib:${LD_LIBRARY_PATH} > > > >>>>>> > > > > > > Renato > > > > > > On Sat, Aug 2, 2025 at 11:11?AM Satish Balay > > wrote: > > > > > > > Please send configure.log > > > > > > > > presumably you are using a pre-installed mpi from hpc-sdk - that uses > > NV > > > > compilers. [and PETSc configure is defaulting to mpicc/mpif90 in PATH] > > > > > > > > If you do not have MPI with gcc/gfortran - you can do something like: > > > > > > > > ./configure --with-cuda=1 --with-cc=gcc --with-cxx=g++ > > --with-fc=gfortran > > > > --download-mpich [other options] > > > > > > > > When no options are specified - configure guesses - and picks up what > > it > > > > can find - and sometimes that might not be what you intend to use.. > > > > > > > > Satish > > > > > > > > On Sat, 2 Aug 2025, Renato E Poli wrote: > > > > > > > > > It seems that the installer is the one choosing to use nvfortran > > instead > > > > of > > > > > gfortran. > > > > > > > > > > I am building a docker container (clean environment). > > > > > I simply install CUDA SDK from > > > > > > > > > > > https://urldefense.us/v3/__https://developer.download.nvidia.com/hpc-sdk/25.5/nvhpc_2025_255_Linux_x86_64_cuda_12.9.tar.gz__;!!G_uCfscf7eWS!cBTymZpXcpFZ7BHpCGxn0ooq_STD1f9CioiFgRypIYNWg9lpwnmLpddrWk0-J-wQoeTw9Ipu2FPm037mQg_C2w$ > > > > > , set paths and call petsc configure. > > > > > > > > > > RUN cd petsc-3.23.5 &&\ > > > > > ./configure --prefix=/usr/local/petsc --with-debugging=0 > > --with-mpi > > > > > --download-hdf5 --download-hypre --with-blaslapack --with-cuda > > && \ > > > > > make all && make install > > > > > > > > > > Do you see any reason for using the wrong compiler? > > > > > > > > > > Renato > > > > > > > > > > On Sat, Aug 2, 2025 at 12:28?AM Satish Balay > > > > > > wrote: > > > > > > > > > > > NVFORTRAN (based on flang-17) does not support some F2008 features > > that > > > > > > are required here (flang-19+ should work) > > > > > > > > > > > > Suggest using gfortran instead. > > > > > > > > > > > > Satish > > > > > > > > > > > > On Fri, 1 Aug 2025, Renato E Poli wrote: > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > I see the error below when compiling with --with-cuda. > > > > > > > > > > > > > > Any suggestion? > > > > > > > > > > > > > > > > > > > > > CC arch-linux-c-opt/obj/src/mat/impls/aij/seq/inode2.o > > > > > > > CC > > > > arch-linux-c-opt/obj/src/mat/impls/aij/seq/aijperm/aijperm.o > > > > > > > CC > > > > > > arch-linux-c-opt/obj/src/mat/impls/aij/seq/mattransposematmult.o > > > > > > > CC arch-linux-c-opt/obj/src/mat/impls/aij/seq/matptap.o > > > > > > > CC arch-linux-c-opt/obj/src/mat/impls/aij/seq/matrart.o > > > > > > > FC arch-linux-c-opt/obj/src/sys/ftn-mod/petscsysmod.o > > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > 6) > > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > > > > 18) > > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > > > > 30) > > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > > > > 42) > > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > > > > 54) > > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > > > > 65) > > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > > > > 76) > > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > > > > 88) > > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > > > > 100) > > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > > > > 112) > > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > > > > 124) > > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > > > > 136) > > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > > > > 148) > > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > > > > 160) > > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > > > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscviewer.h: 6) > > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscviewer.h: > > > > > > > 17) > > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > > > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscoptions.h: > > > > > > > 6) > > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > > > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscoptions.h: > > > > > > > 18) > > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > > > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscoptions.h: > > > > > > > 30) > > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petsclog.h: > > 5) > > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petsclog.h: > > > > > > 17) > > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petsclog.h: > > > > > > 29) > > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > > > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscdevice.h: 6) > > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscdevice.h: > > > > > > > 17) > > > > > > > 0 inform, 0 warnings, 24 severes, 0 fatal for petscsysdef > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From balay.anl at fastmail.org Sat Aug 2 13:04:31 2025 From: balay.anl at fastmail.org (Satish Balay) Date: Sat, 2 Aug 2025 13:04:31 -0500 (CDT) Subject: [petsc-users] Error compiling with cuda In-Reply-To: References: <69cf94ee-442f-6d8d-2d7a-9368106474a9@fastmail.org> Message-ID: <0a6b27ea-1ade-8fea-685e-85d822e22786@fastmail.org> On Sat, 2 Aug 2025, Stefano Zampini wrote: > NVIDIA ships OpenMPI. You can select the FORTRAN compiler by using the > OMPI_FC environment variable, see > https://urldefense.us/v3/__https://docs.open-mpi.org/en/v5.0.x/building-apps/customizing-wrappers.html__;!!G_uCfscf7eWS!dVGG_CfOVumpO5kAMWVbjzj4d6nRX1gwuaGnbv41cwClu5AMW_lX90wMUOZkeZwnbrCat53pFti0ckjLxs7nQJzi0CZkLW4$ One potential issue with switching fortran compiler [as mentioned in the above doc] - is fortran modules built by one compiler (nvfortran) - say mpi.mod, won't be usable from the other (gfortran). Satish From rpoli at utexas.edu Sat Aug 2 13:09:32 2025 From: rpoli at utexas.edu (Renato E Poli) Date: Sat, 2 Aug 2025 13:09:32 -0500 Subject: [petsc-users] Error compiling with cuda In-Reply-To: <0a6b27ea-1ade-8fea-685e-85d822e22786@fastmail.org> References: <69cf94ee-442f-6d8d-2d7a-9368106474a9@fastmail.org> <0a6b27ea-1ade-8fea-685e-85d822e22786@fastmail.org> Message-ID: *>> And a recommendation to use gfortran instead of nvfortran* The other application is using nvfortran, and now we are trying to link PETSC to it. I guess I understood the main points. I need to work on it a bit now and get a better idea of what is going on. Thank you! Renato On Sat, Aug 2, 2025 at 1:04?PM Satish Balay wrote: > On Sat, 2 Aug 2025, Stefano Zampini wrote: > > > NVIDIA ships OpenMPI. You can select the FORTRAN compiler by using the > > OMPI_FC environment variable, see > > > https://urldefense.us/v3/__https://docs.open-mpi.org/en/v5.0.x/building-apps/customizing-wrappers.html__;!!G_uCfscf7eWS!dVGG_CfOVumpO5kAMWVbjzj4d6nRX1gwuaGnbv41cwClu5AMW_lX90wMUOZkeZwnbrCat53pFti0ckjLxs7nQJzi0CZkLW4$ > > One potential issue with switching fortran compiler [as mentioned in the > above doc] - is fortran modules built by one compiler (nvfortran) - say > mpi.mod, won't be usable from the other (gfortran). > > Satish > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay.anl at fastmail.org Sat Aug 2 13:24:58 2025 From: balay.anl at fastmail.org (Satish Balay) Date: Sat, 2 Aug 2025 13:24:58 -0500 (CDT) Subject: [petsc-users] Error compiling with cuda In-Reply-To: References: <69cf94ee-442f-6d8d-2d7a-9368106474a9@fastmail.org> <0a6b27ea-1ade-8fea-685e-85d822e22786@fastmail.org> Message-ID: On Sat, 2 Aug 2025, Renato E Poli wrote: > *>> And a recommendation to use gfortran instead of nvfortran* > The other application is using nvfortran, and now we are trying to link > PETSC to it. > > I guess I understood the main points. > I need to work on it a bit now and get a better idea of what is going on. petsc-3.22 should work with nvfortran. Fortran interface in petsc is being over-hauled (in 3.23, and future 3.24 releases) requiring F2008 (and F2023?) features - that some of the older compilers don't support. [and older petsc releases might have other issues - say using latest cuda version, cmake version etc...] Satish From rpoli at utexas.edu Sat Aug 2 13:34:30 2025 From: rpoli at utexas.edu (Renato E Poli) Date: Sat, 2 Aug 2025 13:34:30 -0500 Subject: [petsc-users] Error compiling with cuda In-Reply-To: References: <69cf94ee-442f-6d8d-2d7a-9368106474a9@fastmail.org> <0a6b27ea-1ade-8fea-685e-85d822e22786@fastmail.org> Message-ID: If I understand correctly, with gfotran I will be able to link to petsc cuda-c++ , but I wont be able to use the fortran cuda interface directly (so I wont be able to write my own gpu-accelerated fortran). Is that correct? On Sat, Aug 2, 2025, 13:25 Satish Balay wrote: > On Sat, 2 Aug 2025, Renato E Poli wrote: > > > *>> And a recommendation to use gfortran instead of nvfortran* > > The other application is using nvfortran, and now we are trying to link > > PETSC to it. > > > > I guess I understood the main points. > > I need to work on it a bit now and get a better idea of what is going on. > > petsc-3.22 should work with nvfortran. > > Fortran interface in petsc is being over-hauled (in 3.23, and future 3.24 > releases) requiring F2008 (and F2023?) features - that some of the older > compilers don't support. > > [and older petsc releases might have other issues - say using latest cuda > version, cmake version etc...] > > Satish > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay.anl at fastmail.org Sat Aug 2 13:48:12 2025 From: balay.anl at fastmail.org (Satish Balay) Date: Sat, 2 Aug 2025 13:48:12 -0500 (CDT) Subject: [petsc-users] Error compiling with cuda In-Reply-To: References: <69cf94ee-442f-6d8d-2d7a-9368106474a9@fastmail.org> <0a6b27ea-1ade-8fea-685e-85d822e22786@fastmail.org> Message-ID: <440dcf6b-bd99-0f39-bfc0-fb512d934ede@fastmail.org> On Sat, 2 Aug 2025, Renato E Poli wrote: > If I understand correctly, with gfotran I will be able to link to petsc > cuda-c++ , but I wont be able to use the fortran cuda interface directly > (so I wont be able to write my own gpu-accelerated fortran). Is that > correct? I think so, nvfortran (with pgf90?) might be the only compiler that supports cuda fortran kernels - but will defer to experts here.. Satish From kiranhegde75 at gmail.com Sat Aug 2 22:56:07 2025 From: kiranhegde75 at gmail.com (kiran hegde) Date: Sun, 3 Aug 2025 09:26:07 +0530 Subject: [petsc-users] petsc-users Digest, Vol 200, Issue 3 In-Reply-To: References: Message-ID: Hi, Please find the attached configure.log file. PETSC installation successfully completes with make check, without any error. But, when compiling fortan codes with petsc library+include in the makefile i get this error. Regards, Kiran On Sat, Aug 2, 2025 at 9:42?PM wrote: > Send petsc-users mailing list submissions to > petsc-users at mcs.anl.gov > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.mcs.anl.gov/mailman/listinfo/petsc-users > or, via email, send a message with subject or body 'help' to > petsc-users-request at mcs.anl.gov > > You can reach the person managing the list at > petsc-users-owner at mcs.anl.gov > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of petsc-users digest..." > > > Today's Topics: > > 1. missing binary operator before token (kiran hegde) > 2. Re: missing binary operator before token (Satish Balay) > 3. Re: Error compiling with cuda (Satish Balay) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sat, 2 Aug 2025 20:51:59 +0530 > From: kiran hegde > To: petsc-users at mcs.anl.gov > Subject: [petsc-users] missing binary operator before token > Message-ID: > 1jYciyLHRc_A at mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hi , > > I am compiling fortran code using petsc include & lib, but getting > below error: > > 298 | #if PetscHasAttribute(format) && > !defined(PETSC_SKIP_ATTRIBUTE_FORMAT) > Error: missing binary operator before token '(' > > /home/kiranhegde/MyLinuxApplications/Installations/petsc/include/petscmacros.h:330:0: > > 330 | #if PetscHasAttribute(__cold__) > Error: missing binary operator before token '(' > > /home/kiranhegde/MyLinuxApplications/Installations/petsc/include/petscmacros.h:332:0: > > 332 | #elif PetscHasAttribute(cold) /* some implementations (old gcc) use > no underscores */ > Error: missing binary operator before token '(' > > /home/kiranhegde/MyLinuxApplications/Installations/petsc/include/petscmacros.h:355:0: > > 298 | #if PetscHasAttribute(format) && > !defined(PETSC_SKIP_ATTRIBUTE_FORMAT) > Error: missing binary operator before token '(' > > /home/kiranhegde/MyLinuxApplications/Installations/petsc/include/petscmacros.h:330:0: > > 330 | #if PetscHasAttribute(__cold__) > Error: missing binary operator before token '(' > > /home/kiranhegde/MyLinuxApplications/Installations/petsc/include/petscmacros.h:332:0: > > 332 | #elif PetscHasAttribute(cold) /* some implementations (old gcc) use > no underscores */ > Error: missing binary operator before token '(' > > /home/kiranhegde/MyLinuxApplications/Installations/petsc/include/petscmacros.h:355:0: > > Same errors repeat for many lines and header files > I am using PETSC at 3.23.4 version with gcc at 15.1.0 > > > > -- > Regards, > > Kiran Hegde > Mobile:+91 9972600235 > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20250802/21975a36/attachment-0001.html > > > > ------------------------------ > > Message: 2 > Date: Sat, 2 Aug 2025 11:04:17 -0500 (CDT) > From: Satish Balay > To: kiran hegde > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] missing binary operator before token > Message-ID: > Content-Type: text/plain; charset=US-ASCII > > Do you get these errors with 'make check' on a petsc install - or when > compiling a petsc fortran example (with the corresponding makefile)? > > Satish > > On Sat, 2 Aug 2025, kiran hegde wrote: > > > Hi , > > > > I am compiling fortran code using petsc include & lib, but getting > > below error: > > > > 298 | #if PetscHasAttribute(format) && > !defined(PETSC_SKIP_ATTRIBUTE_FORMAT) > > Error: missing binary operator before token '(' > > > /home/kiranhegde/MyLinuxApplications/Installations/petsc/include/petscmacros.h:330:0: > > > > 330 | #if PetscHasAttribute(__cold__) > > Error: missing binary operator before token '(' > > > /home/kiranhegde/MyLinuxApplications/Installations/petsc/include/petscmacros.h:332:0: > > > > 332 | #elif PetscHasAttribute(cold) /* some implementations (old gcc) > use > > no underscores */ > > Error: missing binary operator before token '(' > > > /home/kiranhegde/MyLinuxApplications/Installations/petsc/include/petscmacros.h:355:0: > > > > 298 | #if PetscHasAttribute(format) && > !defined(PETSC_SKIP_ATTRIBUTE_FORMAT) > > Error: missing binary operator before token '(' > > > /home/kiranhegde/MyLinuxApplications/Installations/petsc/include/petscmacros.h:330:0: > > > > 330 | #if PetscHasAttribute(__cold__) > > Error: missing binary operator before token '(' > > > /home/kiranhegde/MyLinuxApplications/Installations/petsc/include/petscmacros.h:332:0: > > > > 332 | #elif PetscHasAttribute(cold) /* some implementations (old gcc) > use > > no underscores */ > > Error: missing binary operator before token '(' > > > /home/kiranhegde/MyLinuxApplications/Installations/petsc/include/petscmacros.h:355:0: > > > > Same errors repeat for many lines and header files > > I am using PETSC at 3.23.4 version with gcc at 15.1.0 > > > > > > > > > > > > ------------------------------ > > Message: 3 > Date: Sat, 2 Aug 2025 11:11:48 -0500 (CDT) > From: Satish Balay > To: Renato E Poli > Cc: petsc-users > Subject: Re: [petsc-users] Error compiling with cuda > Message-ID: > Content-Type: text/plain; charset="utf-8" > > Please send configure.log > > presumably you are using a pre-installed mpi from hpc-sdk - that uses NV > compilers. [and PETSc configure is defaulting to mpicc/mpif90 in PATH] > > If you do not have MPI with gcc/gfortran - you can do something like: > > ./configure --with-cuda=1 --with-cc=gcc --with-cxx=g++ --with-fc=gfortran > --download-mpich [other options] > > When no options are specified - configure guesses - and picks up what it > can find - and sometimes that might not be what you intend to use.. > > Satish > > On Sat, 2 Aug 2025, Renato E Poli wrote: > > > It seems that the installer is the one choosing to use nvfortran instead > of > > gfortran. > > > > I am building a docker container (clean environment). > > I simply install CUDA SDK from > > > https://urldefense.us/v3/__https://developer.download.nvidia.com/hpc-sdk/25.5/nvhpc_2025_255_Linux_x86_64_cuda_12.9.tar.gz__;!!G_uCfscf7eWS!cBTymZpXcpFZ7BHpCGxn0ooq_STD1f9CioiFgRypIYNWg9lpwnmLpddrWk0-J-wQoeTw9Ipu2FPm037mQg_C2w$ > > , set paths and call petsc configure. > > > > RUN cd petsc-3.23.5 &&\ > > ./configure --prefix=/usr/local/petsc --with-debugging=0 --with-mpi > > --download-hdf5 --download-hypre --with-blaslapack --with-cuda && \ > > make all && make install > > > > Do you see any reason for using the wrong compiler? > > > > Renato > > > > On Sat, Aug 2, 2025 at 12:28?AM Satish Balay > wrote: > > > > > NVFORTRAN (based on flang-17) does not support some F2008 features that > > > are required here (flang-19+ should work) > > > > > > Suggest using gfortran instead. > > > > > > Satish > > > > > > On Fri, 1 Aug 2025, Renato E Poli wrote: > > > > > > > Hi, > > > > > > > > I see the error below when compiling with --with-cuda. > > > > > > > > Any suggestion? > > > > > > > > > > > > CC arch-linux-c-opt/obj/src/mat/impls/aij/seq/inode2.o > > > > CC > arch-linux-c-opt/obj/src/mat/impls/aij/seq/aijperm/aijperm.o > > > > CC > > > arch-linux-c-opt/obj/src/mat/impls/aij/seq/mattransposematmult.o > > > > CC arch-linux-c-opt/obj/src/mat/impls/aij/seq/matptap.o > > > > CC arch-linux-c-opt/obj/src/mat/impls/aij/seq/matrart.o > > > > FC arch-linux-c-opt/obj/src/sys/ftn-mod/petscsysmod.o > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: 6) > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > 18) > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > 30) > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > 42) > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > 54) > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > 65) > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > 76) > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > 88) > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > 100) > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > 112) > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > 124) > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > 136) > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > 148) > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: > > > 160) > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscviewer.h: 6) > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscviewer.h: > > > > 17) > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscoptions.h: > > > > 6) > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscoptions.h: > > > > 18) > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscoptions.h: > > > > 30) > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petsclog.h: 5) > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petsclog.h: > > > 17) > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petsclog.h: > > > 29) > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscdevice.h: 6) > > > > NVFORTRAN-S-0034-Syntax error at or near end of line > > > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscdevice.h: > > > > 17) > > > > 0 inform, 0 warnings, 24 severes, 0 fatal for petscsysdef > > > > > > > > > > > > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > petsc-users mailing list > petsc-users at mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/petsc-users > > > ------------------------------ > > End of petsc-users Digest, Vol 200, Issue 3 > ******************************************* > -- Regards, Kiran Hegde Mobile:+91 9972600235 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: text/x-log Size: 2142665 bytes Desc: not available URL: From rpoli at utexas.edu Sun Aug 3 07:55:42 2025 From: rpoli at utexas.edu (Renato E Poli) Date: Sun, 3 Aug 2025 07:55:42 -0500 Subject: [petsc-users] Error compiling with cuda In-Reply-To: <440dcf6b-bd99-0f39-bfc0-fb512d934ede@fastmail.org> References: <69cf94ee-442f-6d8d-2d7a-9368106474a9@fastmail.org> <0a6b27ea-1ade-8fea-685e-85d822e22786@fastmail.org> <440dcf6b-bd99-0f39-bfc0-fb512d934ede@fastmail.org> Message-ID: Hi Satish, I could finally compile PETSc, and it seemed to build. I can compile and CUDA drivers look ok. However, configure.log shows many errors or potential errors (attached). Is that supposed to be? Thank you, Renato On Sat, Aug 2, 2025 at 1:48?PM Satish Balay wrote: > On Sat, 2 Aug 2025, Renato E Poli wrote: > > > If I understand correctly, with gfotran I will be able to link to petsc > > cuda-c++ , but I wont be able to use the fortran cuda interface directly > > (so I wont be able to write my own gpu-accelerated fortran). Is that > > correct? > > I think so, nvfortran (with pgf90?) might be the only compiler that > supports cuda fortran kernels - but will defer to experts here.. > > Satish > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log.gz Type: application/x-gzip Size: 384219 bytes Desc: not available URL: From balay.anl at fastmail.org Sun Aug 3 11:22:35 2025 From: balay.anl at fastmail.org (Satish Balay) Date: Sun, 3 Aug 2025 11:22:35 -0500 (CDT) Subject: [petsc-users] Error compiling with cuda In-Reply-To: References: <69cf94ee-442f-6d8d-2d7a-9368106474a9@fastmail.org> <0a6b27ea-1ade-8fea-685e-85d822e22786@fastmail.org> <440dcf6b-bd99-0f39-bfc0-fb512d934ede@fastmail.org> Message-ID: <9c0cc157-66c0-c87a-ade8-2bc9b576d8a3@fastmail.org> On Sun, 3 Aug 2025, Renato E Poli wrote: > Hi Satish, > > I could finally compile PETSc, and it seemed to build. > I can compile and CUDA drivers look ok. > However, configure.log shows many errors or potential errors (attached). > Is that supposed to be? Looks fine to me. configure runs various tests to checks for success or failure - to determine parameters [so there will be some failures listed in the log] Satish From balay.anl at fastmail.org Sun Aug 3 11:26:53 2025 From: balay.anl at fastmail.org (Satish Balay) Date: Sun, 3 Aug 2025 11:26:53 -0500 (CDT) Subject: [petsc-users] petsc-users Digest, Vol 200, Issue 3 In-Reply-To: References: Message-ID: On Sun, 3 Aug 2025, kiran hegde wrote: > Hi, > > Please find the attached configure.log file. > PETSC installation successfully completes with make check, without any > error. > But, when compiling fortan codes with petsc library+include in the > makefile i get this error. Then likely your compile command is missing some options. you can: - compile a petsc example - say: cd src/snes/tutorials make ex5f - now compile your application - And compare the compile commands and see if there are any differences (missing options, wrong order of include files etc..) Satish From rpoli at utexas.edu Sun Aug 3 11:33:57 2025 From: rpoli at utexas.edu (Renato E Poli) Date: Sun, 3 Aug 2025 11:33:57 -0500 Subject: [petsc-users] Error compiling with cuda In-Reply-To: <9c0cc157-66c0-c87a-ade8-2bc9b576d8a3@fastmail.org> References: <69cf94ee-442f-6d8d-2d7a-9368106474a9@fastmail.org> <0a6b27ea-1ade-8fea-685e-85d822e22786@fastmail.org> <440dcf6b-bd99-0f39-bfc0-fb512d934ede@fastmail.org> <9c0cc157-66c0-c87a-ade8-2bc9b576d8a3@fastmail.org> Message-ID: Now things look smooth indeed. I had to downgrade to *ubuntu 22* , *CUDA 11.8 *, and *openMPI 4* Should I expect trouble? Any chance new PETSC distros will support NVFORTRAN soon? Renato On Sun, Aug 3, 2025 at 11:22?AM Satish Balay wrote: > On Sun, 3 Aug 2025, Renato E Poli wrote: > > > Hi Satish, > > > > I could finally compile PETSc, and it seemed to build. > > I can compile and CUDA drivers look ok. > > However, configure.log shows many errors or potential errors (attached). > > Is that supposed to be? > > Looks fine to me. > > configure runs various tests to checks for success or failure - to > determine parameters [so there will be some failures listed in the log] > > Satish > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay.anl at fastmail.org Sun Aug 3 11:49:59 2025 From: balay.anl at fastmail.org (Satish Balay) Date: Sun, 3 Aug 2025 11:49:59 -0500 (CDT) Subject: [petsc-users] Error compiling with cuda In-Reply-To: References: <69cf94ee-442f-6d8d-2d7a-9368106474a9@fastmail.org> <0a6b27ea-1ade-8fea-685e-85d822e22786@fastmail.org> <440dcf6b-bd99-0f39-bfc0-fb512d934ede@fastmail.org> <9c0cc157-66c0-c87a-ade8-2bc9b576d8a3@fastmail.org> Message-ID: On Sun, 3 Aug 2025, Renato E Poli wrote: > Any chance new PETSC distros will support NVFORTRAN soon? Its more of 'when will nvfortran support newer fortran features' For ex: flang-new 19+ versions support these fatures, and compilers that are based on older flang versions fail. AMD compilers are clang/flang based (v17) so whenever they get to rebased over llvm-19+ or newer - they are likely to work https://urldefense.us/v3/__https://www.amd.com/en/developer/aocc.html__;!!G_uCfscf7eWS!c0ugVETwGi3h7i8tmUTHAGKZzhxVNgTlepWYpIYDBZI2CifAqMH3FSRhyXg1QihvrYMt96C8F7ezT5vcZR084UqOnjA$ But I don't know if/when nvfortran is migrating to flang/llvm (or not). petsc main (git) branch has more fortran changes (and more in the pipeline) - so you can try a build on it to check on the current outstanding issues. Satish From rpoli at utexas.edu Sun Aug 3 12:01:11 2025 From: rpoli at utexas.edu (Renato E Poli) Date: Sun, 3 Aug 2025 12:01:11 -0500 Subject: [petsc-users] Error compiling with cuda In-Reply-To: References: <69cf94ee-442f-6d8d-2d7a-9368106474a9@fastmail.org> <0a6b27ea-1ade-8fea-685e-85d822e22786@fastmail.org> <440dcf6b-bd99-0f39-bfc0-fb512d934ede@fastmail.org> <9c0cc157-66c0-c87a-ade8-2bc9b576d8a3@fastmail.org> Message-ID: On Sun, Aug 3, 2025, 11:50 Satish Balay wrote: > On Sun, 3 Aug 2025, Renato E Poli wrote: > > > Any chance new PETSC distros will support NVFORTRAN soon? > > Its more of 'when will nvfortran support newer fortran features' > True. Bad phrasing on my side. Sorry. For ex: flang-new 19+ versions support these fatures, and compilers that > are based on older flang versions fail. AMD compilers are clang/flang based > (v17) so whenever they get to rebased over llvm-19+ or newer - they are > likely to work > > https://urldefense.us/v3/__https://www.amd.com/en/developer/aocc.html__;!!G_uCfscf7eWS!ck1VfD0I9WshE4o1viWmwKxMeTg7z62UFyr_NGZwbKtOdUxUMf1U8YMZgvfyvRtIQtptc2LooGXJAQVxX6nyIA$ > > But I don't know if/when nvfortran is migrating to flang/llvm (or not). > > petsc main (git) branch has more fortran changes (and more in the > pipeline) - so you can try a build on it to check on the current > outstanding issues. > Thank you! > Satish > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kiranhegde75 at gmail.com Mon Aug 4 10:15:55 2025 From: kiranhegde75 at gmail.com (kiran hegde) Date: Mon, 4 Aug 2025 20:45:55 +0530 Subject: [petsc-users] petsc-users Digest, Vol 200, Issue 3 In-Reply-To: References: Message-ID: Hi Satish, I have tried the options ex5f, but getting the same error. The code was running without any issues long back and I was using petsc ver 3.9 using gcc 12.X.X. My makefile flags are : FFLAGS = -march=native -O3 -pipe -ffree-form -cpp -ffree-line-length-none \ -Wall -Wextra -Wtabs -Wintrinsics-std -Wsurprising -Waliasing \ -Wno-unused-parameter -Wno-unused-dummy-argument \ -D$(MODEL) \ -DVERSION=\"$(GIT_VERSION)\" \ -I$(PETSC_DIR)/include FPPFLAGS = -cpp -E -dM -C -P -D$(MODEL) LDFLAGS = -lpetsc -L$(PETSC_DIR)/lib LDFLAGS += -Wl,-rpath=$(PETSC_DIR)/lib Have i missed anything? Regards, Kiran On Sun, Aug 3, 2025 at 9:56?PM Satish Balay wrote: > On Sun, 3 Aug 2025, kiran hegde wrote: > > > Hi, > > > > Please find the attached configure.log file. > > PETSC installation successfully completes with make check, without any > > error. > > But, when compiling fortan codes with petsc library+include in the > > makefile i get this error. > > Then likely your compile command is missing some options. you can: > > - compile a petsc example - say: > cd src/snes/tutorials > make ex5f > > - now compile your application > > - And compare the compile commands and see if there are any differences > (missing options, wrong order of include files etc..) > > Satish > -- Regards, Kiran Hegde Mobile:+91 9972600235 -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay.anl at fastmail.org Mon Aug 4 12:12:50 2025 From: balay.anl at fastmail.org (Satish Balay) Date: Mon, 4 Aug 2025 12:12:50 -0500 (CDT) Subject: [petsc-users] petsc-users Digest, Vol 200, Issue 3 In-Reply-To: References: Message-ID: my suggestion was to "compare compile commands" between a petsc example (via petsc makefile) and your application - with your makefile Its best to use petsc formatted makefiles to avoid issues with compiler option mismatch https://urldefense.us/v3/__https://petsc.org/release/manual/getting_started/*writing-c-c-or-fortran-applications__;Iw!!G_uCfscf7eWS!fLLihqTez1EMMjLzlLEt8-aKcPazlSe4eGJSEfCb2j3P02GdALJLZKL-JF1E3SXUP8LjG4aB2jocfKGtWIR-Yqfxclk$ Either way - you might want to start with a currently working PETSc example (using a PETSc makefile) - and modify the makefile to your desired format - and see if it still works (with this petsc exampe code, and then use this updated makefile format with your application code) Also if your code was with v3.9 - then there are many changes in v3.23, so you would have to check the current example codes wrt how to use '#include', 'use' directives and other usages. Satish On Mon, 4 Aug 2025, kiran hegde wrote: > Hi Satish, > > I have tried the options ex5f, but getting the same error. > The code was running without any issues long back and I was using petsc > ver 3.9 using gcc 12.X.X. > My makefile flags are : > > FFLAGS = -march=native -O3 -pipe -ffree-form -cpp -ffree-line-length-none \ > -Wall -Wextra -Wtabs -Wintrinsics-std -Wsurprising > -Waliasing \ > -Wno-unused-parameter -Wno-unused-dummy-argument \ > -D$(MODEL) \ > -DVERSION=\"$(GIT_VERSION)\" \ > -I$(PETSC_DIR)/include > FPPFLAGS = -cpp -E -dM -C -P -D$(MODEL) > LDFLAGS = -lpetsc -L$(PETSC_DIR)/lib > LDFLAGS += -Wl,-rpath=$(PETSC_DIR)/lib > > Have i missed anything? > > > Regards, > Kiran > > > > > On Sun, Aug 3, 2025 at 9:56?PM Satish Balay wrote: > > > On Sun, 3 Aug 2025, kiran hegde wrote: > > > > > Hi, > > > > > > Please find the attached configure.log file. > > > PETSC installation successfully completes with make check, without any > > > error. > > > But, when compiling fortan codes with petsc library+include in the > > > makefile i get this error. > > > > Then likely your compile command is missing some options. you can: > > > > - compile a petsc example - say: > > cd src/snes/tutorials > > make ex5f > > > > - now compile your application > > > > - And compare the compile commands and see if there are any differences > > (missing options, wrong order of include files etc..) > > > > Satish > > > > > From bsmith at petsc.dev Mon Aug 4 19:30:52 2025 From: bsmith at petsc.dev (Barry Smith) Date: Mon, 4 Aug 2025 20:30:52 -0400 Subject: [petsc-users] [GPU] Jacobi preconditioner In-Reply-To: References: <386853b1efae4269919b977b88c7e679@cea.fr> <49396000-D752-4C95-AF1B-524EC68BC5BC@petsc.dev> Message-ID: I thought the CUPM code was intended to allow us to have common code between NVIDIA and AMD? > On Jul 31, 2025, at 11:05?AM, Junchao Zhang wrote: > > What would embarrass me more is to copy the same code to MatGetDiagonal_SeqAIJHIPSPARSE. > > --Junchao Zhang > > On Wed, Jul 30, 2025 at 1:34?PM Barry Smith > wrote: >> >> We absolutely should have a MatGetDiagonal_SeqAIJCUSPARSE(). It's somewhat embarrassing that we don't provide this. >> >> I have found some potential code at https://urldefense.us/v3/__https://stackoverflow.com/questions/60311408/how-to-get-the-diagonal-of-a-sparse-matrix-in-cusparse__;!!G_uCfscf7eWS!eLhrliC3tod70b6fK9DQP7SuUvKtsIneNpgXgVdXLWFXMXRsKAfGlg0YEjNrjJFgq5MlEJNOJYyHZwCl9AoymJ4$ >> >> Barry >> >> >> >> >>> On Jul 28, 2025, at 11:43?AM, Junchao Zhang > wrote: >>> >>> Yes, MatGetDiagonal_SeqAIJCUSPARSE hasn't been implemented. petsc/cuda and petsc/kokkos backends are separate code. >>> If petsc/kokkos meet your needs, then just use them. For petsc users, we hope it will be just a difference of extra --download-kokkos --download-kokkos-kernels in configuration. >>> >>> --Junchao Zhang >>> >>> >>> On Mon, Jul 28, 2025 at 2:51?AM LEDAC Pierre > wrote: >>>> Hello all, >>>> >>>> >>>> >>>> We are solving with PETSc a linear system updated every time step (constant stencil but coefficients changing). >>>> >>>> >>>> >>>> The matrix is preallocated once with MatSetPreallocationCOO() then filled each time step with MatSetValuesCOO() and we use device pointers for coo_i, coo_j, and coefficients values. >>>> >>>> >>>> >>>> It is working fine with a GMRES Ksp solver and PC Jacobi but we are surprised to see that every time step, during PCSetUp, MatGetDiagonal_SeqAIJ is called whereas the matrix is on the device. Looking at the API, it seems there is no MatGetDiagonal_SeqAIJCUSPARSE() but a MatGetDiagonal_SeqAIJKOKKOS(). >>>> >>>> >>>> >>>> Does it mean we should use Kokkos backend in PETSc to have Jacobi preconditioner built directly on device ? Or I am doing something wrong ? >>>> >>>> NB: Gmres is running well on device. >>>> >>>> >>>> >>>> I could use -ksp_reuse_preconditioner to avoid Jacobi being recreated each solve on host but it increases significantly the number of iterations. >>>> >>>> >>>> >>>> Thanks, >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> Pierre LEDAC >>>> Commissariat ? l??nergie atomique et aux ?nergies alternatives >>>> Centre de SACLAY >>>> DES/ISAS/DM2S/SGLS/LCAN >>>> B?timent 451 ? point courrier n?41 >>>> F-91191 Gif-sur-Yvette >>>> +33 1 69 08 04 03 >>>> +33 6 83 42 05 79 >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Mon Aug 4 19:41:21 2025 From: bsmith at petsc.dev (Barry Smith) Date: Mon, 4 Aug 2025 20:41:21 -0400 Subject: [petsc-users] petsc-users Digest, Vol 200, Issue 3 In-Reply-To: References: Message-ID: "I have tried the options ex5f, but getting the same error." So you did cd src/snes/tutorials/ make ex5f and got some errors? Please cut and paste all the output from make ex5f and sent it here. > On Aug 4, 2025, at 11:15?AM, kiran hegde wrote: > > Hi Satish, > > I have tried the options ex5f, but getting the same error. > The code was running without any issues long back and I was using petsc ver 3.9 using gcc 12.X.X. > My makefile flags are : > > FFLAGS = -march=native -O3 -pipe -ffree-form -cpp -ffree-line-length-none \ > -Wall -Wextra -Wtabs -Wintrinsics-std -Wsurprising -Waliasing \ > -Wno-unused-parameter -Wno-unused-dummy-argument \ > -D$(MODEL) \ > -DVERSION=\"$(GIT_VERSION)\" \ > -I$(PETSC_DIR)/include > FPPFLAGS = -cpp -E -dM -C -P -D$(MODEL) > LDFLAGS = -lpetsc -L$(PETSC_DIR)/lib > LDFLAGS += -Wl,-rpath=$(PETSC_DIR)/lib > > Have i missed anything? > > > Regards, > Kiran > > > > > On Sun, Aug 3, 2025 at 9:56?PM Satish Balay > wrote: >> On Sun, 3 Aug 2025, kiran hegde wrote: >> >> > Hi, >> > >> > Please find the attached configure.log file. >> > PETSC installation successfully completes with make check, without any >> > error. >> > But, when compiling fortan codes with petsc library+include in the >> > makefile i get this error. >> >> Then likely your compile command is missing some options. you can: >> >> - compile a petsc example - say: >> cd src/snes/tutorials >> make ex5f >> >> - now compile your application >> >> - And compare the compile commands and see if there are any differences (missing options, wrong order of include files etc..) >> >> Satish > > > > -- > Regards, > > Kiran Hegde > Mobile:+91 9972600235 -------------- next part -------------- An HTML attachment was scrubbed... URL: From junchao.zhang at gmail.com Mon Aug 4 22:27:46 2025 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Mon, 4 Aug 2025 22:27:46 -0500 Subject: [petsc-users] [GPU] Jacobi preconditioner In-Reply-To: References: <386853b1efae4269919b977b88c7e679@cea.fr> <49396000-D752-4C95-AF1B-524EC68BC5BC@petsc.dev> Message-ID: CUPM is half baked, and only supports Vec and MATDENSE. --Junchao Zhang On Mon, Aug 4, 2025 at 7:31?PM Barry Smith wrote: > > I thought the CUPM code was intended to allow us to have common code > between NVIDIA and AMD? > > > On Jul 31, 2025, at 11:05?AM, Junchao Zhang > wrote: > > What would embarrass me more is to copy the same code to MatGetDiagonal_ > SeqAIJHIPSPARSE. > > --Junchao Zhang > > On Wed, Jul 30, 2025 at 1:34?PM Barry Smith wrote: > >> >> We absolutely should have a MatGetDiagonal_SeqAIJCUSPARSE(). >> It's somewhat embarrassing that we don't provide this. >> >> I have found some potential code at >> https://urldefense.us/v3/__https://stackoverflow.com/questions/60311408/how-to-get-the-diagonal-of-a-sparse-matrix-in-cusparse__;!!G_uCfscf7eWS!azrBQb7GZ-x-aytrnryDJoGQom1TGFZk4Q0BN-DKMOXzaNODwCt5or8ssxWJZb9IxgnQ-uKCTdvZvBX0xN2oESk0qS74$ >> >> Barry >> >> >> >> >> On Jul 28, 2025, at 11:43?AM, Junchao Zhang >> wrote: >> >> Yes, MatGetDiagonal_SeqAIJCUSPARSE hasn't been implemented. petsc/cuda >> and petsc/kokkos backends are separate code. >> If petsc/kokkos meet your needs, then just use them. For petsc users, we >> hope it will be just a difference of extra --download-kokkos >> --download-kokkos-kernels in configuration. >> >> --Junchao Zhang >> >> >> On Mon, Jul 28, 2025 at 2:51?AM LEDAC Pierre wrote: >> >>> Hello all, >>> >>> >>> We are solving with PETSc a linear system updated every time step >>> (constant stencil but coefficients changing). >>> >>> >>> The matrix is preallocated once with MatSetPreallocationCOO() then >>> filled each time step with MatSetValuesCOO() and we use device pointers >>> for coo_i, coo_j, and coefficients values. >>> >>> >>> It is working fine with a GMRES Ksp solver and PC Jacobi but we are >>> surprised to see that every time step, during PCSetUp, >>> MatGetDiagonal_SeqAIJ is called whereas the matrix is on the device. >>> Looking at the API, it seems there is no MatGetDiagonal_SeqAIJCUSPARSE() >>> but a MatGetDiagonal_SeqAIJKOKKOS(). >>> >>> >>> Does it mean we should use Kokkos backend in PETSc to have Jacobi >>> preconditioner built directly on device ? Or I am doing something wrong ? >>> >>> NB: Gmres is running well on device. >>> >>> >>> I could use -ksp_reuse_preconditioner to avoid Jacobi being recreated >>> each solve on host but it increases significantly the number of iterations. >>> >>> >>> Thanks, >>> >>> >>> >>> >>> >>> >>> Pierre LEDAC >>> Commissariat ? l??nergie atomique et aux ?nergies alternatives >>> Centre de SACLAY >>> DES/ISAS/DM2S/SGLS/LCAN >>> B?timent 451 ? point courrier n?41 >>> F-91191 Gif-sur-Yvette >>> +33 1 69 08 04 03 >>> +33 6 83 42 05 79 >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kiranhegde75 at gmail.com Mon Aug 4 23:44:48 2025 From: kiranhegde75 at gmail.com (kiran hegde) Date: Tue, 5 Aug 2025 10:14:48 +0530 Subject: [petsc-users] petsc-users Digest, Vol 200, Issue 3 In-Reply-To: References: Message-ID: Hi, Sorry, the tutorial case run succeeded. No errors reported. When i used the makefile file flags in my code, my code is still reported error. Regards, Kiran Hegde Mobile:+91 9972600235 On Tue, Aug 5, 2025, 6:11?AM Barry Smith wrote: > > "I have tried the options ex5f, but getting the same error." > > So you did > > cd src/snes/tutorials/ > make ex5f > > and got some errors? Please cut and paste all the output from make > ex5f and sent it here. > > > > On Aug 4, 2025, at 11:15?AM, kiran hegde wrote: > > Hi Satish, > > I have tried the options ex5f, but getting the same error. > The code was running without any issues long back and I was using petsc > ver 3.9 using gcc 12.X.X. > My makefile flags are : > > FFLAGS = -march=native -O3 -pipe -ffree-form -cpp -ffree-line-length-none \ > -Wall -Wextra -Wtabs -Wintrinsics-std -Wsurprising > -Waliasing \ > -Wno-unused-parameter -Wno-unused-dummy-argument \ > -D$(MODEL) \ > -DVERSION=\"$(GIT_VERSION)\" \ > -I$(PETSC_DIR)/include > FPPFLAGS = -cpp -E -dM -C -P -D$(MODEL) > LDFLAGS = -lpetsc -L$(PETSC_DIR)/lib > LDFLAGS += -Wl,-rpath=$(PETSC_DIR)/lib > > Have i missed anything? > > > Regards, > Kiran > > > > > On Sun, Aug 3, 2025 at 9:56?PM Satish Balay > wrote: > >> On Sun, 3 Aug 2025, kiran hegde wrote: >> >> > Hi, >> > >> > Please find the attached configure.log file. >> > PETSC installation successfully completes with make check, without any >> > error. >> > But, when compiling fortan codes with petsc library+include in the >> > makefile i get this error. >> >> Then likely your compile command is missing some options. you can: >> >> - compile a petsc example - say: >> cd src/snes/tutorials >> make ex5f >> >> - now compile your application >> >> - And compare the compile commands and see if there are any differences >> (missing options, wrong order of include files etc..) >> >> Satish >> > > > -- > Regards, > > Kiran Hegde > Mobile:+91 9972600235 > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Tue Aug 5 07:25:53 2025 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 5 Aug 2025 08:25:53 -0400 Subject: [petsc-users] [GPU] Jacobi preconditioner In-Reply-To: References: <386853b1efae4269919b977b88c7e679@cea.fr> <49396000-D752-4C95-AF1B-524EC68BC5BC@petsc.dev> Message-ID: <3062A2B3-0A62-442D-9E93-24D2EF6C9668@petsc.dev> Then Jacob has to come back and finish the baking. Barry > On Aug 4, 2025, at 11:27?PM, Junchao Zhang wrote: > > CUPM is half baked, and only supports Vec and MATDENSE. > > --Junchao Zhang > > > On Mon, Aug 4, 2025 at 7:31?PM Barry Smith > wrote: >> >> I thought the CUPM code was intended to allow us to have common code between NVIDIA and AMD? >> >> >>> On Jul 31, 2025, at 11:05?AM, Junchao Zhang > wrote: >>> >>> What would embarrass me more is to copy the same code to MatGetDiagonal_SeqAIJHIPSPARSE. >>> >>> --Junchao Zhang >>> >>> On Wed, Jul 30, 2025 at 1:34?PM Barry Smith > wrote: >>>> >>>> We absolutely should have a MatGetDiagonal_SeqAIJCUSPARSE(). It's somewhat embarrassing that we don't provide this. >>>> >>>> I have found some potential code at https://urldefense.us/v3/__https://stackoverflow.com/questions/60311408/how-to-get-the-diagonal-of-a-sparse-matrix-in-cusparse__;!!G_uCfscf7eWS!dkLsir2oikTcYGJKRYgXdA9kTWZzg2MXrL0j0w750Ji-8wfT-NJG6B0cTCgmxasEkr_G8Fb3gHEi7xsu4I1_bao$ >>>> >>>> Barry >>>> >>>> >>>> >>>> >>>>> On Jul 28, 2025, at 11:43?AM, Junchao Zhang > wrote: >>>>> >>>>> Yes, MatGetDiagonal_SeqAIJCUSPARSE hasn't been implemented. petsc/cuda and petsc/kokkos backends are separate code. >>>>> If petsc/kokkos meet your needs, then just use them. For petsc users, we hope it will be just a difference of extra --download-kokkos --download-kokkos-kernels in configuration. >>>>> >>>>> --Junchao Zhang >>>>> >>>>> >>>>> On Mon, Jul 28, 2025 at 2:51?AM LEDAC Pierre > wrote: >>>>>> Hello all, >>>>>> >>>>>> >>>>>> >>>>>> We are solving with PETSc a linear system updated every time step (constant stencil but coefficients changing). >>>>>> >>>>>> >>>>>> >>>>>> The matrix is preallocated once with MatSetPreallocationCOO() then filled each time step with MatSetValuesCOO() and we use device pointers for coo_i, coo_j, and coefficients values. >>>>>> >>>>>> >>>>>> >>>>>> It is working fine with a GMRES Ksp solver and PC Jacobi but we are surprised to see that every time step, during PCSetUp, MatGetDiagonal_SeqAIJ is called whereas the matrix is on the device. Looking at the API, it seems there is no MatGetDiagonal_SeqAIJCUSPARSE() but a MatGetDiagonal_SeqAIJKOKKOS(). >>>>>> >>>>>> >>>>>> >>>>>> Does it mean we should use Kokkos backend in PETSc to have Jacobi preconditioner built directly on device ? Or I am doing something wrong ? >>>>>> >>>>>> NB: Gmres is running well on device. >>>>>> >>>>>> >>>>>> >>>>>> I could use -ksp_reuse_preconditioner to avoid Jacobi being recreated each solve on host but it increases significantly the number of iterations. >>>>>> >>>>>> >>>>>> >>>>>> Thanks, >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Pierre LEDAC >>>>>> Commissariat ? l??nergie atomique et aux ?nergies alternatives >>>>>> Centre de SACLAY >>>>>> DES/ISAS/DM2S/SGLS/LCAN >>>>>> B?timent 451 ? point courrier n?41 >>>>>> F-91191 Gif-sur-Yvette >>>>>> +33 1 69 08 04 03 >>>>>> +33 6 83 42 05 79 >>>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From herbert.owen at bsc.es Wed Aug 6 10:44:06 2025 From: herbert.owen at bsc.es (howen) Date: Wed, 6 Aug 2025 17:44:06 +0200 Subject: [petsc-users] Bug or unexpected behaviour when running petsc+hypre on CPUs Message-ID: <5CC52BC3-0C49-46A4-B3BC-D1A965D0C342@bsc.es> Dear Sir, I am introducing petsc into our fortran + openacc code, https://urldefense.us/v3/__https://gitlab.com/bsc_sod2d/sod2d_gitlab__;!!G_uCfscf7eWS!f0Y_9okAiFSBeUBOlNplMe6jOCPtdRHfhX2s_rz3N9kCK4OJGiGWGjFyRd3JOwojw7MeNEA-VJtgdn8wBbCmRmJTqdFC$ . My final objective is to run AMG (Boomer from hypre) on the GPU. For the moment I am performing test on the CPU only. I run on Marenostrum-V. https://urldefense.us/v3/__https://www.bsc.es/supportkc/docs/MareNostrum5/intro/__;!!G_uCfscf7eWS!f0Y_9okAiFSBeUBOlNplMe6jOCPtdRHfhX2s_rz3N9kCK4OJGiGWGjFyRd3JOwojw7MeNEA-VJtgdn8wBbCmRraUJcUF$ I am compiling my code with NVHPC and the support team from BSC has compiled petsc + hypre for me. In the configuration they used -with-cuda. I have run petsc and it works correctly both in serial and parallel on the CPU. In my code I use call MatSetType(amat,MATAIJ,ierr). I understand this is teh expected behaviour for Petsc. That is, that even if one compiles with petsc cuda support one can run only on the GPU depending on what one sets in MatSetType. Could you confirm that this is the expected behaviour? Instead it seems that when petsc+hypre is used one needs to compile specific versions for CPU and GPU. When trying to use hypre through petsc if one has compiled petsc using -with-cuda the run fails. >From what I have understood this in not the expected behavior. could you confirm this? Depending of what options I give, the error is different If I use -pc_type hypre -pc_hypre_type boomeramg -pc_hypre_boomeramg_coarsen_type hmis -pc_hypre_boomeramg_interp_type ext+i -pc_hypre_boomeramg_relax_type_all SOR/Jacobi -pc_hypre_boomeramg_relax_type_coarse SOR/Jacobi -pc_hypre_boomeramg_grid_sweeps_coarse 1 -pc_hypre_boomeramg_strong_threshold 0.25 I get [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see https://urldefense.us/v3/__https://petsc.org/release/faq/*valgrind__;Iw!!G_uCfscf7eWS!f0Y_9okAiFSBeUBOlNplMe6jOCPtdRHfhX2s_rz3N9kCK4OJGiGWGjFyRd3JOwojw7MeNEA-VJtgdn8wBbCmRt2ImXnV$ and https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!f0Y_9okAiFSBeUBOlNplMe6jOCPtdRHfhX2s_rz3N9kCK4OJGiGWGjFyRd3JOwojw7MeNEA-VJtgdn8wBbCmRjGOi2o3$ [0]PETSC ERROR: or try https://urldefense.us/v3/__https://docs.nvidia.com/cuda/cuda-memcheck/index.html__;!!G_uCfscf7eWS!f0Y_9okAiFSBeUBOlNplMe6jOCPtdRHfhX2s_rz3N9kCK4OJGiGWGjFyRd3JOwojw7MeNEA-VJtgdn8wBbCmRqNF46dV$ on NVIDIA CUDA systems to find memory corruption errors [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: The line numbers in the error traceback are not always exact. [0]PETSC ERROR: #1 jac->setup() [0]PETSC ERROR: #2 PCSetUp_HYPRE() at /gpfs/apps/MN5/ACC/PETSC/SRC/petsc-v3.21.0_hypre-debug/src/ksp/pc/impls/hypre/hypre.c:422 [0]PETSC ERROR: #3 PCSetUp() at /gpfs/apps/MN5/ACC/PETSC/SRC/petsc-v3.21.0_hypre-debug/src/ksp/pc/interface/precon.c:1079 [0]PETSC ERROR: #4 KSPSetUp() at /gpfs/apps/MN5/ACC/PETSC/SRC/petsc-v3.21.0_hypre-debug/src/ksp/ksp/interface/itfunc.c:415 -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 0 in communicator MPI COMMUNICATOR 3 SPLIT FROM 0 with errorcode 59. Which does not help much. Instead if I use only -pc_type hypre I get [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Invalid argument [0]PETSC ERROR: HYPRE_MEMORY_DEVICE expects a device vector. You need to enable PETSc device support, for example, in some cases, -vec_type cuda This helped my realise that hypre was trying to use the GPU despite in my code Both runs are succesfull when compiling without -with-cuda Best, Herbert Owen Senior Researcher, Dpt. Computer Applications in Science and Engineering Barcelona Supercomputing Center (BSC-CNS) Tel: +34 93 413 4038 Skype: herbert.owen https://urldefense.us/v3/__https://scholar.google.es/citations?user=qe5O2IYAAAAJ&hl=en__;!!G_uCfscf7eWS!f0Y_9okAiFSBeUBOlNplMe6jOCPtdRHfhX2s_rz3N9kCK4OJGiGWGjFyRd3JOwojw7MeNEA-VJtgdn8wBbCmRkKVuPTJ$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Aug 6 11:39:46 2025 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 6 Aug 2025 12:39:46 -0400 Subject: [petsc-users] Bug or unexpected behaviour when running petsc+hypre on CPUs In-Reply-To: <5CC52BC3-0C49-46A4-B3BC-D1A965D0C342@bsc.es> References: <5CC52BC3-0C49-46A4-B3BC-D1A965D0C342@bsc.es> Message-ID: On Wed, Aug 6, 2025 at 11:53?AM howen via petsc-users < petsc-users at mcs.anl.gov> wrote: > Dear Sir, > > I am introducing petsc into our fortran + openacc code, > https://urldefense.us/v3/__https://gitlab.com/bsc_sod2d/sod2d_gitlab__;!!G_uCfscf7eWS!btmKhbl3NeBl7Rgz-znsnwgTYs0aZfcXn-qxj6ei6jgwTQfNYeQ7Rz8De5GhON1QxIqHrvi9CfpPQuwP4t9a$ > > . > > My final objective is to run AMG (Boomer from hypre) on the GPU. > > For the moment I am performing test on the CPU only. > > I run on Marenostrum-V. > https://urldefense.us/v3/__https://www.bsc.es/supportkc/docs/MareNostrum5/intro/__;!!G_uCfscf7eWS!btmKhbl3NeBl7Rgz-znsnwgTYs0aZfcXn-qxj6ei6jgwTQfNYeQ7Rz8De5GhON1QxIqHrvi9CfpPQqfAlkhq$ > > > I am compiling my code with NVHPC and the support team from BSC has > compiled petsc + hypre for me. > > In the configuration they used -with-cuda. > > I have run petsc and it works correctly both in serial and parallel on the > CPU. In my code I use call MatSetType(amat,MATAIJ,ierr). > I understand this is teh expected behaviour for Petsc. > That is, that even if one compiles with petsc cuda support one can run > only on the GPU depending on what one sets in MatSetType. > Could you confirm that this is the expected behaviour? Instead it seems > that when petsc+hypre is used one needs to > compile specific versions for CPU and GPU. > > When trying to use hypre through petsc if one has compiled petsc using > -with-cuda the run fails. > From what I have understood this in not the expected behavior. could you > confirm this? > > > Depending of what options I give, the error is different > > If I use > > -pc_type hypre > -pc_hypre_type boomeramg > -pc_hypre_boomeramg_coarsen_type hmis > -pc_hypre_boomeramg_interp_type ext+i > -pc_hypre_boomeramg_relax_type_all SOR/Jacobi > -pc_hypre_boomeramg_relax_type_coarse SOR/Jacobi > -pc_hypre_boomeramg_grid_sweeps_coarse 1 > -pc_hypre_boomeramg_strong_threshold 0.25 > > > I get > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see https://urldefense.us/v3/__https://petsc.org/release/faq/*valgrind__;Iw!!G_uCfscf7eWS!btmKhbl3NeBl7Rgz-znsnwgTYs0aZfcXn-qxj6ei6jgwTQfNYeQ7Rz8De5GhON1QxIqHrvi9CfpPQlAW-NlE$ > > and https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!btmKhbl3NeBl7Rgz-znsnwgTYs0aZfcXn-qxj6ei6jgwTQfNYeQ7Rz8De5GhON1QxIqHrvi9CfpPQnhULpwg$ > > [0]PETSC ERROR: or try > https://urldefense.us/v3/__https://docs.nvidia.com/cuda/cuda-memcheck/index.html__;!!G_uCfscf7eWS!btmKhbl3NeBl7Rgz-znsnwgTYs0aZfcXn-qxj6ei6jgwTQfNYeQ7Rz8De5GhON1QxIqHrvi9CfpPQge-ms2z$ > > on NVIDIA CUDA systems to find memory corruption errors > [0]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [0]PETSC ERROR: The line numbers in the error traceback are not always > exact. > [0]PETSC ERROR: #1 jac->setup() > [0]PETSC ERROR: #2 PCSetUp_HYPRE() at > /gpfs/apps/MN5/ACC/PETSC/SRC/petsc-v3.21.0_hypre-debug/src/ksp/pc/impls/hypre/hypre.c:422 > [0]PETSC ERROR: #3 PCSetUp() at > /gpfs/apps/MN5/ACC/PETSC/SRC/petsc-v3.21.0_hypre-debug/src/ksp/pc/interface/precon.c:1079 > [0]PETSC ERROR: #4 KSPSetUp() at > /gpfs/apps/MN5/ACC/PETSC/SRC/petsc-v3.21.0_hypre-debug/src/ksp/ksp/interface/itfunc.c:415 > -------------------------------------------------------------------------- > MPI_ABORT was invoked on rank 0 in communicator MPI COMMUNICATOR 3 SPLIT > FROM 0 > with errorcode 59. > > Which does not help much. > > Instead if I use only > > -pc_type hypre > > I get > > > > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Invalid argument > [0]PETSC ERROR: HYPRE_MEMORY_DEVICE expects a device vector. You need to > enable PETSc device support, for example, in some cases, -vec_type cuda > > > This helped my realise that hypre was trying to use the GPU despite in my > code > We try to have a good error message here (If you could give us the code with the SEGV, we will fix it to give an informative error). This is a limitation of Hypre, namely that it runs _either_ on the CPU or the GPU, not both at the same time. Hopefully this will not be the case with a future release. Thanks, Matt > Both runs are succesfull when compiling without -with-cuda > > Best, > Herbert Owen > Senior Researcher, Dpt. Computer Applications in Science and Engineering > Barcelona Supercomputing Center (BSC-CNS) > Tel: +34 93 413 4038 > Skype: herbert.owen > > https://urldefense.us/v3/__https://scholar.google.es/citations?user=qe5O2IYAAAAJ&hl=en__;!!G_uCfscf7eWS!btmKhbl3NeBl7Rgz-znsnwgTYs0aZfcXn-qxj6ei6jgwTQfNYeQ7Rz8De5GhON1QxIqHrvi9CfpPQp3NDHfq$ > > > > > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!btmKhbl3NeBl7Rgz-znsnwgTYs0aZfcXn-qxj6ei6jgwTQfNYeQ7Rz8De5GhON1QxIqHrvi9CfpPQjVmHttn$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefano.zampini at gmail.com Wed Aug 6 11:58:26 2025 From: stefano.zampini at gmail.com (Stefano Zampini) Date: Wed, 6 Aug 2025 18:58:26 +0200 Subject: [petsc-users] Bug or unexpected behaviour when running petsc+hypre on CPUs In-Reply-To: <5CC52BC3-0C49-46A4-B3BC-D1A965D0C342@bsc.es> References: <5CC52BC3-0C49-46A4-B3BC-D1A965D0C342@bsc.es> Message-ID: > > If I use > > -pc_type hypre > -pc_hypre_type boomeramg > -pc_hypre_boomeramg_coarsen_type hmis > -pc_hypre_boomeramg_interp_type ext+i > -pc_hypre_boomeramg_relax_type_all SOR/Jacobi > -pc_hypre_boomeramg_relax_type_coarse SOR/Jacobi > -pc_hypre_boomeramg_grid_sweeps_coarse 1 > -pc_hypre_boomeramg_strong_threshold 0.25 > > > Some HYPRE options for the relaxation and coarsening type are not coded for GPU, that's probably the reason for the segfault. I suggest you run without special options and start adding one by one until you find the ones are not supported. > > > Instead if I use only > > -pc_type hypre > > This probably leads to a successful setup of the preconditioner on the GPU, but then you most probably are passing a CPU vector for the RHS. A convenient way to have always a consistent vector is to use MatCreateVecs > I get > > > > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Invalid argument > [0]PETSC ERROR: HYPRE_MEMORY_DEVICE expects a device vector. You need to > enable PETSc device support, for example, in some cases, -vec_type cuda > > > This helped my realise that hypre was trying to use the GPU despite in my > code > > Both runs are succesfull when compiling without -with-cuda > > Best, > Herbert Owen > Senior Researcher, Dpt. Computer Applications in Science and Engineering > Barcelona Supercomputing Center (BSC-CNS) > Tel: +34 93 413 4038 > Skype: herbert.owen > > https://urldefense.us/v3/__https://scholar.google.es/citations?user=qe5O2IYAAAAJ&hl=en__;!!G_uCfscf7eWS!eUJy7kA8zOa0CWtMLCijo4AR4thJRQNNhM5OhBMp-pokLY6RYnJGRnEk-eGL8I2CaNf-zHz2v5dFWMA7LEJ_D_EJapgyt-Y$ > > > > > > > > > > -- Stefano -------------- next part -------------- An HTML attachment was scrubbed... URL: From Elena.Moral.Sanchez at ipp.mpg.de Thu Aug 7 12:22:00 2025 From: Elena.Moral.Sanchez at ipp.mpg.de (Moral Sanchez, Elena) Date: Thu, 7 Aug 2025 17:22:00 +0000 Subject: [petsc-users] error using CG with matrix-free operator and MG preconditioner Message-ID: Hello, I want to solve a linear system with conjugate gradient and multigrid preconditioner. I am using petsc4py. The solution is a Vector of type NEST and the operator is a matrix-free operator created using PETSc.Mat().createPython. The restriction and interpolation operators are all type NEST and their VecType is set to NEST. As smoother I am using CG without preconditioner. For the moment I am just using a classical V-cycle with 2 levels. I am able to solve the system with CG without the MG preconditioner. However, when I try to solve it with the MG preconditioner, it throws this error: Traceback (most recent call last): File "/home/elena-hp/code/Poisson_MG_Petsc.py", line 321, in solver.solve(b, u) File "petsc4py/PETSc/KSP.pyx", line 1782, in petsc4py.PETSc.KSP.solve petsc4py.PETSc.Error: error code 62 [0] KSPSolve() at /home/elena-hp/petsc/src/ksp/ksp/interface/itfunc.c:1089 [0] KSPSolve_Private() at /home/elena-hp/petsc/src/ksp/ksp/interface/itfunc.c:912 [0] KSPSolve_CG() at /home/elena-hp/petsc/src/ksp/ksp/impls/cg/cg.c:213 [0] KSP_PCApply() at /home/elena-hp/petsc/include/petsc/private/kspimpl.h:414 [0] PCApply() at /home/elena-hp/petsc/src/ksp/pc/interface/precon.c:528 [0] PCApply_MG() at /home/elena-hp/petsc/src/ksp/pc/impls/mg/mg.c:651 [0] PCApply_MG_Internal() at /home/elena-hp/petsc/src/ksp/pc/impls/mg/mg.c:629 [0] PCMGMCycle_Private() at /home/elena-hp/petsc/src/ksp/pc/impls/mg/mg.c:27 [0] KSPSolve() at /home/elena-hp/petsc/src/ksp/ksp/interface/itfunc.c:1089 [0] KSPSolve_Private() at /home/elena-hp/petsc/src/ksp/ksp/interface/itfunc.c:912 [0] KSPSolve_CG() at /home/elena-hp/petsc/src/ksp/ksp/impls/cg/cg.c:162 [0] VecCopy() at /home/elena-hp/petsc/src/vec/vec/interface/vector.c:1796 [0] VecCopyAsync_Private() at /home/elena-hp/petsc/src/vec/vec/interface/vector.c:1755 [0] VecCopy_Nest() at /home/elena-hp/petsc/src/vec/vec/impls/nest/vecnest.c:56 [0] Invalid argument [0] Nest vector argument 2 not setup. The error does not appear when the operator is also of type NEST and the VecType is set to NEST. Unfortunately, for my application I cannot write the operator as type NEST. I think that the problem is related to the fact that the method createVecs() of the matrix-free operator does not return NEST vectors. I tried to do setVecType(PETSc.Vec.Type.NEST) on the matrix-free operator but when I call createVecs() it throws this error: Traceback (most recent call last): File "petsc4py/PETSc/libpetsc4py.pyx", line 965, in petsc4py.PETSc.MatCreateVecs_Python petsc4py.PETSc.Error: error code 86 [0] MatCreateVecs() at /home/elena-hp/petsc/src/mat/interface/matrix.c:9557 [0] VecSetType() at /home/elena-hp/petsc/src/vec/vec/interface/vecreg.c:140 [0] Unknown type. Check for miss-spelling or missing package: https://urldefense.us/v3/__https://petsc.org/release/install/install/*external-packages__;Iw!!G_uCfscf7eWS!dEBOuSXxIqEDpI33Xza0TlRPC5n6sN3fXNslohXzfITEmMwZLwQWJUTyNE-mMJHiXZjxQ-OlWu5625crKTR81qTlPMyhvkR09mUB$ [0] Unknown vector type: nest The above exception was the direct cause of the following exception: Traceback (most recent call last): File "", line 1, in File "petsc4py/PETSc/Mat.pyx", line 3411, in petsc4py.PETSc.Mat.createVecs petsc4py.PETSc.Error: error code 101 [0] MatCreateVecs() at /home/elena-hp/petsc/src/mat/interface/matrix.c:9552 Do you know how to fix this? Thank you in advance. Regards, Elena -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefano.zampini at gmail.com Thu Aug 7 12:55:12 2025 From: stefano.zampini at gmail.com (Stefano Zampini) Date: Thu, 7 Aug 2025 19:55:12 +0200 Subject: [petsc-users] error using CG with matrix-free operator and MG preconditioner In-Reply-To: References: Message-ID: If the problem is that the vector is not of nest type, you can overload the createVecs for the python matrix type, see e.g. https://urldefense.us/v3/__https://petsc.org/release/petsc4py/petsc_python_types.html__;!!G_uCfscf7eWS!aZjDCYTy1PCHmDJYxODxFR7QrOBVMVDa5NcHQ12xgd56_59kgzghGzTR42NjUnNXhIPGEMTWdWtzqqPdtK9UtT_taE4lvMw$ Stefano On Thu, Aug 7, 2025, 19:52 Moral Sanchez, Elena < Elena.Moral.Sanchez at ipp.mpg.de> wrote: > Hello, > I want to solve a linear system with conjugate gradient and multigrid > preconditioner. I am using petsc4py. The solution is a Vector of type NEST > and the operator is a matrix-free operator created using > PETSc.Mat().createPython. The restriction and interpolation operators are > all type NEST and their VecType is set to NEST. As smoother I am using CG > without preconditioner. For the moment I am just using a classical V-cycle > with 2 levels. > > I am able to solve the system with CG without the MG preconditioner. > However, when I try to solve it with the MG preconditioner, it throws this > error: > > Traceback (most recent call last): > File "/home/elena-hp/code/Poisson_MG_Petsc.py", line 321, in > solver.solve(b, u) > File "petsc4py/PETSc/KSP.pyx", line 1782, in petsc4py.PETSc.KSP.solve > petsc4py.PETSc.Error: error code 62 > [0] KSPSolve() at /home/elena-hp/petsc/src/ksp/ksp/interface/itfunc.c:1089 > [0] KSPSolve_Private() at /home/elena-hp/petsc/src/ksp/ksp/interface/ > itfunc.c:912 > [0] KSPSolve_CG() at /home/elena-hp/petsc/src/ksp/ksp/impls/cg/cg.c:213 > [0] KSP_PCApply() at /home/elena-hp/petsc/include/petsc/private/ > kspimpl.h:414 > [0] PCApply() at /home/elena-hp/petsc/src/ksp/pc/interface/precon.c:528 > [0] PCApply_MG() at /home/elena-hp/petsc/src/ksp/pc/impls/mg/mg.c:651 > [0] PCApply_MG_Internal() at /home/elena-hp/petsc/src/ksp/pc/impls/mg/ > mg.c:629 > [0] PCMGMCycle_Private() at /home/elena-hp/petsc/src/ksp/pc/impls/mg/ > mg.c:27 > [0] KSPSolve() at /home/elena-hp/petsc/src/ksp/ksp/interface/itfunc.c:1089 > [0] KSPSolve_Private() at /home/elena-hp/petsc/src/ksp/ksp/interface/ > itfunc.c:912 > [0] KSPSolve_CG() at /home/elena-hp/petsc/src/ksp/ksp/impls/cg/cg.c:162 > [0] VecCopy() at /home/elena-hp/petsc/src/vec/vec/interface/vector.c:1796 > [0] VecCopyAsync_Private() at /home/elena-hp/petsc/src/vec/vec/interface/ > vector.c:1755 > [0] VecCopy_Nest() at /home/elena-hp/petsc/src/vec/vec/impls/nest/ > vecnest.c:56 > [0] Invalid argument > [0] Nest vector argument 2 not setup. > > The error does not appear when the operator is also of type NEST and the > VecType is set to NEST. Unfortunately, for my application I cannot write > the operator as type NEST. > > I think that the problem is related to the fact that the method > createVecs() of the matrix-free operator does not return NEST vectors. I > tried to do setVecType(PETSc.Vec.Type.NEST) on the matrix-free operator but > when I call createVecs() it throws this error: > > Traceback (most recent call last): > File "petsc4py/PETSc/libpetsc4py.pyx", line 965, in > petsc4py.PETSc.MatCreateVecs_Python > petsc4py.PETSc.Error: error code 86 > [0] MatCreateVecs() at /home/elena-hp/petsc/src/mat/interface/ > matrix.c:9557 > [0] VecSetType() at /home/elena-hp/petsc/src/vec/vec/interface/ > vecreg.c:140 > [0] Unknown type. Check for miss-spelling or missing package: > https://urldefense.us/v3/__https://petsc.org/release/install/install/*external-packages__;Iw!!G_uCfscf7eWS!aZjDCYTy1PCHmDJYxODxFR7QrOBVMVDa5NcHQ12xgd56_59kgzghGzTR42NjUnNXhIPGEMTWdWtzqqPdtK9UtT_tTSSXMDQ$ > > [0] Unknown vector type: nest > > The above exception was the direct cause of the following exception: > > Traceback (most recent call last): > File "", line 1, in > File "petsc4py/PETSc/Mat.pyx", line 3411, in > petsc4py.PETSc.Mat.createVecs > petsc4py.PETSc.Error: error code 101 > [0] MatCreateVecs() at /home/elena-hp/petsc/src/mat/interface/ > matrix.c:9552 > > Do you know how to fix this? > > Thank you in advance. > > Regards, > Elena > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Thu Aug 7 20:09:07 2025 From: bsmith at petsc.dev (Barry Smith) Date: Thu, 7 Aug 2025 21:09:07 -0400 Subject: [petsc-users] petsc-users Digest, Vol 200, Issue 3 In-Reply-To: References: Message-ID: <0D4EBFFD-AE78-49ED-8E1D-2C4E1B9A3D1E@petsc.dev> Please send (cut and paste) all the output of make ex5f (should succeed) and make "your code" we need to see the exact calls for both codes and the exact error message. > On Aug 5, 2025, at 12:44?AM, kiran hegde wrote: > > Hi, > > Sorry, the tutorial case run succeeded. No errors reported. When i used the makefile file flags in my code, my code is still reported error. > > Regards, > > Kiran Hegde > Mobile:+91 9972600235 > > On Tue, Aug 5, 2025, 6:11?AM Barry Smith > wrote: >> >> "I have tried the options ex5f, but getting the same error." >> >> So you did >> >> cd src/snes/tutorials/ >> make ex5f >> >> and got some errors? Please cut and paste all the output from make ex5f and sent it here. >> >> >> >>> On Aug 4, 2025, at 11:15?AM, kiran hegde > wrote: >>> >>> Hi Satish, >>> >>> I have tried the options ex5f, but getting the same error. >>> The code was running without any issues long back and I was using petsc ver 3.9 using gcc 12.X.X. >>> My makefile flags are : >>> >>> FFLAGS = -march=native -O3 -pipe -ffree-form -cpp -ffree-line-length-none \ >>> -Wall -Wextra -Wtabs -Wintrinsics-std -Wsurprising -Waliasing \ >>> -Wno-unused-parameter -Wno-unused-dummy-argument \ >>> -D$(MODEL) \ >>> -DVERSION=\"$(GIT_VERSION)\" \ >>> -I$(PETSC_DIR)/include >>> FPPFLAGS = -cpp -E -dM -C -P -D$(MODEL) >>> LDFLAGS = -lpetsc -L$(PETSC_DIR)/lib >>> LDFLAGS += -Wl,-rpath=$(PETSC_DIR)/lib >>> >>> Have i missed anything? >>> >>> >>> Regards, >>> Kiran >>> >>> >>> >>> >>> On Sun, Aug 3, 2025 at 9:56?PM Satish Balay > wrote: >>>> On Sun, 3 Aug 2025, kiran hegde wrote: >>>> >>>> > Hi, >>>> > >>>> > Please find the attached configure.log file. >>>> > PETSC installation successfully completes with make check, without any >>>> > error. >>>> > But, when compiling fortan codes with petsc library+include in the >>>> > makefile i get this error. >>>> >>>> Then likely your compile command is missing some options. you can: >>>> >>>> - compile a petsc example - say: >>>> cd src/snes/tutorials >>>> make ex5f >>>> >>>> - now compile your application >>>> >>>> - And compare the compile commands and see if there are any differences (missing options, wrong order of include files etc..) >>>> >>>> Satish >>> >>> >>> >>> -- >>> Regards, >>> >>> Kiran Hegde >>> Mobile:+91 9972600235 >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From Elena.Moral.Sanchez at ipp.mpg.de Fri Aug 8 11:14:12 2025 From: Elena.Moral.Sanchez at ipp.mpg.de (Moral Sanchez, Elena) Date: Fri, 8 Aug 2025 16:14:12 +0000 Subject: [petsc-users] error using CG with matrix-free operator and MG preconditioner In-Reply-To: References: , Message-ID: <890224b18b2b4b2792b0008f1a1142b1@ipp.mpg.de> Dear Stefano, That was exactly what I needed. It works now, thank you! Regards, Elena ________________________________ From: Stefano Zampini Sent: 07 August 2025 19:55:12 To: Moral Sanchez, Elena Cc: PETSc users list Subject: Re: [petsc-users] error using CG with matrix-free operator and MG preconditioner If the problem is that the vector is not of nest type, you can overload the createVecs for the python matrix type, see e.g. https://urldefense.us/v3/__https://petsc.org/release/petsc4py/petsc_python_types.html__;!!G_uCfscf7eWS!ci0JOiv6LHgM3Yfprnw7HmOcqzmdfgfp959aPpTEkIEr74xepQ4MSMrVMnS89dMTBA3e_vujFa3L5OfLfHuYdRsBecF9tispRqmV$ Stefano On Thu, Aug 7, 2025, 19:52 Moral Sanchez, Elena > wrote: Hello, I want to solve a linear system with conjugate gradient and multigrid preconditioner. I am using petsc4py. The solution is a Vector of type NEST and the operator is a matrix-free operator created using PETSc.Mat().createPython. The restriction and interpolation operators are all type NEST and their VecType is set to NEST. As smoother I am using CG without preconditioner. For the moment I am just using a classical V-cycle with 2 levels. I am able to solve the system with CG without the MG preconditioner. However, when I try to solve it with the MG preconditioner, it throws this error: Traceback (most recent call last): File "/home/elena-hp/code/Poisson_MG_Petsc.py", line 321, in solver.solve(b, u) File "petsc4py/PETSc/KSP.pyx", line 1782, in petsc4py.PETSc.KSP.solve petsc4py.PETSc.Error: error code 62 [0] KSPSolve() at /home/elena-hp/petsc/src/ksp/ksp/interface/itfunc.c:1089 [0] KSPSolve_Private() at /home/elena-hp/petsc/src/ksp/ksp/interface/itfunc.c:912 [0] KSPSolve_CG() at /home/elena-hp/petsc/src/ksp/ksp/impls/cg/cg.c:213 [0] KSP_PCApply() at /home/elena-hp/petsc/include/petsc/private/kspimpl.h:414 [0] PCApply() at /home/elena-hp/petsc/src/ksp/pc/interface/precon.c:528 [0] PCApply_MG() at /home/elena-hp/petsc/src/ksp/pc/impls/mg/mg.c:651 [0] PCApply_MG_Internal() at /home/elena-hp/petsc/src/ksp/pc/impls/mg/mg.c:629 [0] PCMGMCycle_Private() at /home/elena-hp/petsc/src/ksp/pc/impls/mg/mg.c:27 [0] KSPSolve() at /home/elena-hp/petsc/src/ksp/ksp/interface/itfunc.c:1089 [0] KSPSolve_Private() at /home/elena-hp/petsc/src/ksp/ksp/interface/itfunc.c:912 [0] KSPSolve_CG() at /home/elena-hp/petsc/src/ksp/ksp/impls/cg/cg.c:162 [0] VecCopy() at /home/elena-hp/petsc/src/vec/vec/interface/vector.c:1796 [0] VecCopyAsync_Private() at /home/elena-hp/petsc/src/vec/vec/interface/vector.c:1755 [0] VecCopy_Nest() at /home/elena-hp/petsc/src/vec/vec/impls/nest/vecnest.c:56 [0] Invalid argument [0] Nest vector argument 2 not setup. The error does not appear when the operator is also of type NEST and the VecType is set to NEST. Unfortunately, for my application I cannot write the operator as type NEST. I think that the problem is related to the fact that the method createVecs() of the matrix-free operator does not return NEST vectors. I tried to do setVecType(PETSc.Vec.Type.NEST) on the matrix-free operator but when I call createVecs() it throws this error: Traceback (most recent call last): File "petsc4py/PETSc/libpetsc4py.pyx", line 965, in petsc4py.PETSc.MatCreateVecs_Python petsc4py.PETSc.Error: error code 86 [0] MatCreateVecs() at /home/elena-hp/petsc/src/mat/interface/matrix.c:9557 [0] VecSetType() at /home/elena-hp/petsc/src/vec/vec/interface/vecreg.c:140 [0] Unknown type. Check for miss-spelling or missing package: https://urldefense.us/v3/__https://petsc.org/release/install/install/*external-packages__;Iw!!G_uCfscf7eWS!ci0JOiv6LHgM3Yfprnw7HmOcqzmdfgfp959aPpTEkIEr74xepQ4MSMrVMnS89dMTBA3e_vujFa3L5OfLfHuYdRsBecF9tkMvgYZ5$ [0] Unknown vector type: nest The above exception was the direct cause of the following exception: Traceback (most recent call last): File "", line 1, in File "petsc4py/PETSc/Mat.pyx", line 3411, in petsc4py.PETSc.Mat.createVecs petsc4py.PETSc.Error: error code 101 [0] MatCreateVecs() at /home/elena-hp/petsc/src/mat/interface/matrix.c:9552 Do you know how to fix this? Thank you in advance. Regards, Elena -------------- next part -------------- An HTML attachment was scrubbed... URL: From bramkamp at nsc.liu.se Fri Aug 8 11:21:56 2025 From: bramkamp at nsc.liu.se (Frank Bramkamp) Date: Fri, 8 Aug 2025 18:21:56 +0200 Subject: [petsc-users] General Questions Message-ID: <11F32C05-34A5-4956-A657-7BFDBF306FB9@nsc.liu.se> Dear PETSc Team, I have the following general questions. When using GMRES with classic gram schmidt, you have an optional refinement. The default is not to use any refinement, is that correct ? So you use the standard classic gram schmidt without any additional refinement step. I originally thought that you use one refinement step if needed as default. Or was that somehow changed one day and I remember it from older settings ?! With refinement one also needs about twice the MPI communication, if refinement is done in each iteration, which can be a bit more costly due to the MPI_ALLREDUCE for the dot products. I solve the compressible navier stokes equations with turbulence. In which cases do you have the experience one should better use a refinement step ?! So far I typically used the default options, which seemed to work well for me so far. for classic gram schmidt one also has to compute multiple dot products which is VecMDot in PETSC if I see this correct. Do you by standard use blas GEMV to compute multiple dot products at once or do you typically prefer your custom implementation. I think you also have a version where you unroll manual loops by a factor of 4 or so. I am not quite sure what is the typical default for this, or in which situation one would not use GEMV. Thanks a lot. I wish you a nice weekend, Frank From knepley at gmail.com Fri Aug 8 15:28:55 2025 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 8 Aug 2025 16:28:55 -0400 Subject: [petsc-users] General Questions In-Reply-To: <11F32C05-34A5-4956-A657-7BFDBF306FB9@nsc.liu.se> References: <11F32C05-34A5-4956-A657-7BFDBF306FB9@nsc.liu.se> Message-ID: On Fri, Aug 8, 2025 at 12:22?PM Frank Bramkamp wrote: > Dear PETSc Team, > > I have the following general questions. > When using GMRES with classic gram schmidt, you have an optional > refinement. > > The default is not to use any refinement, is that correct ? Yes. It has been this way at least since 2013 from checking the log. > So you use the standard classic gram schmidt > without any additional refinement step. > > I originally thought that you use one refinement step if needed as > default. Or was that somehow changed one day > and I remember it from older settings ?! > It is available, but we changed it since refinement was almost never happening. > With refinement one also needs about twice the MPI communication, if > refinement is done in each iteration, > which can be a bit more costly due to the MPI_ALLREDUCE for the dot > products. > It is reduction that is expensive on large runs. > I solve the compressible navier stokes equations with turbulence. In which > cases do you have the experience one should > better use a refinement step ?! So far I typically used the default > options, which seemed to work well for me so far. > If it is converging, everything is likely fine. You can always check the true residual to detect breakdown, or turn on refinement if necessary with the option. > for classic gram schmidt one also has to compute multiple dot products > which is VecMDot in PETSC if I see this correct. > Yes. > Do you by standard use blas GEMV to compute multiple dot products at once > or do you typically prefer your > custom implementation. I think you also have a version where you unroll > manual loops by a factor of 4 or so. > I am not quite sure what is the typical default for this, or in which > situation one would not use GEMV. > It turns out that Junchao did extensive tests on this last year (I believe). By default, we use GEMV for MDot, but not for MAXPY (since performance here was worse with vendor GEMV). It is difficult (or impossible) to tell which algorithm will be better without testing, so there are options to select each one. Thanks, Matt > Thanks a lot. I wish you a nice weekend, Frank > > > > > > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fhbzgnVYBw6ICZM6N0XXNLIcj9qe9bcopUK7lxSsqxqVlv-SvpS8bdrcwYq0TvTbHmbgbSinCLXrvGLYDSzf$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From junchao.zhang at gmail.com Fri Aug 8 16:03:38 2025 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Fri, 8 Aug 2025 16:03:38 -0500 Subject: [petsc-users] General Questions In-Reply-To: References: <11F32C05-34A5-4956-A657-7BFDBF306FB9@nsc.liu.se> Message-ID: On Fri, Aug 8, 2025 at 3:29?PM Matthew Knepley wrote: > On Fri, Aug 8, 2025 at 12:22?PM Frank Bramkamp > wrote: > >> Dear PETSc Team, >> >> I have the following general questions. >> When using GMRES with classic gram schmidt, you have an optional >> refinement. >> >> The default is not to use any refinement, is that correct ? > > > Yes. It has been this way at least since 2013 from checking the log. > > >> So you use the standard classic gram schmidt >> without any additional refinement step. >> >> I originally thought that you use one refinement step if needed as >> default. Or was that somehow changed one day >> and I remember it from older settings ?! >> > > It is available, but we changed it since refinement was almost never > happening. > > >> With refinement one also needs about twice the MPI communication, if >> refinement is done in each iteration, >> which can be a bit more costly due to the MPI_ALLREDUCE for the dot >> products. >> > > It is reduction that is expensive on large runs. > > >> I solve the compressible navier stokes equations with turbulence. In >> which cases do you have the experience one should >> better use a refinement step ?! So far I typically used the default >> options, which seemed to work well for me so far. >> > > If it is converging, everything is likely fine. You can always check the > true residual to detect breakdown, or turn on refinement if necessary with > the option. > > >> for classic gram schmidt one also has to compute multiple dot products >> which is VecMDot in PETSC if I see this correct. >> > > Yes. > > >> Do you by standard use blas GEMV to compute multiple dot products at once >> or do you typically prefer your >> custom implementation. I think you also have a version where you unroll >> manual loops by a factor of 4 or so. >> I am not quite sure what is the typical default for this, or in which >> situation one would not use GEMV. >> > > It turns out that Junchao did extensive tests on this last year (I > believe). By default, we use GEMV for MDot, but not for MAXPY (since > performance here was worse with vendor GEMV). It is difficult (or > impossible) to tell which algorithm will be better without testing, so > there are options to select each one. > The experimental results were here, https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/merge_requests/6580*note_1477784906__;Iw!!G_uCfscf7eWS!ZtYn-pBESeQYKuVZWdDAExVVX8TAYzfmL3zhMHnUbQcF73fi0TYJOnIs2Kufu58HSzDbtYoMUGGF-E7vzU_1s4L9-tBB$ > > Thanks, > > Matt > > >> Thanks a lot. I wish you a nice weekend, Frank >> >> >> >> >> >> >> >> >> >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!ZtYn-pBESeQYKuVZWdDAExVVX8TAYzfmL3zhMHnUbQcF73fi0TYJOnIs2Kufu58HSzDbtYoMUGGF-E7vzU_1s11XGS3n$ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bramkamp at nsc.liu.se Fri Aug 8 16:09:37 2025 From: bramkamp at nsc.liu.se (Frank Bramkamp) Date: Fri, 8 Aug 2025 23:09:37 +0200 Subject: [petsc-users] General Questions In-Reply-To: References: <11F32C05-34A5-4956-A657-7BFDBF306FB9@nsc.liu.se> Message-ID: <64F5FC61-F581-47C5-80EE-A2EAD0401A76@nsc.liu.se> Thanks for the detailed answer, that helps me to understand better what you do. Good to know that for MAXPY the GEMV might be sometimes not the best choice. I also try to understand the algorithms and details better how you optimize them. Then I will also look at the results for MAXPY that Juchao just shared as well. Thanks a lot, Frank From bsmith at petsc.dev Sat Aug 9 16:45:57 2025 From: bsmith at petsc.dev (Barry Smith) Date: Sat, 9 Aug 2025 17:45:57 -0400 Subject: [petsc-users] General Questions In-Reply-To: References: <11F32C05-34A5-4956-A657-7BFDBF306FB9@nsc.liu.se> Message-ID: > On Aug 8, 2025, at 4:28?PM, Matthew Knepley wrote: > > On Fri, Aug 8, 2025 at 12:22?PM Frank Bramkamp > wrote: >> Dear PETSc Team, >> >> I have the following general questions. >> When using GMRES with classic gram schmidt, you have an optional refinement. >> >> The default is not to use any refinement, is that correct ? > > Yes. It has been this way at least since 2013 from checking the log. > >> So you use the standard classic gram schmidt >> without any additional refinement step. >> >> I originally thought that you use one refinement step if needed as default. Or was that somehow changed one day >> and I remember it from older settings ?! > > It is available, but we changed it since refinement was almost never happening. > >> With refinement one also needs about twice the MPI communication, if refinement is done in each iteration, >> which can be a bit more costly due to the MPI_ALLREDUCE for the dot products. > > It is reduction that is expensive on large runs. > >> I solve the compressible navier stokes equations with turbulence. In which cases do you have the experience one should >> better use a refinement step ?! So far I typically used the default options, which seemed to work well for me so far. > > If it is converging, everything is likely fine. You can always check the true residual to detect breakdown, or turn on refinement if necessary with the option. > >> for classic gram schmidt one also has to compute multiple dot products which is VecMDot in PETSC if I see this correct. > > Yes. > >> Do you by standard use blas GEMV to compute multiple dot products at once or do you typically prefer your >> custom implementation. I think you also have a version where you unroll manual loops by a factor of 4 or so. >> I am not quite sure what is the typical default for this, or in which situation one would not use GEMV. > > It turns out that Junchao did extensive tests on this last year (I believe). By default, we use GEMV for MDot, Frank A little more detail on how this is done and how to take advantage of it. In VecMultiDot_Seq_GEMV() it checks if the array addresses of multiple vectors have the same offset from the previous vector's array. If they have the same offset that means that Xgemv() can be used. Remember Xgemv() is a Fortran routine with two dimensional array arguments that are column major order and use the lda argument to indice the length of each column allocation (which can be larger than the length of the column actually used). So Xgemv() can only be used if it is "as if" all the columns have the same lda. Now if the collection of vectors were obtained by seperate calls to VecCreate() then their array addresses would have no relationship to each other because each was obtained with a seperate malloc() so one cannot take advantage of Xgemv(). But KSPGMRES (and friends) use (inside KSPSolve_GMRES()) VecDuplicateVecs() to obtain the vectors and the default VecDuplicateVecs() uses a single malloc to obtain space for all the needed vector arrays it creates so each vector's array is immediately after the previous vector's array. Hence all the vector's arrays have the same offset from the previous vector's array and so Xgemv() can be used. Finally to take maximum advantage of the Xgemv() you want KSPGMES to allocate all its needed work vectors together (for example with the default restart of 30 you would want it to allocate all 30 vectors together so that the Xgemv() can be used on all 30 vectors together). You can ensure this by using the option -ksp_gmres_preallocate We do not do this by default because often with good preconditioners one converges before it even gets to the restart that is set, instead we allocate GMRES_DELTA_DIRECTIONS which is 10 at a time if needed. It turns out using Xgemv(), for example, 3 times with 10 arrays each is not much slower than using Xgemv once with 30 arrays so turning on -ksp_gmres_preallocate doesn't buy you much but you might see a nonzero improvement in the time. So if you are hunting for that final 1 percent time improvement make sure you have a good restart number for your problem (the important part and I only know how to do this by experimentation) and use -ksp_gmres_preallocate Barry > but not for MAXPY (since performance here was worse with vendor GEMV). It is difficult (or impossible) to tell which algorithm will be better without testing, so there are options to select each one. > > Thanks, > > Matt > >> Thanks a lot. I wish you a nice weekend, Frank >> >> >> >> >> >> >> >> >> >> > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!eCIItN2ljxrUFEI_LvvVwVwAG2qdF1PWwMd8oxEqcQZyjFiDYWZl3oRpMBgtBwUfaVZeC4AW2FwAfRjH8SzO0Qs$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From kiranhegde75 at gmail.com Sun Aug 10 03:12:53 2025 From: kiranhegde75 at gmail.com (kiran hegde) Date: Sun, 10 Aug 2025 13:42:53 +0530 Subject: [petsc-users] petsc-users Digest, Vol 200, Issue 3 In-Reply-To: <0D4EBFFD-AE78-49ED-8E1D-2C4E1B9A3D1E@petsc.dev> References: <0D4EBFFD-AE78-49ED-8E1D-2C4E1B9A3D1E@petsc.dev> Message-ID: Hi, The error highlighted is gone, the path to the include file was wrong. The code was developed during petsc-3.9 and it looks like i have to fix many of the petsc call functions. There are big changes in post petsc-3.9. Currently, the error shifted to another location in the code: for the below line i am getting error: call VecCreateSeqWithArray(PETSC_COMM_WORLD, nvar, nvar*g%nvl, & s%res(1,1), p%v_res, ierr); CHKERRQ(ierr) Error: There is no specific subroutine for the generic 'veccreateseqwitharray' at (1) Regards, Kiran On Fri, Aug 8, 2025 at 6:39?AM Barry Smith wrote: > > Please send (cut and paste) all the output of > > make ex5f (should succeed) > > and > > make "your code" > > we need to see the exact calls for both codes and the exact error message. > > > > > On Aug 5, 2025, at 12:44?AM, kiran hegde wrote: > > Hi, > > Sorry, the tutorial case run succeeded. No errors reported. When i used > the makefile file flags in my code, my code is still reported error. > > Regards, > > Kiran Hegde > Mobile:+91 9972600235 > > On Tue, Aug 5, 2025, 6:11?AM Barry Smith wrote: > >> >> "I have tried the options ex5f, but getting the same error." >> >> So you did >> >> cd src/snes/tutorials/ >> make ex5f >> >> and got some errors? Please cut and paste all the output from make >> ex5f and sent it here. >> >> >> >> On Aug 4, 2025, at 11:15?AM, kiran hegde wrote: >> >> Hi Satish, >> >> I have tried the options ex5f, but getting the same error. >> The code was running without any issues long back and I was using petsc >> ver 3.9 using gcc 12.X.X. >> My makefile flags are : >> >> FFLAGS = -march=native -O3 -pipe -ffree-form -cpp -ffree-line-length-none >> \ >> -Wall -Wextra -Wtabs -Wintrinsics-std >> -Wsurprising -Waliasing \ >> -Wno-unused-parameter -Wno-unused-dummy-argument \ >> -D$(MODEL) \ >> -DVERSION=\"$(GIT_VERSION)\" \ >> -I$(PETSC_DIR)/include >> FPPFLAGS = -cpp -E -dM -C -P -D$(MODEL) >> LDFLAGS = -lpetsc -L$(PETSC_DIR)/lib >> LDFLAGS += -Wl,-rpath=$(PETSC_DIR)/lib >> >> Have i missed anything? >> >> >> Regards, >> Kiran >> >> >> >> >> On Sun, Aug 3, 2025 at 9:56?PM Satish Balay >> wrote: >> >>> On Sun, 3 Aug 2025, kiran hegde wrote: >>> >>> > Hi, >>> > >>> > Please find the attached configure.log file. >>> > PETSC installation successfully completes with make check, without any >>> > error. >>> > But, when compiling fortan codes with petsc library+include in the >>> > makefile i get this error. >>> >>> Then likely your compile command is missing some options. you can: >>> >>> - compile a petsc example - say: >>> cd src/snes/tutorials >>> make ex5f >>> >>> - now compile your application >>> >>> - And compare the compile commands and see if there are any differences >>> (missing options, wrong order of include files etc..) >>> >>> Satish >>> >> >> >> -- >> Regards, >> >> Kiran Hegde >> Mobile:+91 9972600235 >> >> >> > -- Regards, Kiran Hegde Mobile:+91 9972600235 -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Mon Aug 11 09:54:17 2025 From: bsmith at petsc.dev (Barry Smith) Date: Mon, 11 Aug 2025 10:54:17 -0400 Subject: [petsc-users] petsc-users Digest, Vol 200, Issue 3 In-Reply-To: References: <0D4EBFFD-AE78-49ED-8E1D-2C4E1B9A3D1E@petsc.dev> Message-ID: PETSc now provides Fortran interface definitions in the Fortran modules for most PETSc functions. This means the argument types and array dimensions of the function must match the expected type and dimension. In this case, the argument s%res(1,1) is likely a scalar while VecCreateSeqWithArray is expected to be a one-dimensional array. You can change your code to declare res(1,1) to be a one-dimensional array, or possibly use reshape to have the compiler treat it as a one-dimensional array when passing it in as an argument. > On Aug 10, 2025, at 4:12?AM, kiran hegde wrote: > > Hi, > The error highlighted is gone, the path to the include file was wrong. > The code was developed during petsc-3.9 and it looks like i have to fix many of the petsc call functions. > There are big changes in post petsc-3.9. > > Currently, the error shifted to another location in the code: > for the below line i am getting error: > call VecCreateSeqWithArray(PETSC_COMM_WORLD, nvar, nvar*g%nvl, & > s%res(1,1), p%v_res, ierr); CHKERRQ(ierr) > > Error: There is no specific subroutine for the generic 'veccreateseqwitharray' at (1) > > Regards, > Kiran > > > > > On Fri, Aug 8, 2025 at 6:39?AM Barry Smith > wrote: >> >> Please send (cut and paste) all the output of >> >> make ex5f (should succeed) >> >> and >> >> make "your code" >> >> we need to see the exact calls for both codes and the exact error message. >> >> >> >> >>> On Aug 5, 2025, at 12:44?AM, kiran hegde > wrote: >>> >>> Hi, >>> >>> Sorry, the tutorial case run succeeded. No errors reported. When i used the makefile file flags in my code, my code is still reported error. >>> >>> Regards, >>> >>> Kiran Hegde >>> Mobile:+91 9972600235 >>> >>> On Tue, Aug 5, 2025, 6:11?AM Barry Smith > wrote: >>>> >>>> "I have tried the options ex5f, but getting the same error." >>>> >>>> So you did >>>> >>>> cd src/snes/tutorials/ >>>> make ex5f >>>> >>>> and got some errors? Please cut and paste all the output from make ex5f and sent it here. >>>> >>>> >>>> >>>>> On Aug 4, 2025, at 11:15?AM, kiran hegde > wrote: >>>>> >>>>> Hi Satish, >>>>> >>>>> I have tried the options ex5f, but getting the same error. >>>>> The code was running without any issues long back and I was using petsc ver 3.9 using gcc 12.X.X. >>>>> My makefile flags are : >>>>> >>>>> FFLAGS = -march=native -O3 -pipe -ffree-form -cpp -ffree-line-length-none \ >>>>> -Wall -Wextra -Wtabs -Wintrinsics-std -Wsurprising -Waliasing \ >>>>> -Wno-unused-parameter -Wno-unused-dummy-argument \ >>>>> -D$(MODEL) \ >>>>> -DVERSION=\"$(GIT_VERSION)\" \ >>>>> -I$(PETSC_DIR)/include >>>>> FPPFLAGS = -cpp -E -dM -C -P -D$(MODEL) >>>>> LDFLAGS = -lpetsc -L$(PETSC_DIR)/lib >>>>> LDFLAGS += -Wl,-rpath=$(PETSC_DIR)/lib >>>>> >>>>> Have i missed anything? >>>>> >>>>> >>>>> Regards, >>>>> Kiran >>>>> >>>>> >>>>> >>>>> >>>>> On Sun, Aug 3, 2025 at 9:56?PM Satish Balay > wrote: >>>>>> On Sun, 3 Aug 2025, kiran hegde wrote: >>>>>> >>>>>> > Hi, >>>>>> > >>>>>> > Please find the attached configure.log file. >>>>>> > PETSC installation successfully completes with make check, without any >>>>>> > error. >>>>>> > But, when compiling fortan codes with petsc library+include in the >>>>>> > makefile i get this error. >>>>>> >>>>>> Then likely your compile command is missing some options. you can: >>>>>> >>>>>> - compile a petsc example - say: >>>>>> cd src/snes/tutorials >>>>>> make ex5f >>>>>> >>>>>> - now compile your application >>>>>> >>>>>> - And compare the compile commands and see if there are any differences (missing options, wrong order of include files etc..) >>>>>> >>>>>> Satish >>>>> >>>>> >>>>> >>>>> -- >>>>> Regards, >>>>> >>>>> Kiran Hegde >>>>> Mobile:+91 9972600235 >>>> >> > > > > -- > Regards, > > Kiran Hegde > Mobile:+91 9972600235 -------------- next part -------------- An HTML attachment was scrubbed... URL: From s_g at berkeley.edu Mon Aug 11 11:08:22 2025 From: s_g at berkeley.edu (Sanjay Govindjee) Date: Mon, 11 Aug 2025 09:08:22 -0700 Subject: [petsc-users] petsc-users Digest, Vol 200, Issue 3 In-Reply-To: References: <0D4EBFFD-AE78-49ED-8E1D-2C4E1B9A3D1E@petsc.dev> Message-ID: Would a construct like call VecCreateSeqWithArray(PETSC_COMM_WORLD, nvar, nvar*g%nvl, & [s%res(1,1)] , p%v_res, ierr); CHKERRQ(ierr) or call VecCreateSeqWithArray(PETSC_COMM_WORLD, nvar, nvar*g%nvl, & (/ s%res(1,1) /), p%v_res, ierr); CHKERRQ(ierr) be sufficient in this case? -sanjay On Mon, Aug 11, 2025 at 7:54?AM Barry Smith wrote: > > PETSc now provides Fortran interface definitions in the Fortran modules > for most PETSc functions. This means the argument types and array > dimensions of the function must match the expected type and dimension. > > In this case, the argument s%res(1,1) is likely a scalar while > VecCreateSeqWithArray is expected to be a one-dimensional array. You can > change your code to declare res(1,1) to be a one-dimensional array, or > possibly use reshape to have the compiler treat it as a one-dimensional > array when passing it in as an argument. > > > > On Aug 10, 2025, at 4:12?AM, kiran hegde wrote: > > Hi, > The error highlighted is gone, the path to the include file was wrong. > The code was developed during petsc-3.9 and it looks like i have to fix > many of the petsc call functions. > There are big changes in post petsc-3.9. > > Currently, the error shifted to another location in the code: > for the below line i am getting error: > call VecCreateSeqWithArray(PETSC_COMM_WORLD, nvar, nvar*g%nvl, & > s%res(1,1), p%v_res, ierr); CHKERRQ(ierr) > > Error: There is no specific subroutine for the generic > 'veccreateseqwitharray' at (1) > > Regards, > Kiran > > > > > On Fri, Aug 8, 2025 at 6:39?AM Barry Smith wrote: > >> >> Please send (cut and paste) all the output of >> >> make ex5f (should succeed) >> >> and >> >> make "your code" >> >> we need to see the exact calls for both codes and the exact error message. >> >> >> >> >> On Aug 5, 2025, at 12:44?AM, kiran hegde wrote: >> >> Hi, >> >> Sorry, the tutorial case run succeeded. No errors reported. When i used >> the makefile file flags in my code, my code is still reported error. >> >> Regards, >> >> Kiran Hegde >> Mobile:+91 9972600235 >> >> On Tue, Aug 5, 2025, 6:11?AM Barry Smith wrote: >> >>> >>> "I have tried the options ex5f, but getting the same error." >>> >>> So you did >>> >>> cd src/snes/tutorials/ >>> make ex5f >>> >>> and got some errors? Please cut and paste all the output from make >>> ex5f and sent it here. >>> >>> >>> >>> On Aug 4, 2025, at 11:15?AM, kiran hegde wrote: >>> >>> Hi Satish, >>> >>> I have tried the options ex5f, but getting the same error. >>> The code was running without any issues long back and I was using petsc >>> ver 3.9 using gcc 12.X.X. >>> My makefile flags are : >>> >>> FFLAGS = -march=native -O3 -pipe -ffree-form -cpp >>> -ffree-line-length-none \ >>> -Wall -Wextra -Wtabs -Wintrinsics-std >>> -Wsurprising -Waliasing \ >>> -Wno-unused-parameter -Wno-unused-dummy-argument >>> \ >>> -D$(MODEL) \ >>> -DVERSION=\"$(GIT_VERSION)\" \ >>> -I$(PETSC_DIR)/include >>> FPPFLAGS = -cpp -E -dM -C -P -D$(MODEL) >>> LDFLAGS = -lpetsc -L$(PETSC_DIR)/lib >>> LDFLAGS += -Wl,-rpath=$(PETSC_DIR)/lib >>> >>> Have i missed anything? >>> >>> >>> Regards, >>> Kiran >>> >>> >>> >>> >>> On Sun, Aug 3, 2025 at 9:56?PM Satish Balay >>> wrote: >>> >>>> On Sun, 3 Aug 2025, kiran hegde wrote: >>>> >>>> > Hi, >>>> > >>>> > Please find the attached configure.log file. >>>> > PETSC installation successfully completes with make check, without any >>>> > error. >>>> > But, when compiling fortan codes with petsc library+include in the >>>> > makefile i get this error. >>>> >>>> Then likely your compile command is missing some options. you can: >>>> >>>> - compile a petsc example - say: >>>> cd src/snes/tutorials >>>> make ex5f >>>> >>>> - now compile your application >>>> >>>> - And compare the compile commands and see if there are any differences >>>> (missing options, wrong order of include files etc..) >>>> >>>> Satish >>>> >>> >>> >>> -- >>> Regards, >>> >>> Kiran Hegde >>> Mobile:+91 9972600235 >>> >>> >>> >> > > -- > Regards, > > Kiran Hegde > Mobile:+91 9972600235 > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Mon Aug 11 11:21:12 2025 From: bsmith at petsc.dev (Barry Smith) Date: Mon, 11 Aug 2025 12:21:12 -0400 Subject: [petsc-users] petsc-users Digest, Vol 200, Issue 3 In-Reply-To: References: <0D4EBFFD-AE78-49ED-8E1D-2C4E1B9A3D1E@petsc.dev> Message-ID: <6956FCDB-7F4C-4B21-896B-D0D58DDF3618@petsc.dev> I do not know. It depends on how the Fortran compiler decides to interprete the [] and pass the address down. In the end the address of the actual array in p has to sent down since it will be used later so if the compiler makes a new array with [] or does something fancy it may or may not work. > On Aug 11, 2025, at 12:08?PM, Sanjay Govindjee wrote: > > > Would a construct like > call VecCreateSeqWithArray(PETSC_COMM_WORLD, nvar, nvar*g%nvl, & > [s%res(1,1)] , p%v_res, ierr); CHKERRQ(ierr) > or > call VecCreateSeqWithArray(PETSC_COMM_WORLD, nvar, nvar*g%nvl, & > (/ s%res(1,1) /), p%v_res, ierr); CHKERRQ(ierr) > be sufficient in this case? > > -sanjay > > On Mon, Aug 11, 2025 at 7:54?AM Barry Smith > wrote: >> >> PETSc now provides Fortran interface definitions in the Fortran modules for most PETSc functions. This means the argument types and array dimensions of the function must match the expected type and dimension. >> >> In this case, the argument s%res(1,1) is likely a scalar while VecCreateSeqWithArray is expected to be a one-dimensional array. You can change your code to declare res(1,1) to be a one-dimensional array, or possibly use reshape to have the compiler treat it as a one-dimensional array when passing it in as an argument. >> >> >> >>> On Aug 10, 2025, at 4:12?AM, kiran hegde > wrote: >>> >>> Hi, >>> The error highlighted is gone, the path to the include file was wrong. >>> The code was developed during petsc-3.9 and it looks like i have to fix many of the petsc call functions. >>> There are big changes in post petsc-3.9. >>> >>> Currently, the error shifted to another location in the code: >>> for the below line i am getting error: >>> call VecCreateSeqWithArray(PETSC_COMM_WORLD, nvar, nvar*g%nvl, & >>> s%res(1,1), p%v_res, ierr); CHKERRQ(ierr) >>> >>> Error: There is no specific subroutine for the generic 'veccreateseqwitharray' at (1) >>> >>> Regards, >>> Kiran >>> >>> >>> >>> >>> On Fri, Aug 8, 2025 at 6:39?AM Barry Smith > wrote: >>>> >>>> Please send (cut and paste) all the output of >>>> >>>> make ex5f (should succeed) >>>> >>>> and >>>> >>>> make "your code" >>>> >>>> we need to see the exact calls for both codes and the exact error message. >>>> >>>> >>>> >>>> >>>>> On Aug 5, 2025, at 12:44?AM, kiran hegde > wrote: >>>>> >>>>> Hi, >>>>> >>>>> Sorry, the tutorial case run succeeded. No errors reported. When i used the makefile file flags in my code, my code is still reported error. >>>>> >>>>> Regards, >>>>> >>>>> Kiran Hegde >>>>> Mobile:+91 9972600235 >>>>> >>>>> On Tue, Aug 5, 2025, 6:11?AM Barry Smith > wrote: >>>>>> >>>>>> "I have tried the options ex5f, but getting the same error." >>>>>> >>>>>> So you did >>>>>> >>>>>> cd src/snes/tutorials/ >>>>>> make ex5f >>>>>> >>>>>> and got some errors? Please cut and paste all the output from make ex5f and sent it here. >>>>>> >>>>>> >>>>>> >>>>>>> On Aug 4, 2025, at 11:15?AM, kiran hegde > wrote: >>>>>>> >>>>>>> Hi Satish, >>>>>>> >>>>>>> I have tried the options ex5f, but getting the same error. >>>>>>> The code was running without any issues long back and I was using petsc ver 3.9 using gcc 12.X.X. >>>>>>> My makefile flags are : >>>>>>> >>>>>>> FFLAGS = -march=native -O3 -pipe -ffree-form -cpp -ffree-line-length-none \ >>>>>>> -Wall -Wextra -Wtabs -Wintrinsics-std -Wsurprising -Waliasing \ >>>>>>> -Wno-unused-parameter -Wno-unused-dummy-argument \ >>>>>>> -D$(MODEL) \ >>>>>>> -DVERSION=\"$(GIT_VERSION)\" \ >>>>>>> -I$(PETSC_DIR)/include >>>>>>> FPPFLAGS = -cpp -E -dM -C -P -D$(MODEL) >>>>>>> LDFLAGS = -lpetsc -L$(PETSC_DIR)/lib >>>>>>> LDFLAGS += -Wl,-rpath=$(PETSC_DIR)/lib >>>>>>> >>>>>>> Have i missed anything? >>>>>>> >>>>>>> >>>>>>> Regards, >>>>>>> Kiran >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Sun, Aug 3, 2025 at 9:56?PM Satish Balay > wrote: >>>>>>>> On Sun, 3 Aug 2025, kiran hegde wrote: >>>>>>>> >>>>>>>> > Hi, >>>>>>>> > >>>>>>>> > Please find the attached configure.log file. >>>>>>>> > PETSC installation successfully completes with make check, without any >>>>>>>> > error. >>>>>>>> > But, when compiling fortan codes with petsc library+include in the >>>>>>>> > makefile i get this error. >>>>>>>> >>>>>>>> Then likely your compile command is missing some options. you can: >>>>>>>> >>>>>>>> - compile a petsc example - say: >>>>>>>> cd src/snes/tutorials >>>>>>>> make ex5f >>>>>>>> >>>>>>>> - now compile your application >>>>>>>> >>>>>>>> - And compare the compile commands and see if there are any differences (missing options, wrong order of include files etc..) >>>>>>>> >>>>>>>> Satish >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Regards, >>>>>>> >>>>>>> Kiran Hegde >>>>>>> Mobile:+91 9972600235 >>>>>> >>>> >>> >>> >>> >>> -- >>> Regards, >>> >>> Kiran Hegde >>> Mobile:+91 9972600235 >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From yongzhong.li at mail.utoronto.ca Mon Aug 11 11:31:49 2025 From: yongzhong.li at mail.utoronto.ca (Yongzhong Li) Date: Mon, 11 Aug 2025 16:31:49 +0000 Subject: [petsc-users] Inquiry about configuring PETSc with AOCL BLAS/LAPACK Message-ID: Dear PETSc?s developer, Hi, I am a user of PETSc. I have some questions about how we can configure PETSc with AOCL BLAS and LAPACK. Previously, we linked PETSc with Intel MKL BLAS. This solution provides us with much better multithreading capability for sparse matrix-vector product compared with configuring PETSc with OpenBLAS. Now, our compute nodes have been upgraded with AMD CPUs, we are considering switching from Intel MKL to AMD AOCL. My questions are: 1. If we configure PETSc in compile time with ?with-blaslapack-dir = $AOCLROOT, will we be able to use AOCL BLAS as the backend of PETSc MatMult() API? 2. What if AOCL BLAS and AOCL LAPACK are installed in two different directories, not under AOCLROOT? 3. PETSc has MatAIJMKL type for sparse matrix stored in Intel MKL format. Does PETSc also have another type for AMD AOCL? Thanks! Yongzhong -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Aug 11 11:44:30 2025 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 11 Aug 2025 12:44:30 -0400 Subject: [petsc-users] Inquiry about configuring PETSc with AOCL BLAS/LAPACK In-Reply-To: References: Message-ID: On Mon, Aug 11, 2025 at 12:32?PM Yongzhong Li wrote: > Dear PETSc?s developer, > > Hi, I am a user of PETSc. I have some questions about how we can configure > PETSc with AOCL BLAS and LAPACK. > > > > Previously, we linked PETSc with Intel MKL BLAS. This solution provides us > with much better multithreading capability for sparse matrix-vector product > compared with configuring PETSc with OpenBLAS. Now, our compute nodes have > been upgraded with AMD CPUs, we are considering switching from Intel MKL to > AMD AOCL. > > My questions are: > > > > 1. If we configure PETSc in compile time with *?with-blaslapack-dir = > $AOCLROOT*, will we be able to use AOCL BLAS as the backend of PETSc > MatMult() API? > > Do you mean use the AMD sparse matvec? We have a HIP implementation ( https://urldefense.us/v3/__https://petsc.org/main/manualpages/Mat/MATAIJHIPSPARSE/__;!!G_uCfscf7eWS!buskXbl-Io-3aDFucXGpiYbfFRVN3fRcpRosr-WXF1em8NBn5a8PHltFj0SILAhBQspywGsr9f7BuSqHFfDA$ ), but nothing for AOCL comparable to the AIJMKL class. If you think we need it, we would certainly help implement it. > > 1. > 2. What if AOCL BLAS and AOCL LAPACK are installed in two different > directories, not under AOCLROOT? > > You would use -with-blaslapack-lib=[liblist] > > 1. > 2. PETSc has MatAIJMKL type for sparse matrix stored in Intel MKL > format. Does PETSc also have another type for AMD AOCL? > > No, but it would be straightforward to add. Thanks, Matt > Thanks! > > Yongzhong > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!buskXbl-Io-3aDFucXGpiYbfFRVN3fRcpRosr-WXF1em8NBn5a8PHltFj0SILAhBQspywGsr9f7BuSj6N8eE$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From yongzhong.li at mail.utoronto.ca Mon Aug 11 16:26:25 2025 From: yongzhong.li at mail.utoronto.ca (Yongzhong Li) Date: Mon, 11 Aug 2025 21:26:25 +0000 Subject: [petsc-users] Inquiry about configuring PETSc with AOCL BLAS/LAPACK In-Reply-To: References: Message-ID: Thank you Matt! It appears that the HIP implementation is designed for sparse matvec on AMD GPUs, similar to CUDA for Nvidia GPUs, is that correct? We are primarily focusing on CPUs with AOCL. It would be great if PETSc offered an AOCL AIJ class. We could use AOCL in the same manner as we use MKL. In the meantime, I wonder if I can simply use MATAIJ-type matrices and configure PETSc with AOCL during compilation. Will the MatMult() API then use AOCL BLAS as the backend? Thanks, Yongzhong From: Matthew Knepley Date: Monday, August 11, 2025 at 12:44?PM To: Yongzhong Li Cc: petsc-users at mcs.anl.gov , petsc-maint at mcs.anl.gov , Jasper Hatton , Piero Triverio , Atacan Tuhan Subject: Re: [petsc-users] Inquiry about configuring PETSc with AOCL BLAS/LAPACK On Mon, Aug 11, 2025 at 12:32?PM Yongzhong Li > wrote: Dear PETSc?s developer, Hi, I am a user of PETSc. I have some questions about how we can configure PETSc with AOCL BLAS and LAPACK. Previously, we linked PETSc with Intel MKL BLAS. This solution provides us with much better multithreading capability for sparse matrix-vector product compared with configuring PETSc with OpenBLAS. Now, our compute nodes have been upgraded with AMD CPUs, we are considering switching from Intel MKL to AMD AOCL. My questions are: 1. If we configure PETSc in compile time with ?with-blaslapack-dir = $AOCLROOT, will we be able to use AOCL BLAS as the backend of PETSc MatMult() API? Do you mean use the AMD sparse matvec? We have a HIP implementation (https://urldefense.us/v3/__https://petsc.org/main/manualpages/Mat/MATAIJHIPSPARSE/__;!!G_uCfscf7eWS!erObo4bZtlWw2VuQgcNv1mF6FQ0629E3AUF29xJ1MC425B_G8NL2xhSbiULNL58ZGQcMnQWy85kVeTiFbTeeREz4Ubko4R7i9vQ$ ), but nothing for AOCL comparable to the AIJMKL class. If you think we need it, we would certainly help implement it. 1. 2. What if AOCL BLAS and AOCL LAPACK are installed in two different directories, not under AOCLROOT? You would use -with-blaslapack-lib=[liblist] 1. 2. PETSc has MatAIJMKL type for sparse matrix stored in Intel MKL format. Does PETSc also have another type for AMD AOCL? No, but it would be straightforward to add. Thanks, Matt Thanks! Yongzhong -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!erObo4bZtlWw2VuQgcNv1mF6FQ0629E3AUF29xJ1MC425B_G8NL2xhSbiULNL58ZGQcMnQWy85kVeTiFbTeeREz4UbkoEUTinu8$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Mon Aug 11 18:33:51 2025 From: bsmith at petsc.dev (Barry Smith) Date: Mon, 11 Aug 2025 19:33:51 -0400 Subject: [petsc-users] Inquiry about configuring PETSc with AOCL BLAS/LAPACK In-Reply-To: References: Message-ID: <39B792C5-4D1E-41F5-9957-7AC2FF9ECEE4@petsc.dev> > On Aug 11, 2025, at 5:26?PM, Yongzhong Li via petsc-users wrote: > > Thank you Matt! > > It appears that the HIP implementation is designed for sparse matvec on AMD GPUs, similar to CUDA for Nvidia GPUs, is that correct? > > We are primarily focusing on CPUs with AOCL. It would be great if PETSc offered an AOCL AIJ class. We could use AOCL in the same manner as we use MKL. > > In the meantime, I wonder if I can simply use MATAIJ-type matrices and configure PETSc with AOCL during compilation. Will the MatMult() API then use AOCL BLAS as the backend? BLAS (and LAPACK) are for vectors and dense matrices so the quality (or lack of) of BLAS and LAPACK doesn't have any affect on the MATAIJ operations which are sparse matrix operations. So yes use AOCL BLAS and MATAIJ for your sparse matrices. > > Thanks, > Yongzhong > > > > > From: Matthew Knepley > > Date: Monday, August 11, 2025 at 12:44?PM > To: Yongzhong Li > > Cc: petsc-users at mcs.anl.gov >, petsc-maint at mcs.anl.gov >, Jasper Hatton >, Piero Triverio >, Atacan Tuhan > > Subject: Re: [petsc-users] Inquiry about configuring PETSc with AOCL BLAS/LAPACK > > On Mon, Aug 11, 2025 at 12:32?PM Yongzhong Li > wrote: > Dear PETSc?s developer, > > Hi, I am a user of PETSc. I have some questions about how we can configure PETSc with AOCL BLAS and LAPACK. > > Previously, we linked PETSc with Intel MKL BLAS. This solution provides us with much better multithreading capability for sparse matrix-vector product compared with configuring PETSc with OpenBLAS. Now, our compute nodes have been upgraded with AMD CPUs, we are considering switching from Intel MKL to AMD AOCL. > > My questions are: > > If we configure PETSc in compile time with ?with-blaslapack-dir = $AOCLROOT, will we be able to use AOCL BLAS as the backend of PETSc MatMult() API? > > Do you mean use the AMD sparse matvec? We have a HIP implementation (https://urldefense.us/v3/__https://petsc.org/main/manualpages/Mat/MATAIJHIPSPARSE/__;!!G_uCfscf7eWS!eyVo6h2c-IbELNug6c05khR0jl7oQV3tAilAdQ7grI3_V2I9VXubG17fp0RKIcLhP1sVbNxZOr_mdLuzo1a6fi0$ ), but nothing for AOCL comparable to the AIJMKL class. If you think we need it, we would certainly help implement it. > > > What if AOCL BLAS and AOCL LAPACK are installed in two different directories, not under AOCLROOT? > > You would use -with-blaslapack-lib=[liblist] > > > PETSc has MatAIJMKL type for sparse matrix stored in Intel MKL format. Does PETSc also have another type for AMD AOCL? > > No, but it would be straightforward to add. > > Thanks, > > Matt > > Thanks! > Yongzhong > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!eyVo6h2c-IbELNug6c05khR0jl7oQV3tAilAdQ7grI3_V2I9VXubG17fp0RKIcLhP1sVbNxZOr_mdLuzM-9yasU$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From kiranhegde75 at gmail.com Wed Aug 13 00:28:27 2025 From: kiranhegde75 at gmail.com (kiran hegde) Date: Wed, 13 Aug 2025 10:58:27 +0530 Subject: [petsc-users] petsc-users Digest, Vol 200, Issue 3 In-Reply-To: <6956FCDB-7F4C-4B21-896B-D0D58DDF3618@petsc.dev> References: <0D4EBFFD-AE78-49ED-8E1D-2C4E1B9A3D1E@petsc.dev> <6956FCDB-7F4C-4B21-896B-D0D58DDF3618@petsc.dev> Message-ID: Dear Sanjay, Both the construction works, no error reported. I need to test the code further to see if it really passes the address. There are a few more errors reported afterwards. Eg. preconditioner (PC) is now included in petsc/finclude/petscpc.h whereas in older versions it is available from petsc/finclude/petscpc.h. I will try to fix issues and update to you later : call VecCreateSeqWithArray(PETSC_COMM_WORLD, nvar, nvar*g%nvl, & [s%res(1,1)] , p%v_res, ierr); CHKERRQ(ierr) OR call VecCreateSeqWithArray(PETSC_COMM_WORLD, nvar, nvar*g%nvl, & (/ s%res(1,1) /), p%v_res, ierr); CHKERRQ(ierr) On Mon, Aug 11, 2025 at 9:51?PM Barry Smith wrote: > > I do not know. It depends on how the Fortran compiler decides to > interprete the [] and pass the address down. In the end the address of the > actual array in p has to sent down since it will be used later so if the > compiler makes a new array with [] or does something fancy it may or may > not work. > > On Aug 11, 2025, at 12:08?PM, Sanjay Govindjee wrote: > > > Would a construct like > call VecCreateSeqWithArray(PETSC_COMM_WORLD, nvar, nvar*g%nvl, & > [s%res(1,1)] , p%v_res, ierr); > CHKERRQ(ierr) > or > call VecCreateSeqWithArray(PETSC_COMM_WORLD, nvar, nvar*g%nvl, & > (/ s%res(1,1) /), p%v_res, ierr); > CHKERRQ(ierr) > be sufficient in this case? > > -sanjay > > On Mon, Aug 11, 2025 at 7:54?AM Barry Smith wrote: > >> >> PETSc now provides Fortran interface definitions in the Fortran modules >> for most PETSc functions. This means the argument types and array >> dimensions of the function must match the expected type and dimension. >> >> In this case, the argument s%res(1,1) is likely a scalar while >> VecCreateSeqWithArray is expected to be a one-dimensional array. You can >> change your code to declare res(1,1) to be a one-dimensional array, or >> possibly use reshape to have the compiler treat it as a one-dimensional >> array when passing it in as an argument. >> >> >> >> On Aug 10, 2025, at 4:12?AM, kiran hegde wrote: >> >> Hi, >> The error highlighted is gone, the path to the include file was wrong. >> The code was developed during petsc-3.9 and it looks like i have to fix >> many of the petsc call functions. >> There are big changes in post petsc-3.9. >> >> Currently, the error shifted to another location in the code: >> for the below line i am getting error: >> call VecCreateSeqWithArray(PETSC_COMM_WORLD, nvar, nvar*g%nvl, & >> s%res(1,1), p%v_res, ierr); CHKERRQ(ierr) >> >> Error: There is no specific subroutine for the generic >> 'veccreateseqwitharray' at (1) >> >> Regards, >> Kiran >> >> >> >> >> On Fri, Aug 8, 2025 at 6:39?AM Barry Smith wrote: >> >>> >>> Please send (cut and paste) all the output of >>> >>> make ex5f (should succeed) >>> >>> and >>> >>> make "your code" >>> >>> we need to see the exact calls for both codes and the exact error >>> message. >>> >>> >>> >>> >>> On Aug 5, 2025, at 12:44?AM, kiran hegde wrote: >>> >>> Hi, >>> >>> Sorry, the tutorial case run succeeded. No errors reported. When i used >>> the makefile file flags in my code, my code is still reported error. >>> >>> Regards, >>> >>> Kiran Hegde >>> Mobile:+91 9972600235 >>> >>> On Tue, Aug 5, 2025, 6:11?AM Barry Smith wrote: >>> >>>> >>>> "I have tried the options ex5f, but getting the same error." >>>> >>>> So you did >>>> >>>> cd src/snes/tutorials/ >>>> make ex5f >>>> >>>> and got some errors? Please cut and paste all the output from make >>>> ex5f and sent it here. >>>> >>>> >>>> >>>> On Aug 4, 2025, at 11:15?AM, kiran hegde >>>> wrote: >>>> >>>> Hi Satish, >>>> >>>> I have tried the options ex5f, but getting the same error. >>>> The code was running without any issues long back and I was using >>>> petsc ver 3.9 using gcc 12.X.X. >>>> My makefile flags are : >>>> >>>> FFLAGS = -march=native -O3 -pipe -ffree-form -cpp >>>> -ffree-line-length-none \ >>>> -Wall -Wextra -Wtabs -Wintrinsics-std >>>> -Wsurprising -Waliasing \ >>>> -Wno-unused-parameter >>>> -Wno-unused-dummy-argument \ >>>> -D$(MODEL) \ >>>> -DVERSION=\"$(GIT_VERSION)\" \ >>>> -I$(PETSC_DIR)/include >>>> FPPFLAGS = -cpp -E -dM -C -P -D$(MODEL) >>>> LDFLAGS = -lpetsc -L$(PETSC_DIR)/lib >>>> LDFLAGS += -Wl,-rpath=$(PETSC_DIR)/lib >>>> >>>> Have i missed anything? >>>> >>>> >>>> Regards, >>>> Kiran >>>> >>>> >>>> >>>> >>>> On Sun, Aug 3, 2025 at 9:56?PM Satish Balay >>>> wrote: >>>> >>>>> On Sun, 3 Aug 2025, kiran hegde wrote: >>>>> >>>>> > Hi, >>>>> > >>>>> > Please find the attached configure.log file. >>>>> > PETSC installation successfully completes with make check, without >>>>> any >>>>> > error. >>>>> > But, when compiling fortan codes with petsc library+include in the >>>>> > makefile i get this error. >>>>> >>>>> Then likely your compile command is missing some options. you can: >>>>> >>>>> - compile a petsc example - say: >>>>> cd src/snes/tutorials >>>>> make ex5f >>>>> >>>>> - now compile your application >>>>> >>>>> - And compare the compile commands and see if there are any >>>>> differences (missing options, wrong order of include files etc..) >>>>> >>>>> Satish >>>>> >>>> >>>> >>>> -- >>>> Regards, >>>> >>>> Kiran Hegde >>>> Mobile:+91 9972600235 >>>> >>>> >>>> >>> >> >> -- >> Regards, >> >> Kiran Hegde >> Mobile:+91 9972600235 >> >> >> > -- Regards, Kiran Hegde Mobile:+91 9972600235 -------------- next part -------------- An HTML attachment was scrubbed... URL: From ali.ali_ahmad at utt.fr Wed Aug 13 14:45:33 2025 From: ali.ali_ahmad at utt.fr (Ali ALI AHMAD) Date: Wed, 13 Aug 2025 21:45:33 +0200 (CEST) Subject: [petsc-users] [petsc-maint] norm L2 problemQuestion about changing the norm used in nonlinear solvers (L2 Euclidean vs. L2 Lebesgue) In-Reply-To: References: <414475981.6714047.1749631527145.JavaMail.zimbra@utt.fr> <323745907.8383516.1749804945465.JavaMail.zimbra@utt.fr> <85CD2CA9-7B77-4288-87BA-9E108D40C7E8@petsc.dev> <82133477.13270009.1750410624370.JavaMail.zimbra@utt.fr> <86F87C6A-DEB3-4125-AF51-2B2E577EBFDD@petsc.dev> <36326459.7563988.1753714811037.JavaMail.zimbra@utt.fr> Message-ID: <1185563625.11757748.1755114333036.JavaMail.zimbra@utt.fr> * As I mentioned earlier, the weighted norm I am referring to is: ||f||^2_2 = \int_{\Omega} |f(x)|^2 \, dx \approx \sum_{K \in \Omega} \sum_q |f(G_q)|^2 \, area(K) where G_q is the centroid of triangle K , computed using a first-order Gaussian quadrature rule. Yes, I am using the inexact Newton method, where the method stops when ||F(x_k) + J_{x_k} \, d_k|| \leq \nu_k \, ||F(x_k)||. Here, I would also like to change the norm to see whether this modification affects the quality of the descent direction. I know that these norms are theoretically equivalent, but I would like to test them to observe any difference between the two. My initial expectation is that the method may converge faster when using the L^2 norm. Best regards, Ali ALI AHMAD De: "Matthew Knepley" ?: "Ali ALI AHMAD" Cc: "Barry Smith" , "petsc-users" , "petsc-maint" Envoy?: Lundi 28 Juillet 2025 18:55:04 Objet: Re: [petsc-maint] norm L2 problemQuestion about changing the norm used in nonlinear solvers (L2 Euclidean vs. L2 Lebesgue) On Mon, Jul 28, 2025 at 11:00 AM Ali ALI AHMAD < [ mailto:ali.ali_ahmad at utt.fr | ali.ali_ahmad at utt.fr ] > wrote: I?m sorry for getting back to you so late. Thank you for your patience and understanding. * For example, when using L2 algorithms where a different norm is applied in the line search, see Line_search_L2.png as an example from this reference: [ https://urldefense.us/v3/__https://arxiv.org/abs/1607.04254__;!!G_uCfscf7eWS!ep85s6aXge00Uc1_kixR9yGEU0FBO9W8HTZ6LsVYzy9n9Jel5_r26mPMP138k1Ilhi2CAN6cR4DGe4kKZG997W1E0q3AIw$ | https://urldefense.us/v3/__https://arxiv.org/abs/1607.04254__;!!G_uCfscf7eWS!fiC1n5vGsbqOyw1uxsDvcoyysDPSuMVX-X60t2a22q6RxvgToFh_JTCqhACBSTB33XlfjUocJoQh741gmrYyKTsWaDMi_Q$ ] . Here, we can change the norm from the L2 Euclidean to the L2 Lebesgue norm. What does "L2 Lebesgue" mean here? I know what I mean by L_2. It is ||f||^2_2 = \int_Omega |f(x)|^2 dx \approx \sum_q |f(x_q)|^2 w_q Oh, from below it seems you want a weight function wt(x) in the norm. So we would have \sum_q |f(x_q)|^2 wt(x_q) w_q BQ_BEGIN * For GMRES, we need to replace NORM_2 (L2 Euclidean) in your code with weighted_NormL2 (L2 Lebesgue) everywhere, including all the details such as in the Arnoldi algorithm... BQ_END I do not quite understand here, because GMRES does not use the L:_2 norm. It uses the l_2 norm, which is ||v||^2_2 = \sum |v_i|^2 The things in the vector are coefficients of basis functions. I guess, if you had an interpolatory element, you could interpret this as quadrature rule, meaning you would have \sum wt(x_i) |v_i|^2 where x_i were the coordinates of the dual basis evaluation functionals. BQ_BEGIN * For the convergence test as well, in the linear system for finding the direction d (A ? d = b), BQ_END You want to use the inner product that generates your weighted L_2 norm? BQ_BEGIN * and also when we search for a good step using the line search formula: xk+1=xk+step?d. BQ_END You want to minimize your weighted L_2 norm. That should work in the same way. Thanks, Matt BQ_BEGIN I hope this explanation is clear for you. Best regards, Ali ALI AHMAD De: "Barry Smith" < [ mailto:bsmith at petsc.dev | bsmith at petsc.dev ] > ?: "Ali ALI AHMAD" < [ mailto:ali.ali_ahmad at utt.fr | ali.ali_ahmad at utt.fr ] > Cc: "petsc-users" < [ mailto:petsc-users at mcs.anl.gov | petsc-users at mcs.anl.gov ] >, "petsc-maint" < [ mailto:petsc-maint at mcs.anl.gov | petsc-maint at mcs.anl.gov ] > Envoy?: Vendredi 20 Juin 2025 15:50:17 Objet: Re: [petsc-maint] norm L2 problemQuestion about changing the norm used in nonlinear solvers (L2 Euclidean vs. L2 Lebesgue) BQ_BEGIN On Jun 20, 2025, at 5:10 AM, Ali ALI AHMAD < [ mailto:ali.ali_ahmad at utt.fr | ali.ali_ahmad at utt.fr ] > wrote: * Yes, I am indeed using an inexact Newton method in my code. The descent direction is computed by solving a linear system involving the Jacobian, so the update follows the classical formula "J(un)^{-1}d(un)=-F(un)" I'm also trying to use a line search strategy based on a weighted L2 norm (in the Lebesgue sense), which a priori should lead to better accuracy and faster convergence in anisotropic settings. BQ_END Ok, could you point to sample code (any language) or written algorithms where a different norm is used in the line search? BQ_BEGIN * During the subsequent iterations, I apply the Eisenstat?Walker method to adapt the tolerance, which should also involve modifying the norm used in the algorithm. * The current implementation still uses the standard Euclidean L2 norm in PETSc's linear solver and in GMRES. I believe this should ideally be replaced by a weighted L2 norm consistent with the discretization. However, I haven't yet succeeded in modifying the norm used internally by the linear solver in PETSc, so, I'm not yet sure how much impact this change would have on the overall convergence, but I suspect it could improve robustness, especially for highly anisotropic problems. I would greatly appreciate any guidance on how to implement this properly in PETSc. BQ_END Norms are used in multiple ways in GMRES. 1) defining convergence 2) as part of preconditioning Again can you point to sample code (any language) or written algorithms that describe exactly what you would like to accomplish. Barry BQ_BEGIN Do not hesitate to contact me again if anything remains unclear or if you need further information. Best regards, Ali ALI AHMAD De: "Barry Smith" < [ mailto:bsmith at petsc.dev | bsmith at petsc.dev ] > ?: "Ali ALI AHMAD" < [ mailto:ali.ali_ahmad at utt.fr | ali.ali_ahmad at utt.fr ] > Cc: "petsc-users" < [ mailto:petsc-users at mcs.anl.gov | petsc-users at mcs.anl.gov ] >, "petsc-maint" < [ mailto:petsc-maint at mcs.anl.gov | petsc-maint at mcs.anl.gov ] > Envoy?: Samedi 14 Juin 2025 01:06:52 Objet: Re: [petsc-maint] norm L2 problemQuestion about changing the norm used in nonlinear solvers (L2 Euclidean vs. L2 Lebesgue) I appreciate the clarification. I would call 3) preconditioning. To increase my understanding, you are already using Newton's method? That is, you compute the Jacobian of the function and use - J^{-1}(u^n) F(u^n) as your update direction? When you switch the inner product (or precondition) how will the search direction be different? Thanks Barry The case you need support for is becoming important to PETSc so we need to understand it well and support it well which is why I am asking these (perhaps to you) trivial questions. BQ_BEGIN On Jun 13, 2025, at 4:55 AM, Ali ALI AHMAD < [ mailto:ali.ali_ahmad at utt.fr | ali.ali_ahmad at utt.fr ] > wrote: Thank you for your message. To answer your question: I would like to use the L2 norm in the sense of Lebesgue for all three purposes , especially the third one . 1- For displaying residuals during the nonlinear iterations, I would like to observe the convergence behavior using a norm that better reflects the physical properties of the problem. 2- For convergence testing , I would like the stopping criterion to be based on a weighted L2 norm that accounts for the geometry of the mesh (since I am working with unstructured, anisotropic triangular meshes). 3 - Most importantly , I would like to modify the inner product used in the algorithm so that it aligns with the weighted L2 norm (since I am working with unstructured, anisotropic triangular meshes). Best regards, Ali ALI AHMAD De: "Barry Smith" < [ mailto:bsmith at petsc.dev | bsmith at petsc.dev ] > ?: "Ali ALI AHMAD" < [ mailto:ali.ali_ahmad at utt.fr | ali.ali_ahmad at utt.fr ] > Cc: "petsc-users" < [ mailto:petsc-users at mcs.anl.gov | petsc-users at mcs.anl.gov ] >, "petsc-maint" < [ mailto:petsc-maint at mcs.anl.gov | petsc-maint at mcs.anl.gov ] > Envoy?: Vendredi 13 Juin 2025 03:14:06 Objet: Re: [petsc-maint] norm L2 problemQuestion about changing the norm used in nonlinear solvers (L2 Euclidean vs. L2 Lebesgue) You haven't answered my question. Where (conceptually) and for what purpose do you want to use the L2 norm. 1) displaying norms to observe the convergence behavior 2) in the convergence testing to determine when to stop 3) changing the "inner product" in the algorithm which amounts to preconditioning. Barry BQ_BEGIN On Jun 12, 2025, at 9:42 AM, Ali ALI AHMAD < [ mailto:ali.ali_ahmad at utt.fr | ali.ali_ahmad at utt.fr ] > wrote: Thank you for your answer. I am currently working with the nonlinear solvers newtonls (with bt , l2 , etc.) and newtontr (using newton , cauchy , and dogleg strategies) combined with the linear solver gmres and the ILU preconditioner, since my Jacobian matrix is nonsymmetric. I also use the Eisenstat-Walker method for newtonls , as my initial guess is often very far from the exact solution. What I would like to do now is to replace the standard Euclidean L2 norm with the L2 norm in the Lebesgue sense in the above numerical algorithm , because my problem is defined on an unstructured, anisotropic triangular mesh where a weighted norm would be more physically appropriate. Would you be able to advise me on how to implement this change properly? I would deeply appreciate any guidance or suggestions you could provide. Thank you in advance for your help. Best regards, Ali ALI AHMAD De: "Ali ALI AHMAD" < [ mailto:ali.ali_ahmad at utt.fr | ali.ali_ahmad at utt.fr ] > ?: "Barry Smith" < [ mailto:bsmith at petsc.dev | bsmith at petsc.dev ] > Cc: "petsc-users" < [ mailto:petsc-users at mcs.anl.gov | petsc-users at mcs.anl.gov ] >, "petsc-maint" < [ mailto:petsc-maint at mcs.anl.gov | petsc-maint at mcs.anl.gov ] > Envoy?: Jeudi 12 Juin 2025 15:28:02 Objet: Re: [petsc-maint] norm L2 problemQuestion about changing the norm used in nonlinear solvers (L2 Euclidean vs. L2 Lebesgue) Thank you for your answer. I am currently working with the nonlinear solvers newtonls (with bt , l2 , etc.) and newtontr (using newton , cauchy , and dogleg strategies) combined with the linear solver gmres and the ILU preconditioner, since my Jacobian matrix is nonsymmetric. I also use the Eisenstat-Walker method for newtonls , as my initial guess is often very far from the exact solution. What I would like to do now is to replace the standard Euclidean L2 norm with the L2 norm in the Lebesgue sense , because my problem is defined on an unstructured, anisotropic triangular mesh where a weighted norm would be more physically appropriate. Would you be able to advise me on how to implement this change properly? I would deeply appreciate any guidance or suggestions you could provide. Thank you in advance for your help. Best regards, Ali ALI AHMAD De: "Barry Smith" < [ mailto:bsmith at petsc.dev | bsmith at petsc.dev ] > ?: "Ali ALI AHMAD" < [ mailto:ali.ali_ahmad at utt.fr | ali.ali_ahmad at utt.fr ] > Cc: "petsc-users" < [ mailto:petsc-users at mcs.anl.gov | petsc-users at mcs.anl.gov ] >, "petsc-maint" < [ mailto:petsc-maint at mcs.anl.gov | petsc-maint at mcs.anl.gov ] > Envoy?: Jeudi 12 Juin 2025 14:57:40 Objet: Re: [petsc-maint] norm L2 problemQuestion about changing the norm used in nonlinear solvers (L2 Euclidean vs. L2 Lebesgue) Do you wish to use a different norm 1) ONLY for displaying (printing out) the residual norms to track progress 2) in the convergence testing 3) to change the numerical algorithm (for example using the L2 inner product instead of the usual linear algebra R^N l2 inner product). For 1) use SNESMonitorSet() and in your monitor function use SNESGetSolution() to grab the solution and then VecGetArray(). Now you can compute any weighted norm you want on the solution. For 2) similar but you need to use SNESSetConvergenceTest For 3) yes, but you need to ask us specifically. Barry BQ_BEGIN On Jun 11, 2025, at 4:45 AM, Ali ALI AHMAD < [ mailto:ali.ali_ahmad at utt.fr | ali.ali_ahmad at utt.fr ] > wrote: Dear PETSc team, I hope this message finds you well. I am currently using PETSc in a C++, where I rely on the nonlinear solvers `SNES` with either `newtonls` or `newtontr` methods. I would like to ask if it is possible to change the default norm used (typically the L2 Euclidean norm) to a custom norm, specifically the L2 norm in the sense of Lebesgue (e.g., involving cell-wise weighted integrals over the domain). My main goal is to define a custom residual norm that better reflects the physical quantities of interest in my simulation. Would this be feasible within the PETSc framework? If so, could you point me to the recommended approach (e.g., redefining the norm manually, using specific PETSc hooks or options)? Thank you very much in advance for your help and for the great work on PETSc! Best regards, Ali ALI AHMAD PhD Student University of Technology of Troyes - UTT - France GAMMA3 Project - Office H008 - Phone No: +33 7 67 44 68 18 12 rue Marie Curie - CS 42060 10004 TROYES Cedex BQ_END BQ_END BQ_END BQ_END BQ_END -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener [ https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fiC1n5vGsbqOyw1uxsDvcoyysDPSuMVX-X60t2a22q6RxvgToFh_JTCqhACBSTB33XlfjUocJoQh741gmrYyKTuWi3OAYA$ | https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fiC1n5vGsbqOyw1uxsDvcoyysDPSuMVX-X60t2a22q6RxvgToFh_JTCqhACBSTB33XlfjUocJoQh741gmrYyKTvgry72Ng$ ] -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Aug 13 14:52:09 2025 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 13 Aug 2025 15:52:09 -0400 Subject: [petsc-users] [petsc-maint] norm L2 problemQuestion about changing the norm used in nonlinear solvers (L2 Euclidean vs. L2 Lebesgue) In-Reply-To: <1185563625.11757748.1755114333036.JavaMail.zimbra@utt.fr> References: <414475981.6714047.1749631527145.JavaMail.zimbra@utt.fr> <323745907.8383516.1749804945465.JavaMail.zimbra@utt.fr> <85CD2CA9-7B77-4288-87BA-9E108D40C7E8@petsc.dev> <82133477.13270009.1750410624370.JavaMail.zimbra@utt.fr> <86F87C6A-DEB3-4125-AF51-2B2E577EBFDD@petsc.dev> <36326459.7563988.1753714811037.JavaMail.zimbra@utt.fr> <1185563625.11757748.1755114333036.JavaMail.zimbra@utt.fr> Message-ID: Ah, then you can just use https://urldefense.us/v3/__https://petsc.org/main/manualpages/SNES/SNESSetConvergenceTest/__;!!G_uCfscf7eWS!b5Rm6Uca7nRuIRvz5loe-2If8OgOq1pxZCIhkk6IOO8PsiK7P7ooXpEK053J03WuANNw-ZftUDLu1NeNMT8G$ and compute your weighted norm yourself. Thanks, Matt On Wed, Aug 13, 2025 at 3:45?PM Ali ALI AHMAD wrote: > * As I mentioned earlier, the weighted norm I am referring to is: > > ||f||^2_2 = \int_{\Omega} |f(x)|^2 \, dx \approx \sum_{K \in \Omega} > \sum_q |f(G_q)|^2 \, area(K) > > where G_q is the centroid of triangle K, computed using a first-order > Gaussian quadrature rule. > > > Yes, I am using the inexact Newton method, where the method stops when > > > ||F(x_k) + J_{x_k} \, d_k|| \leq \nu_k \, ||F(x_k)||. > > > Here, I would also like to change the norm to see whether this > modification affects the quality of the descent direction. I know that > these norms are theoretically equivalent, but I would like to test them to > observe any difference between the two. My initial expectation is that the > method may converge faster when using the L^2 norm. > > > Best regards, > > Ali ALI AHMAD > ------------------------------ > *De: *"Matthew Knepley" > *?: *"Ali ALI AHMAD" > *Cc: *"Barry Smith" , "petsc-users" < > petsc-users at mcs.anl.gov>, "petsc-maint" > *Envoy?: *Lundi 28 Juillet 2025 18:55:04 > *Objet: *Re: [petsc-maint] norm L2 problemQuestion about changing the > norm used in nonlinear solvers (L2 Euclidean vs. L2 Lebesgue) > > On Mon, Jul 28, 2025 at 11:00?AM Ali ALI AHMAD > wrote: > >> I?m sorry for getting back to you so late. Thank you for your patience >> and understanding. >> >> - >> >> For example, when using L2 algorithms where a different norm is >> applied in the line search, see *Line_search_L2.png* as an example >> from this reference: https://urldefense.us/v3/__https://arxiv.org/abs/1607.04254__;!!G_uCfscf7eWS!b5Rm6Uca7nRuIRvz5loe-2If8OgOq1pxZCIhkk6IOO8PsiK7P7ooXpEK053J03WuANNw-ZftUDLu1CcZFu6I$ >> . >> Here, we can change the norm from the L2 Euclidean to the L2 Lebesgue norm. >> >> What does "L2 Lebesgue" mean here? I know what I mean by L_2. It is > > ||f||^2_2 = \int_Omega |f(x)|^2 dx \approx \sum_q |f(x_q)|^2 w_q > > Oh, from below it seems you want a weight function wt(x) in the norm. So > we would have > > \sum_q |f(x_q)|^2 wt(x_q) w_q > >> >> - For GMRES, we need to replace NORM_2 (L2 Euclidean) in your code >> with weighted_NormL2 (L2 Lebesgue) everywhere, including all the >> details such as in the Arnoldi algorithm... >> >> I do not quite understand here, because GMRES does not use the L:_2 norm. > It uses the l_2 norm, which is > > ||v||^2_2 = \sum |v_i|^2 > > The things in the vector are coefficients of basis functions. I guess, if > you had an interpolatory element, you could interpret this as quadrature > rule, meaning you would have > > \sum wt(x_i) |v_i|^2 > > where x_i were the coordinates of the dual basis evaluation functionals. > >> >> - >> >> For the convergence test as well, in the linear system for finding >> the direction *d* (A???d = b), >> >> You want to use the inner product that generates your weighted L_2 norm? > >> >> - >> >> and also when we search for a good step using the line search >> formula: xk+1=xk+step?d. >> >> You want to minimize your weighted L_2 norm. That should work in the same > way. > > Thanks, > > Matt > > >> I hope this explanation is clear for you. >> >> Best regards, >> Ali ALI AHMAD >> >> ------------------------------ >> *De: *"Barry Smith" >> *?: *"Ali ALI AHMAD" >> *Cc: *"petsc-users" , "petsc-maint" < >> petsc-maint at mcs.anl.gov> >> *Envoy?: *Vendredi 20 Juin 2025 15:50:17 >> *Objet: *Re: [petsc-maint] norm L2 problemQuestion about changing the >> norm used in nonlinear solvers (L2 Euclidean vs. L2 Lebesgue) >> >> >> >> On Jun 20, 2025, at 5:10?AM, Ali ALI AHMAD wrote: >> >> >> * Yes, I am indeed using an inexact Newton method in my code. The descent >> direction is computed by solving a linear system involving the Jacobian, so >> the update follows the classical formula "J(un)^{-1}d(un)=-F(un)" I'm also >> trying to use a line search strategy based on a weighted L2 norm (in the >> Lebesgue sense), which a priori should lead to better accuracy and faster >> convergence in anisotropic settings. >> >> >> Ok, could you point to sample code (any language) or written >> algorithms where a different norm is used in the line search? >> >> >> * During the subsequent iterations, I apply the Eisenstat?Walker method >> to adapt the tolerance, which should also involve modifying the norm used >> in the algorithm. >> >> * The current implementation still uses the standard Euclidean L2 norm in >> PETSc's linear solver and in GMRES. I believe this should ideally be >> replaced by a weighted L2 norm consistent with the discretization. However, >> I haven't yet succeeded in modifying the norm used internally by the linear >> solver in PETSc, so, I'm not yet sure how much impact this change would >> have on the overall convergence, but I suspect it could improve robustness, >> especially for highly anisotropic problems. I would greatly appreciate any >> guidance on how to implement this properly in PETSc. >> >> >> Norms are used in multiple ways in GMRES. >> >> 1) defining convergence >> >> 2) as part of preconditioning >> >> Again can you point to sample code (any language) or written algorithms >> that describe exactly what you would like to accomplish. >> >> Barry >> >> >> Do not hesitate to contact me again if anything remains unclear or if you >> need further information. >> >> Best regards, >> Ali ALI AHMAD >> >> ------------------------------ >> *De: *"Barry Smith" >> *?: *"Ali ALI AHMAD" >> *Cc: *"petsc-users" , "petsc-maint" < >> petsc-maint at mcs.anl.gov> >> *Envoy?: *Samedi 14 Juin 2025 01:06:52 >> *Objet: *Re: [petsc-maint] norm L2 problemQuestion about changing the >> norm used in nonlinear solvers (L2 Euclidean vs. L2 Lebesgue) >> >> I appreciate the clarification. I would call 3) preconditioning. >> To increase my understanding, you are already using Newton's method? >> That is, you compute the Jacobian of the function and use - J^{-1}(u^n) >> F(u^n) as your update direction? >> >> When you switch the inner product (or precondition) how will the >> search direction be different? >> >> Thanks >> >> Barry >> >> The case you need support for is becoming important to PETSc so we need >> to understand it well and support it well which is why I am asking these >> (perhaps to you) trivial questions. >> >> >> >> On Jun 13, 2025, at 4:55?AM, Ali ALI AHMAD wrote: >> >> Thank you for your message. >> >> To answer your question: I would like to use the L2 norm in the sense of >> Lebesgue for *all three purposes*, especially the *third one*. >> >> *1- For displaying residuals* during the nonlinear iterations, I would >> like to observe the convergence behavior using a norm that better reflects >> the physical properties of the problem. >> >> *2- For convergence testing*, I would like the stopping criterion to be >> based on a weighted L2 norm that accounts for the geometry of the mesh >> (since I am working with unstructured, anisotropic triangular meshes). >> >> *3 - Most importantly*, I would like to modify the *inner product* used >> in the algorithm so that it aligns with the weighted L2 norm (since I am >> working with unstructured, anisotropic triangular meshes). >> >> Best regards, >> Ali ALI AHMAD >> ------------------------------ >> *De: *"Barry Smith" >> *?: *"Ali ALI AHMAD" >> *Cc: *"petsc-users" , "petsc-maint" < >> petsc-maint at mcs.anl.gov> >> *Envoy?: *Vendredi 13 Juin 2025 03:14:06 >> *Objet: *Re: [petsc-maint] norm L2 problemQuestion about changing the >> norm used in nonlinear solvers (L2 Euclidean vs. L2 Lebesgue) >> >> You haven't answered my question. Where (conceptually) and for what >> purpose do you want to use the L2 norm. >> 1) displaying norms to observe the convergence behavior >> >> 2) in the convergence testing to determine when to stop >> >> 3) changing the "inner product" in the algorithm which amounts to >> preconditioning. >> >> Barry >> >> >> On Jun 12, 2025, at 9:42?AM, Ali ALI AHMAD wrote: >> >> Thank you for your answer. >> >> I am currently working with the nonlinear solvers *newtonls* (with bt, l2, >> etc.) and *newtontr* (using newton, cauchy, and dogleg strategies) >> combined with the linear solver *gmres* and the *ILU* preconditioner, >> since my Jacobian matrix is nonsymmetric. >> >> I also use the Eisenstat-Walker method for newtonls, as my initial guess >> is often very far from the exact solution. >> >> What I would like to do now is to *replace the standard Euclidean L2 >> norm* with the *L2 norm in the Lebesgue sense in the above numerical >> algorithm*, because my problem is defined on an *unstructured, >> anisotropic triangular mesh* where a weighted norm would be more >> physically appropriate. >> >> Would you be able to advise me on how to implement this change properly? >> >> I would deeply appreciate any guidance or suggestions you could provide. >> >> Thank you in advance for your help. >> >> Best regards, >> Ali ALI AHMAD >> >> ------------------------------ >> *De: *"Ali ALI AHMAD" >> *?: *"Barry Smith" >> *Cc: *"petsc-users" , "petsc-maint" < >> petsc-maint at mcs.anl.gov> >> *Envoy?: *Jeudi 12 Juin 2025 15:28:02 >> *Objet: *Re: [petsc-maint] norm L2 problemQuestion about changing the >> norm used in nonlinear solvers (L2 Euclidean vs. L2 Lebesgue) >> >> Thank you for your answer. >> >> I am currently working with the nonlinear solvers *newtonls* (with bt, l2, >> etc.) and *newtontr* (using newton, cauchy, and dogleg strategies) >> combined with the linear solver *gmres* and the *ILU* preconditioner, >> since my Jacobian matrix is nonsymmetric. >> >> I also use the Eisenstat-Walker method for newtonls, as my initial guess >> is often very far from the exact solution. >> >> What I would like to do now is to *replace the standard Euclidean L2 >> norm* with the *L2 norm in the Lebesgue sense*, because my problem is >> defined on an *unstructured, anisotropic triangular mesh* where a >> weighted norm would be more physically appropriate. >> >> Would you be able to advise me on how to implement this change properly? >> >> I would deeply appreciate any guidance or suggestions you could provide. >> >> Thank you in advance for your help. >> >> Best regards, >> Ali ALI AHMAD >> ------------------------------ >> *De: *"Barry Smith" >> *?: *"Ali ALI AHMAD" >> *Cc: *"petsc-users" , "petsc-maint" < >> petsc-maint at mcs.anl.gov> >> *Envoy?: *Jeudi 12 Juin 2025 14:57:40 >> *Objet: *Re: [petsc-maint] norm L2 problemQuestion about changing the >> norm used in nonlinear solvers (L2 Euclidean vs. L2 Lebesgue) >> >> Do you wish to use a different norm >> >> 1) ONLY for displaying (printing out) the residual norms to track >> progress >> >> 2) in the convergence testing >> >> 3) to change the numerical algorithm (for example using the L2 inner >> product instead of the usual linear algebra R^N l2 inner product). >> >> For 1) use SNESMonitorSet() and in your monitor function use >> SNESGetSolution() to grab the solution and then VecGetArray(). Now you can >> compute any weighted norm you want on the solution. >> >> For 2) similar but you need to use SNESSetConvergenceTest >> >> For 3) yes, but you need to ask us specifically. >> >> Barry >> >> >> On Jun 11, 2025, at 4:45?AM, Ali ALI AHMAD wrote: >> >> Dear PETSc team, >> >> I hope this message finds you well. >> >> I am currently using PETSc in a C++, where I rely on the nonlinear >> solvers `SNES` with either `newtonls` or `newtontr` methods. I would like >> to ask if it is possible to change the default norm used (typically the L2 >> Euclidean norm) to a custom norm, specifically the L2 norm in the sense of >> Lebesgue (e.g., involving cell-wise weighted integrals over the domain). >> >> My main goal is to define a custom residual norm that better reflects the >> physical quantities of interest in my simulation. >> >> Would this be feasible within the PETSc framework? If so, could you point >> me to the recommended approach (e.g., redefining the norm manually, using >> specific PETSc hooks or options)? >> >> Thank you very much in advance for your help and for the great work on >> PETSc! >> >> Best regards, >> >> ------------------------------ >> Ali ALI AHMAD >> PhD Student >> University of Technology of Troyes - UTT - France >> GAMMA3 Project - Office H008 - Phone No: +33 7 67 44 68 18 >> 12 rue Marie Curie - CS 42060 10004 TROYES Cedex >> >> >> >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!b5Rm6Uca7nRuIRvz5loe-2If8OgOq1pxZCIhkk6IOO8PsiK7P7ooXpEK053J03WuANNw-ZftUDLu1NXIqWUQ$ > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!b5Rm6Uca7nRuIRvz5loe-2If8OgOq1pxZCIhkk6IOO8PsiK7P7ooXpEK053J03WuANNw-ZftUDLu1NXIqWUQ$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From em459 at bath.ac.uk Mon Aug 18 06:55:46 2025 From: em459 at bath.ac.uk (Eike Mueller) Date: Mon, 18 Aug 2025 11:55:46 +0000 Subject: [petsc-users] petsc4py installation via conda-forge Message-ID: Dear PETSc developers, I need to install petsc4py on a specific machine for teaching, and one of my colleagues pointed out that this can be done with conda-forge using this: https://urldefense.us/v3/__https://anaconda.org/conda-forge/petsc4py__;!!G_uCfscf7eWS!aObnO6iDXElFvWx8fXEjFR-baM6bUK1tm-qPqLb5kFKPoo_GYUC260Pm5WkJmanUStqmZ1HhWtmNudJeMLyxcg$ Are you aware of any problems with this approach? It?s not listed as an option under the petsc4py installation instructions https://urldefense.us/v3/__https://petsc.org/release/petsc4py/install.html__;!!G_uCfscf7eWS!aObnO6iDXElFvWx8fXEjFR-baM6bUK1tm-qPqLb5kFKPoo_GYUC260Pm5WkJmanUStqmZ1HhWtmNudKT9XFvlA$ . I had tried these first, but the pip install of petsc/petsc4py didn?t work. The install via anaconda/conda-forge did give me a working petsc4py, however. My colleague says that he also successfully used it on google collab. Thank you very much, Eike -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay.anl at fastmail.org Mon Aug 18 07:07:50 2025 From: balay.anl at fastmail.org (Satish Balay) Date: Mon, 18 Aug 2025 07:07:50 -0500 (CDT) Subject: [petsc-users] petsc4py installation via conda-forge In-Reply-To: References: Message-ID: <85ae0848-5786-bab1-74ca-16cf77189ab9@fastmail.org> On Mon, 18 Aug 2025, Eike Mueller via petsc-users wrote: > Dear PETSc developers, > > I need to install petsc4py on a specific machine for teaching, and one of my colleagues pointed out that this can be done with conda-forge using this: https://urldefense.us/v3/__https://anaconda.org/conda-forge/petsc4py__;!!G_uCfscf7eWS!aObnO6iDXElFvWx8fXEjFR-baM6bUK1tm-qPqLb5kFKPoo_GYUC260Pm5WkJmanUStqmZ1HhWtmNudJeMLyxcg$ > > Are you aware of any problems with this approach? It?s not listed as an option under the petsc4py installation instructions https://urldefense.us/v3/__https://petsc.org/release/petsc4py/install.html__;!!G_uCfscf7eWS!aObnO6iDXElFvWx8fXEjFR-baM6bUK1tm-qPqLb5kFKPoo_GYUC260Pm5WkJmanUStqmZ1HhWtmNudKT9XFvlA$ . Hm we have conda (conda-forge) listed for PETSc at https://urldefense.us/v3/__https://petsc.org/release/install/__;!!G_uCfscf7eWS!YT8MIxLRgM1M4EgDQyr_BM0VkjMdsGa8sKUiEY6MeTB0XmurcA8ihfNry-uSv2zR1bQR7n-ceTf9ue71hwpmgSTl8a4$ > I had tried these first, but the pip install of petsc/petsc4py didn?t work. It usually requires PETSc installed correctly. Some of us developers just install PETSc+PETSc4py via: ./configure [PETSc configure options] --with-petsc4py=1 > The install via anaconda/conda-forge did give me a working petsc4py, however. My colleague says that he also successfully used it on google collab. Good to know. Thanks! Satish > > Thank you very much, > > Eike > From stefano.zampini at gmail.com Mon Aug 18 08:20:34 2025 From: stefano.zampini at gmail.com (Stefano Zampini) Date: Mon, 18 Aug 2025 16:20:34 +0300 Subject: [petsc-users] petsc4py installation via conda-forge In-Reply-To: References: Message-ID: Il giorno lun 18 ago 2025 alle ore 15:00 Eike Mueller via petsc-users < petsc-users at mcs.anl.gov> ha scritto: > Dear PETSc developers, > > I need to install petsc4py on a specific machine for teaching, and one of > my colleagues pointed out that this can be done with conda-forge using > this: https://urldefense.us/v3/__https://anaconda.org/conda-forge/petsc4py__;!!G_uCfscf7eWS!fQAEr8wAGM70H_zFHV7LthaM1y__6NUa2lRLnml_eARS9RWvdY52k7tV2UIogJ_G7Dgjj5mDZ_8hGQW8lVgxbV-j_0ZzvNE$ > > > > Are you aware of any problems with this approach? > Based on personal experience, you may encounter issues sometimes using the conda compilers and trying to compile PETSc applications. But if you plan to just use petsc4py, there won't be any problems. > It?s not listed as an option under the petsc4py installation instructions > https://urldefense.us/v3/__https://petsc.org/release/petsc4py/install.html__;!!G_uCfscf7eWS!fQAEr8wAGM70H_zFHV7LthaM1y__6NUa2lRLnml_eARS9RWvdY52k7tV2UIogJ_G7Dgjj5mDZ_8hGQW8lVgxbV-jbKVQpCc$ > . > I had tried these first, but the pip install of petsc/petsc4py didn?t work. > That is supposed to install petsc and petsc4py automatically and should work. Can you attach some failing logs? > The install via anaconda/conda-forge did give me a working petsc4py, > however. My colleague says that he also successfully used it on google > collab. > > Thank you very much, > > Eike > -- Stefano -------------- next part -------------- An HTML attachment was scrubbed... URL: From em459 at bath.ac.uk Mon Aug 18 12:19:38 2025 From: em459 at bath.ac.uk (Eike Mueller) Date: Mon, 18 Aug 2025 17:19:38 +0000 Subject: [petsc-users] petsc4py installation via conda-forge In-Reply-To: References: Message-ID: <23C4C3D4-726A-4332-9945-453B0C7CF416@bath.ac.uk> Hi Satish and Stefano, Thank you for your prompt replies! Sorry, I hadn?t seen that there is conda option for petsc, only looked at petsc4py. pip install petsc failed due to missing BLAS libraries. It?s not so much a question of eventually installing it as finding a way that is simple and which students can follow, so knowing that there is no issue with petsc and petsc4py on conda-forge is already very helpful. All coding will be done in Python and I?ll only use petsc4py, so won?t have to compile any PETSc C code. All the best, Eike On 18 Aug 2025, at 14:20, Stefano Zampini wrote: You don't often get email from stefano.zampini at gmail.com. Learn why this is important CAUTION: This email came from outside of the University. To keep your account safe, only click on links and open attachments if you know the person who sent the email, or you expected to receive this communication. Il giorno lun 18 ago 2025 alle ore 15:00 Eike Mueller via petsc-users > ha scritto: Dear PETSc developers, I need to install petsc4py on a specific machine for teaching, and one of my colleagues pointed out that this can be done with conda-forge using this: https://urldefense.us/v3/__https://anaconda.org/conda-forge/petsc4py__;!!G_uCfscf7eWS!ez8Nn_BYj2UcIPdVXqCPD-fd0eutgiyC7n6oplyX7USA18S1JCYY0ZwDAHZCLyjHmGMJXZE_khQkT0qUw-4qhw$ Are you aware of any problems with this approach? Based on personal experience, you may encounter issues sometimes using the conda compilers and trying to compile PETSc applications. But if you plan to just use petsc4py, there won't be any problems. It?s not listed as an option under the petsc4py installation instructions https://urldefense.us/v3/__https://petsc.org/release/petsc4py/install.html__;!!G_uCfscf7eWS!ez8Nn_BYj2UcIPdVXqCPD-fd0eutgiyC7n6oplyX7USA18S1JCYY0ZwDAHZCLyjHmGMJXZE_khQkT0qrFrbzSA$ . I had tried these first, but the pip install of petsc/petsc4py didn?t work. That is supposed to install petsc and petsc4py automatically and should work. Can you attach some failing logs? The install via anaconda/conda-forge did give me a working petsc4py, however. My colleague says that he also successfully used it on google collab. Thank you very much, Eike -- Stefano -------------- next part -------------- An HTML attachment was scrubbed... URL: From jasper.hatton at mail.utoronto.ca Thu Aug 21 14:42:20 2025 From: jasper.hatton at mail.utoronto.ca (Jasper Hatton) Date: Thu, 21 Aug 2025 19:42:20 +0000 Subject: [petsc-users] Suggestions for single process setup code Message-ID: Hi, I am looking for advice on using PETSc in a situation where I don't want the initial setup part of my application to run on multiple processes. The solution stage is taking up most of the time, so I would like to avoid making the setup stage a fully parallel code for now. My solution stage takes a shell matrix, which includes an FFT as well as multiple sparse mat-vec products. So I was planning to use the rank zero process to setup the distributed FFT and put values into the distributed matrices, then execute KSPSolve on all ranks. However, I see there is also the MPI linear solver server option. It seems mostly useful for cases where the input to KSPSolve is a sparse/dense matrix rather than a shell code. Is it something I should consider for my case? This is meant to be an first step which can at least run well on a 2-socket node with many cores available, and eventually could be made a fully distributed code in the future. Any advice would be apreciated! Thanks, Jasper -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Thu Aug 21 16:31:26 2025 From: bsmith at petsc.dev (Barry Smith) Date: Thu, 21 Aug 2025 17:31:26 -0400 Subject: [petsc-users] Suggestions for single process setup code In-Reply-To: References: Message-ID: <0BD5FCA3-5E0E-4C0A-8F77-4D6B00FE8451@petsc.dev> As you note, the MPI linear solver server is for matrices whose entries are explicitly provided (so xAIJ sparse and dense) it doesn't have code that would help with shell matrices. You should write a parallel code, but have rank 0 do most of the work initially. You can have rank 0, for example, do the MatSetValues() calls. Any specific questions, feel free to ask. Barry > On Aug 21, 2025, at 3:42?PM, Jasper Hatton via petsc-users wrote: > > Hi, > > I am looking for advice on using PETSc in a situation where I don't want the initial setup part of my application to run on multiple processes. The solution stage is taking up most of the time, so I would like to avoid making the setup stage a fully parallel code for now. > > My solution stage takes a shell matrix, which includes an FFT as well as multiple sparse mat-vec products. So I was planning to use the rank zero process to setup the distributed FFT and put values into the distributed matrices, then execute KSPSolve on all ranks. > > However, I see there is also the MPI linear solver server option. It seems mostly useful for cases where the input to KSPSolve is a sparse/dense matrix rather than a shell code. Is it something I should consider for my case? > > This is meant to be an first step which can at least run well on a 2-socket node with many cores available, and eventually could be made a fully distributed code in the future. > > Any advice would be apreciated! > > Thanks, > > Jasper -------------- next part -------------- An HTML attachment was scrubbed... URL: From jasper.hatton at mail.utoronto.ca Thu Aug 21 17:22:38 2025 From: jasper.hatton at mail.utoronto.ca (Jasper Hatton) Date: Thu, 21 Aug 2025 22:22:38 +0000 Subject: [petsc-users] Suggestions for single process setup code In-Reply-To: <0BD5FCA3-5E0E-4C0A-8F77-4D6B00FE8451@petsc.dev> References: <0BD5FCA3-5E0E-4C0A-8F77-4D6B00FE8451@petsc.dev> Message-ID: Hi Barry, Thanks! That answers my question. Cheers, Jasper ________________________________ From: Barry Smith Sent: Thursday, August 21, 2025 5:31 PM To: Jasper Hatton Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Suggestions for single process setup code You don't often get email from bsmith at petsc.dev. Learn why this is important As you note, the MPI linear solver server is for matrices whose entries are explicitly provided (so xAIJ sparse and dense) it doesn't have code that would help with shell matrices. You should write a parallel code, but have rank 0 do most of the work initially. You can have rank 0, for example, do the MatSetValues() calls. Any specific questions, feel free to ask. Barry On Aug 21, 2025, at 3:42?PM, Jasper Hatton via petsc-users wrote: Hi, I am looking for advice on using PETSc in a situation where I don't want the initial setup part of my application to run on multiple processes. The solution stage is taking up most of the time, so I would like to avoid making the setup stage a fully parallel code for now. My solution stage takes a shell matrix, which includes an FFT as well as multiple sparse mat-vec products. So I was planning to use the rank zero process to setup the distributed FFT and put values into the distributed matrices, then execute KSPSolve on all ranks. However, I see there is also the MPI linear solver server option. It seems mostly useful for cases where the input to KSPSolve is a sparse/dense matrix rather than a shell code. Is it something I should consider for my case? This is meant to be an first step which can at least run well on a 2-socket node with many cores available, and eventually could be made a fully distributed code in the future. Any advice would be apreciated! Thanks, Jasper -------------- next part -------------- An HTML attachment was scrubbed... URL: From lzou at anl.gov Fri Aug 22 16:25:34 2025 From: lzou at anl.gov (Zou, Ling) Date: Fri, 22 Aug 2025 21:25:34 +0000 Subject: [petsc-users] Understanding 'snes' object Message-ID: Hi all, Can someone please help me understand the output of the following two outputs using ?-snes_view?? Option 1: SNES Object: 1 MPI process type: newtonls maximum iterations=15, maximum function evaluations=10000 tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 total number of linear solver iterations=5 total number of function evaluations=14 norm schedule ALWAYS Jacobian is built using finite differences with coloring Option 2: SNES Object: 1 MPI process type: newtonls maximum iterations=50, maximum function evaluations=10000 tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 total number of linear solver iterations=8 total number of function evaluations=17 norm schedule ALWAYS Jacobian is applied matrix-free with differencing Preconditioning Jacobian is built using finite differences with coloring Does it mean that * Option 1 uses Newton?s method, such that J du = b is solved with J being explicitly constructed? * Option 2 uses matrix free method, such that J du = b is solved with J v ~ (F(u + dv) ? F(u)) / h when needed, e.g., in a Krylov method. Thanks, -Ling -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Fri Aug 22 16:31:14 2025 From: jed at jedbrown.org (Jed Brown) Date: Fri, 22 Aug 2025 15:31:14 -0600 Subject: [petsc-users] Understanding 'snes' object In-Reply-To: References: Message-ID: <87a53rm4bh.fsf@jedbrown.org> "Zou, Ling via petsc-users" writes: > Hi all, > > Can someone please help me understand the output of the following two outputs using ?-snes_view?? > > Option 1: > > SNES Object: 1 MPI process > > type: newtonls > > maximum iterations=15, maximum function evaluations=10000 > > tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 > > total number of linear solver iterations=5 > > total number of function evaluations=14 > > norm schedule ALWAYS > > Jacobian is built using finite differences with coloring > > Option 2: > > SNES Object: 1 MPI process > > type: newtonls > > maximum iterations=50, maximum function evaluations=10000 > > tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 > > total number of linear solver iterations=8 > > total number of function evaluations=17 > > norm schedule ALWAYS > > Jacobian is applied matrix-free with differencing > > Preconditioning Jacobian is built using finite differences with coloring > > > Does it mean that > > * Option 1 uses Newton?s method, such that J du = b is solved with J being explicitly constructed? > * Option 2 uses matrix free method, such that J du = b is solved with J v ~ (F(u + dv) ? F(u)) / h when needed, e.g., in a Krylov method. Roughly, yes. J is assembled (using FD with coloring in both cases). That sparse matrix representation of J is used for preconditioning in both cases. (Your output is truncated, but if you scroll down it'll show the Krylov solver (KSP) configuration and the preconditioner.) When the Krylov solver needs to just apply the operator, it's as you say (Option 1 uses matrix-free differencing while Option 2 uses the assembled matrix J). From stefano.zampini at gmail.com Fri Aug 22 16:41:49 2025 From: stefano.zampini at gmail.com (Stefano Zampini) Date: Sat, 23 Aug 2025 00:41:49 +0300 Subject: [petsc-users] Understanding 'snes' object In-Reply-To: References: Message-ID: Il giorno sab 23 ago 2025 alle ore 00:25 Zou, Ling via petsc-users < petsc-users at mcs.anl.gov> ha scritto: > Hi all, > > > > Can someone please help me understand the output of the following two > outputs using ?-snes_view?? > > > > Option 1: > > SNES Object: 1 MPI process > > type: newtonls > > maximum iterations=15, maximum function evaluations=10000 > > tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 > > total number of linear solver iterations=5 > > total number of function evaluations=14 > > norm schedule ALWAYS > > Jacobian is built using finite differences with coloring > > > > Option 2: > > SNES Object: 1 MPI process > > type: newtonls > > maximum iterations=50, maximum function evaluations=10000 > > tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 > > total number of linear solver iterations=8 > > total number of function evaluations=17 > > norm schedule ALWAYS > > Jacobian is applied matrix-free with differencing > > Preconditioning Jacobian is built using finite differences with coloring > > > > > > Does it mean that > > - Option 1 uses Newton?s method, such that J du = b is solved with J > being explicitly constructed? > > yes > > - > - Option 2 uses matrix free method, such that J du = b is solved with > J v ~ (F(u + dv) ? F(u)) / h when needed, e.g., in a Krylov method. > > yes, and the preconditioner is constructed using an explicitly constructed Jacobian matrix > > - > > > > Thanks, > > -Ling > > > > > -- Stefano -------------- next part -------------- An HTML attachment was scrubbed... URL: From lzou at anl.gov Sat Aug 23 09:03:31 2025 From: lzou at anl.gov (Zou, Ling) Date: Sat, 23 Aug 2025 14:03:31 +0000 Subject: [petsc-users] Understanding 'snes' object In-Reply-To: References: Message-ID: Thank you both Jed and Stefano. Then, somehow, I may have introduced inconsistence in my code, because I expected they have exactly the same behavior. Best, -Ling From: Stefano Zampini Date: Friday, August 22, 2025 at 4:42?PM To: Zou, Ling Cc: PETSc Subject: Re: [petsc-users] Understanding 'snes' object Il giorno sab 23 ago 2025 alle ore 00:?25 Zou, Ling via petsc-users ha scritto: Hi all, Can someone please help me understand the output of the following two outputs using ?-snes_view?? Option 1: SNES Object: 1 ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Il giorno sab 23 ago 2025 alle ore 00:25 Zou, Ling via petsc-users > ha scritto: Hi all, Can someone please help me understand the output of the following two outputs using ?-snes_view?? Option 1: SNES Object: 1 MPI process type: newtonls maximum iterations=15, maximum function evaluations=10000 tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 total number of linear solver iterations=5 total number of function evaluations=14 norm schedule ALWAYS Jacobian is built using finite differences with coloring Option 2: SNES Object: 1 MPI process type: newtonls maximum iterations=50, maximum function evaluations=10000 tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 total number of linear solver iterations=8 total number of function evaluations=17 norm schedule ALWAYS Jacobian is applied matrix-free with differencing Preconditioning Jacobian is built using finite differences with coloring Does it mean that * Option 1 uses Newton?s method, such that J du = b is solved with J being explicitly constructed? yes * * Option 2 uses matrix free method, such that J du = b is solved with J v ~ (F(u + dv) ? F(u)) / h when needed, e.g., in a Krylov method. yes, and the preconditioner is constructed using an explicitly constructed Jacobian matrix * Thanks, -Ling -- Stefano -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Sun Aug 24 15:30:25 2025 From: bsmith at petsc.dev (Barry Smith) Date: Sun, 24 Aug 2025 16:30:25 -0400 Subject: [petsc-users] Understanding 'snes' object In-Reply-To: References: Message-ID: <3D5FD262-12FC-4022-89A5-DF86DEE5C691@petsc.dev> You should have very similar behavior, but not identical behavior. The Jacobian-vector product will produce slightly different values with the two approaches, which means that the computed residual norms displayed with -ksp_monitor -snes_monitor etc will be somewhat different. With long runs that involve many nonlinear solves the differences in the norms computed will become nontrivial and you will start getting different iteration counts, but the solutions computed should remain very similar. Barry > On Aug 23, 2025, at 10:03?AM, Zou, Ling via petsc-users wrote: > > Thank you both Jed and Stefano. > Then, somehow, I may have introduced inconsistence in my code, because I expected they have exactly the same behavior. > > Best, > -Ling > > From: Stefano Zampini > > Date: Friday, August 22, 2025 at 4:42?PM > To: Zou, Ling > > Cc: PETSc > > Subject: Re: [petsc-users] Understanding 'snes' object > > Il giorno sab 23 ago 2025 alle ore 00:?25 Zou, Ling via petsc-users ha scritto: Hi all, Can someone please help me understand the output of the following two outputs using ?-snes_view?? Option 1: SNES Object: 1 > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > > > Il giorno sab 23 ago 2025 alle ore 00:25 Zou, Ling via petsc-users > ha scritto: > Hi all, > > Can someone please help me understand the output of the following two outputs using ?-snes_view?? > > Option 1: > SNES Object: 1 MPI process > > type: newtonls > > maximum iterations=15, maximum function evaluations=10000 > > tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 > > total number of linear solver iterations=5 > > total number of function evaluations=14 > > norm schedule ALWAYS > > Jacobian is built using finite differences with coloring > > > Option 2: > SNES Object: 1 MPI process > > type: newtonls > > maximum iterations=50, maximum function evaluations=10000 > > tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 > > total number of linear solver iterations=8 > > total number of function evaluations=17 > > norm schedule ALWAYS > > Jacobian is applied matrix-free with differencing > > Preconditioning Jacobian is built using finite differences with coloring > > > > Does it mean that > Option 1 uses Newton?s method, such that J du = b is solved with J being explicitly constructed? > yes > > > Option 2 uses matrix free method, such that J du = b is solved with J v ~ (F(u + dv) ? F(u)) / h when needed, e.g., in a Krylov method. > > yes, and the preconditioner is constructed using an explicitly constructed Jacobian matrix > > > > Thanks, > -Ling > > > > > -- > Stefano -------------- next part -------------- An HTML attachment was scrubbed... URL: From ali.ali_ahmad at utt.fr Sun Aug 24 19:16:00 2025 From: ali.ali_ahmad at utt.fr (Ali ALI AHMAD) Date: Mon, 25 Aug 2025 02:16:00 +0200 (CEST) Subject: [petsc-users] Question about NewtonTR / Inexact Newton with piecewise continuous functions Message-ID: <958788052.14717071.1756080960168.JavaMail.zimbra@utt.fr> Hello, ? I am currently working with PETSc to solve a nonlinear system. My function is piecewise continuous (so it can be discontinuous), and its derivative (Jacobian) is also piecewise continuous. ? I implemented the residual and also compute the Jacobian analytically. ? When the discontinuities are small, both Inexact Newton and NewtonTR (with scaled Newton direction) converge without problem. ? However, for test cases with larger discontinuities, sometimes the solver converges, but in other cases it fails to converge. ? I initially tried , but I cannot use them in my case, because my problem involves nonlinear constraints. ? So my question is: how does PETSc handle such situations internally (piecewise continuous objective/residual functions)? And is there a recommended strategy within PETSc to deal with nonlinear solvers when and are discontinuous? ? I would like to continue working with PETSc and I am looking for a robust method to treat this type of problem. ? Thank you very much for your help and suggestions. ? Best regards, ALI ALI AHMAD -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Sun Aug 24 19:34:38 2025 From: bsmith at petsc.dev (Barry Smith) Date: Sun, 24 Aug 2025 20:34:38 -0400 Subject: [petsc-users] Question about NewtonTR / Inexact Newton with piecewise continuous functions In-Reply-To: <958788052.14717071.1756080960168.JavaMail.zimbra@utt.fr> References: <958788052.14717071.1756080960168.JavaMail.zimbra@utt.fr> Message-ID: <52C25099-2869-4341-815A-469FF59482ED@petsc.dev> "my problem involves nonlinear constraints". Do you mean that you have nonlinear inequality constraints on certain variables (SNESVI only handles bound inequality constraints) so cannot use SNESVI? How do you handle such constraints within SNES then? Is handling them what introduces the discontinuous nature of your SNES? Barry Please do not cross list to both petsc-maint and petsc-users. The questions will be answered the same way if submitted to either list. > On Aug 24, 2025, at 8:16?PM, Ali ALI AHMAD wrote: > > Hello, > > I am currently working with PETSc to solve a nonlinear system. My function is piecewise continuous (so it can be discontinuous), and its derivative (Jacobian) is also piecewise continuous. > > I implemented the residual and also compute the Jacobian analytically. > > When the discontinuities are small, both Inexact Newton and NewtonTR (with scaled Newton direction) converge without problem. > > However, for test cases with larger discontinuities, sometimes the solver converges, but in other cases it fails to converge. > > I initially tried , but I cannot use them in my case, because my problem involves nonlinear constraints. > > So my question is: how does PETSc handle such situations internally (piecewise continuous objective/residual functions)? > And is there a recommended strategy within PETSc to deal with nonlinear solvers when and are discontinuous? > > I would like to continue working with PETSc and I am looking for a robust method to treat this type of problem. > > Thank you very much for your help and suggestions. > > Best regards, > ALI ALI AHMAD -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sun Aug 24 20:42:36 2025 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 24 Aug 2025 21:42:36 -0400 Subject: [petsc-users] [petsc-maint] Question about NewtonTR / Inexact Newton with piecewise continuous functions In-Reply-To: <958788052.14717071.1756080960168.JavaMail.zimbra@utt.fr> References: <958788052.14717071.1756080960168.JavaMail.zimbra@utt.fr> Message-ID: On Sun, Aug 24, 2025 at 8:16?PM Ali ALI AHMAD wrote: > Hello, > > I am currently working with PETSc to solve a nonlinear system. My function > is piecewise continuous (so it can be discontinuous), and its derivative > (Jacobian) is also piecewise continuous. > I am not aware of any convergence framework for piecewise continuous functions. They are not technically computable, meaning you could have to compute for a very very long time, and still not get an accurate output (since you have the jump). This also means that the derivative is not computable, and you can have arbitrarily large errors near the discontinuity. I don't see how you could prove convergence here, but maybe someone else knows something I don't. Thanks, Matt > I implemented the residual and also compute the Jacobian analytically. > > When the discontinuities are small, both Inexact Newton and NewtonTR (with > scaled Newton direction) converge without problem. > > However, for test cases with larger discontinuities, sometimes the solver > converges, but in other cases it fails to converge. > > I initially tried , but I cannot use them in > my case, because my problem involves nonlinear constraints. > > So my question is: how does PETSc handle such situations internally > (piecewise continuous objective/residual functions)? > And is there a recommended strategy within PETSc to deal with nonlinear > solvers when and are discontinuous? > > I would like to continue working with PETSc and I am looking for a robust > method to treat this type of problem. > > Thank you very much for your help and suggestions. > > Best regards, > ALI ALI AHMAD > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!Yg82hvg9sA6Ah-__bPQ_4r_664JidfkOdAY-N29Pk73B45u5wi_E81N7EdOpAQF40CXZXuaKdfFg1UCotgQZ$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexis.salzman at ec-nantes.fr Mon Aug 25 06:31:14 2025 From: alexis.salzman at ec-nantes.fr (Alexis SALZMAN) Date: Mon, 25 Aug 2025 13:31:14 +0200 Subject: [petsc-users] MatCreateSubMatricesMPI strange behavior In-Reply-To: References: <9d648a1d-72d2-44d0-8a3b-a9d64b01604f@ec-nantes.fr> Message-ID: Hi, Thanks for your answer, Mark. Perhaps MatCreateSubMatricesMPI is the only PETSc function that acts on a sub-communicator ? I'm not sure ? but it's clear that there's no ambiguity on that point. The first line of the documentation for that function states that it 'may live on subcomms'. This is confirmed by the 'src/mat/tests/ex183.c' test case. I used this test case to understand the function, which helped me with my code and the example I provided in my initial post. Unfortunately, in this example, the matrix from which the sub-matrices are extracted is dense, even though it uses a sparse structure. This does not clarify how to define sub-matrices when extracting from a sparse distributed matrix. Since my initial post, I have discovered that having more columns than rows can also result in the same error message. So, my questions boil down to: Can MatCreateSubMatricesMPI extract rectangular matrices from a square distributed sparse matrix? If not, the fact that only square matrices can be extracted in this context should perhaps be mentioned in the documentation. If so, I would be very grateful for any assistance in defining an IS pair in this context. Regards A.S. Le 27/07/2025 ? 00:15, Mark Adams a ?crit?: > First, you can not mix communicators in PETSc calls in general > (ever?), but this error looks like you?might be?asking for a row from > the matrix that?does not exist. > You should start with a PETSc example code. Test it and modify?it to > suit your needs. > > Good luck, > Mark > > On Fri, Jul 25, 2025 at 9:31?AM Alexis SALZMAN > wrote: > > Hi, > > As I am relatively new to Petsc, I may have misunderstood how to > use the > MatCreateSubMatricesMPI function. The attached code is tuned for > three > processes and extracts one matrix for each colour of a > subcommunicator > that has been created using the MPI_Comm_split function from an? > MPIAij > matrix. The following error message appears when the code is set > to its > default configuration (i.e. when a rectangular matrix is extracted > with > more rows than columns for colour 0): > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Argument out of range > [0]PETSC ERROR: Column too large: col 4 max 3 > [0]PETSC ERROR: See > https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!ZqH097BZ0G0O3WI7RWrwIKFNpyk0czSWEqfusAeTlgEygAffwpgBUzsLw1TIoGkjZ3mYG-NRQxxFoxU4y8EyY0ofiz9I43Qwe0w$ > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.22.2, unknown > > ... petsc git hash 2a89477b25f compiled on a dell i9 computer with > Gcc > 14.3, mkl 2025.2, ..... > [0]PETSC ERROR: #1 MatSetValues_SeqAIJ() at > ...petsc/src/mat/impls/aij/seq/aij.c:426 > [0]PETSC ERROR: #2 MatSetValues() at > ...petsc/src/mat/interface/matrix.c:1543 > [0]PETSC ERROR: #3 MatSetSeqMats_MPIAIJ() at > .../petsc/src/mat/impls/aij/mpi/mpiov.c:2965 > [0]PETSC ERROR: #4 MatCreateSubMatricesMPI_MPIXAIJ() at > .../petsc/src/mat/impls/aij/mpi/mpiov.c:3163 > [0]PETSC ERROR: #5 MatCreateSubMatricesMPI_MPIAIJ() at > .../petsc/src/mat/impls/aij/mpi/mpiov.c:3196 > [0]PETSC ERROR: #6 MatCreateSubMatricesMPI() at > .../petsc/src/mat/interface/matrix.c:7293 > [0]PETSC ERROR: #7 main() at sub.c:169 > > When the '-ok' option is selected, the code extracts a square > matrix for > colour 0, which runs smoothly in this case. Selecting the '-trans' > option swaps the row and column selection indices, providing a > transposed submatrix smoothly. For colour 1, which uses only one > process > and is therefore sequential, rectangular extraction is OK > regardless of > the shape. > > Is this dependency on the shape expected? Have I missed an important > tuning step somewhere? > > Thank you in advance for any clarification. > > Regards > > A.S. > > P.S.: I'm sorry, but as I'm leaving my office for the following weeks > this evening, I won't be very responsive during this period. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Mon Aug 25 10:55:23 2025 From: mfadams at lbl.gov (Mark Adams) Date: Mon, 25 Aug 2025 11:55:23 -0400 Subject: [petsc-users] MatCreateSubMatricesMPI strange behavior In-Reply-To: References: <9d648a1d-72d2-44d0-8a3b-a9d64b01604f@ec-nantes.fr> Message-ID: Ah, OK, never say never. MatCreateSubMatrices seems to support creating a new matrix with the communicator of the IS. It just needs to read from the input matrix and does not use it for communication, so it can do that. As far as rectangular matrices, there is no reason not to support that (the row IS and column IS can be distinct). Can you send the whole error message? There may not be a test that does this, but src/mat/tests/ex23.c looks like it may be a rectangular matrix output. And, it should not matter if the input matrix has a 100% full sparse matrix. It is still MatAIJ. The semantics and API is the same for sparse or dense matrices. Thanks, Mark On Mon, Aug 25, 2025 at 7:31?AM Alexis SALZMAN wrote: > Hi, > > Thanks for your answer, Mark. Perhaps MatCreateSubMatricesMPI is the only > PETSc function that acts on a sub-communicator ? I'm not sure ? but it's > clear that there's no ambiguity on that point. The first line of the > documentation for that function states that it 'may live on subcomms'. This > is confirmed by the 'src/mat/tests/ex183.c' test case. I used this test > case to understand the function, which helped me with my code and the > example I provided in my initial post. Unfortunately, in this example, the > matrix from which the sub-matrices are extracted is dense, even though it > uses a sparse structure. This does not clarify how to define sub-matrices > when extracting from a sparse distributed matrix. Since my initial post, I > have discovered that having more columns than rows can also result in the > same error message. > > So, my questions boil down to: > > Can MatCreateSubMatricesMPI extract rectangular matrices from a square > distributed sparse matrix? > > If not, the fact that only square matrices can be extracted in this > context should perhaps be mentioned in the documentation. > > If so, I would be very grateful for any assistance in defining an IS pair > in this context. > > Regards > > A.S. > Le 27/07/2025 ? 00:15, Mark Adams a ?crit : > > First, you can not mix communicators in PETSc calls in general (ever?), > but this error looks like you might be asking for a row from the matrix > that does not exist. > You should start with a PETSc example code. Test it and modify it to suit > your needs. > > Good luck, > Mark > > On Fri, Jul 25, 2025 at 9:31?AM Alexis SALZMAN < > alexis.salzman at ec-nantes.fr> wrote: > >> Hi, >> >> As I am relatively new to Petsc, I may have misunderstood how to use the >> MatCreateSubMatricesMPI function. The attached code is tuned for three >> processes and extracts one matrix for each colour of a subcommunicator >> that has been created using the MPI_Comm_split function from an MPIAij >> matrix. The following error message appears when the code is set to its >> default configuration (i.e. when a rectangular matrix is extracted with >> more rows than columns for colour 0): >> >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: Argument out of range >> [0]PETSC ERROR: Column too large: col 4 max 3 >> [0]PETSC ERROR: See >> https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!ZqH097BZ0G0O3WI7RWrwIKFNpyk0czSWEqfusAeTlgEygAffwpgBUzsLw1TIoGkjZ3mYG-NRQxxFoxU4y8EyY0ofiz9I43Qwe0w$ >> for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.22.2, unknown >> >> ... petsc git hash 2a89477b25f compiled on a dell i9 computer with Gcc >> 14.3, mkl 2025.2, ..... >> [0]PETSC ERROR: #1 MatSetValues_SeqAIJ() at >> ...petsc/src/mat/impls/aij/seq/aij.c:426 >> [0]PETSC ERROR: #2 MatSetValues() at >> ...petsc/src/mat/interface/matrix.c:1543 >> [0]PETSC ERROR: #3 MatSetSeqMats_MPIAIJ() at >> .../petsc/src/mat/impls/aij/mpi/mpiov.c:2965 >> [0]PETSC ERROR: #4 MatCreateSubMatricesMPI_MPIXAIJ() at >> .../petsc/src/mat/impls/aij/mpi/mpiov.c:3163 >> [0]PETSC ERROR: #5 MatCreateSubMatricesMPI_MPIAIJ() at >> .../petsc/src/mat/impls/aij/mpi/mpiov.c:3196 >> [0]PETSC ERROR: #6 MatCreateSubMatricesMPI() at >> .../petsc/src/mat/interface/matrix.c:7293 >> [0]PETSC ERROR: #7 main() at sub.c:169 >> >> When the '-ok' option is selected, the code extracts a square matrix for >> colour 0, which runs smoothly in this case. Selecting the '-trans' >> option swaps the row and column selection indices, providing a >> transposed submatrix smoothly. For colour 1, which uses only one process >> and is therefore sequential, rectangular extraction is OK regardless of >> the shape. >> >> Is this dependency on the shape expected? Have I missed an important >> tuning step somewhere? >> >> Thank you in advance for any clarification. >> >> Regards >> >> A.S. >> >> P.S.: I'm sorry, but as I'm leaving my office for the following weeks >> this evening, I won't be very responsive during this period. >> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexis.salzman at ec-nantes.fr Mon Aug 25 13:00:54 2025 From: alexis.salzman at ec-nantes.fr (Alexis SALZMAN) Date: Mon, 25 Aug 2025 20:00:54 +0200 Subject: [petsc-users] MatCreateSubMatricesMPI strange behavior In-Reply-To: References: <9d648a1d-72d2-44d0-8a3b-a9d64b01604f@ec-nantes.fr> Message-ID: <113ece7d-25cf-4b16-b29b-bdd1ad2a004a@ec-nantes.fr> Thanks Mark for your attention. The uncleaned error message, compared to my post in July, is as follows: [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Argument out of range [0]PETSC ERROR: Column too large: col 4 max 3 [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!dFZ4KSyqnoKD_8HJEOBBrLiK5TUCQmZbw09Dxau1D-3pxswHNP1D3HpEP-nXcrUdppQnRXo6rVLtt26Bd50bK6i7-_w38dW0qu4$ for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.22.2, unknown [0]PETSC ERROR: subnb with 3 MPI process(es) and PETSC_ARCH ?on pc-str97.ec-nantes.fr by salzman Mon Aug 25 19:11:37 2025 [0]PETSC ERROR: Configure options: PETSC_ARCH=real_fc41_Release_gcc_i4 PETSC_DIR=/home/salzman/devel/ExternalLib/build/PETSC/petsc --doCleanup=1 --with-scalar-type=real --known-level1-dcach e-linesize=64 --with-cc=gcc --CFLAGS="-fPIC " --CC_LINKER_FLAGS=-fopenmp --with-cxx=g++ --with-cxx-dialect=c++20 --CXXFLAGS="-fPIC " --CXX_LINKER_FLAGS=-fopenmp --with-fc=gfortran --FFLAGS= "-fPIC " --FC_LINKER_FLAGS=-fopenmp --with-debugging=0 --with-fortran-bindings=0 --with-fortran-kernels=1 --with-mpi-compilers=0 --with-mpi-include=/usr/include/openmpi-x86_64 --with-mpi-li b="[/usr/lib64/openmpi/lib/libmpi.so,/usr/lib64/openmpi/lib/libmpi.so,/usr/lib64/openmpi/lib/libmpi_mpifh.so]" --with-blas-lib="[/opt/intel/oneapi/mkl/latest/lib/libmkl_intel_lp64.so,/opt/i ntel/oneapi/mkl/latest/lib/libmkl_gnu_thread.so,/opt/intel/oneapi/mkl/latest/lib/libmkl_core.so]" --with-lapack-lib="[/opt/intel/oneapi/mkl/latest/lib/libmkl_intel_lp64.so,/opt/intel/oneapi /mkl/latest/lib/libmkl_gnu_thread.so,/opt/intel/oneapi/mkl/latest/lib/libmkl_core.so]" --with-mumps=1 --with-mumps-include=/home/salzman/local/i4_gcc/include --with-mumps-lib="[/home/salzma n/local/i4_gcc/lib/libdmumps.so,/home/salzman/local/i4_gcc/lib/libmumps_common.so,/home/salzman/local/i4_gcc/lib/libpord.so]" --with-scalapack-lib="[/opt/intel/oneapi/mkl/latest/lib/libmkl_ scalapack_lp64.so,/opt/intel/oneapi/mkl/latest/lib/libmkl_blacs_openmpi_lp64.so]" --with-mkl_pardiso=1 --with-mkl_pardiso-include=/opt/intel/oneapi/mkl/latest/include --with-mkl_pardiso-lib ="[/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_intel_lp64.so]" --with-hdf5=1 --with-hdf5-include=/usr/include/openmpi-x86_64 --with-hdf5-lib="[/usr/lib64/openmpi/lib/libhdf5.so]" --with -pastix=0 --download-pastix=no --with-hwloc=1 --with-hwloc-dir=/home/salzman/local/i4_gcc --download-hwloc=no --with-ptscotch-include=/home/salzman/local/i4_gcc/include --with-ptscotch-lib= "[/home/salzman/local/i4_gcc/lib/libptscotch.a,/home/salzman/local/i4_gcc/lib/libptscotcherr.a,/home/salzman/local/i4_gcc/lib/libptscotcherrexit.a,/home/salzman/local/i4_gcc/lib/libscotch.a ,/home/salzman/local/i4_gcc/lib/libscotcherr.a,/home/salzman/local/i4_gcc/lib/libscotcherrexit.a]" --with-hypre=1 --download-hypre=yes --with-suitesparse=1 --with-suitesparse-include=/home/ salzman/local/i4_gcc/include --with-suitesparse-lib="[/home/salzman/local/i4_gcc/lib/libsuitesparseconfig.so,/home/salzman/local/i4_gcc/lib/libumfpack.so,/home/salzman/local/i4_gcc/lib/libk lu.so,/home/salzman/local/i4_gcc/lib/libcholmod.so,/home/salzman/local/i4_gcc/lib/libspqr.so,/home/salzman/local/i4_gcc/lib/libcolamd.so,/home/salzman/local/i4_gcc/lib/libccolamd.so,/home/s alzman/local/i4_gcc/lib/libcamd.so,/home/salzman/local/i4_gcc/lib/libamd.so,/home/salzman/local/i4_gcc/lib/libmetis.so]" --download-suitesparse=no --with-python-exec=python3.12 --have-numpy =1 ---with-petsc4py=1 ---with-petsc4py-test-np=4 ---with-mpi4py=1 --prefix=/home/salzman/local/i4_gcc/real_arithmetic COPTFLAGS="-O3 -g " CXXOPTFLAGS="-O3 -g " FOPTFLAGS="-O3 -g " [0]PETSC ERROR: #1 MatSetValues_SeqAIJ() at /home/salzman/devel/PETSc/petsc/src/mat/impls/aij/seq/aij.c:426 [0]PETSC ERROR: #2 MatSetValues() at /home/salzman/devel/PETSc/petsc/src/mat/interface/matrix.c:1543 [0]PETSC ERROR: #3 MatSetSeqMats_MPIAIJ() at /home/salzman/devel/PETSc/petsc/src/mat/impls/aij/mpi/mpiov.c:2965 [0]PETSC ERROR: #4 MatCreateSubMatricesMPI_MPIXAIJ() at /home/salzman/devel/PETSc/petsc/src/mat/impls/aij/mpi/mpiov.c:3163 [0]PETSC ERROR: #5 MatCreateSubMatricesMPI_MPIAIJ() at /home/salzman/devel/PETSc/petsc/src/mat/impls/aij/mpi/mpiov.c:3196 [0]PETSC ERROR: #6 MatCreateSubMatricesMPI() at /home/salzman/devel/PETSc/petsc/src/mat/interface/matrix.c:7293 [0]PETSC ERROR: #7 main() at subnb.c:181 [0]PETSC ERROR: No PETSc Option Table entries [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- -------------------------------------------------------------------------- This message comes from executing the attached test (I simplified the test by removing the block size from the matrix used for extraction, compared to the July test). In proc_xx_output.txt, you will find the output from the code execution with the -ok option (i.e. irow/idxr and icol/idxc are the same, i.e. a square sub-block for colour 0 distributed across the first two processes). Has expected in this case we obtain the 0,3,6,9 sub-block terms, which are distributed across processes 0 and 1 (two rows per proc). When asking for rectangular sub-block (i.e. with no option) it crash with column to large on process 0: 4 col max 3 ??? I ask for 4 rows and 2 columns in this process ??? Otherwise, I mention the dense aspect of the matrix in ex183.c, because, in this case, no matter what selection is requested, all terms are non-null. If there is an issue with the way the selection is coded in the user program, I think it will be masked thanks to the full graph representation. However, this may not be the case ? I should test it. I'll take a look at ex23.c. Thanks, A.S. Le 25/08/2025 ? 17:55, Mark Adams a ?crit?: > Ah, OK, never say never. > > MatCreateSubMatrices seems to support creating a new matrix with the > communicator of the IS. > It just needs to read from the input matrix and does not use it for > communication, so it can do that. > > As far as rectangular matrices, there is no reason not to support that > (the row IS and column IS can be distinct). > Can you send the whole error message? > There may not be a test that does this, but?src/mat/tests/ex23.c looks > like it may be a rectangular?matrix output. > > And, it should not matter if the input matrix has a 100% full sparse > matrix. It is still MatAIJ. > The semantics and API is the same for sparse or dense matrices. > > Thanks, > Mark > > On Mon, Aug 25, 2025 at 7:31?AM Alexis SALZMAN > wrote: > > Hi, > > Thanks for your answer, Mark. Perhaps MatCreateSubMatricesMPI is > the only PETSc function that acts on a sub-communicator ? I'm not > sure ? but it's clear that there's no ambiguity on that point. The > first line of the documentation for that function states that it > 'may live on subcomms'. This is confirmed by the > 'src/mat/tests/ex183.c' test case. I used this test case to > understand the function, which helped me with my code and the > example I provided in my initial post. Unfortunately, in this > example, the matrix from which the sub-matrices are extracted is > dense, even though it uses a sparse structure. This does not > clarify how to define sub-matrices when extracting from a sparse > distributed matrix. Since my initial post, I have discovered that > having more columns than rows can also result in the same error > message. > > So, my questions boil down to: > > Can MatCreateSubMatricesMPI extract rectangular matrices from a > square distributed sparse matrix? > > If not, the fact that only square matrices can be extracted in > this context should perhaps be mentioned in the documentation. > > If so, I would be very grateful for any assistance in defining an > IS pair in this context. > > Regards > > A.S. > > Le 27/07/2025 ? 00:15, Mark Adams a ?crit?: >> First, you can not mix communicators in PETSc calls in general >> (ever?), but this error looks like you?might be?asking for a row >> from the matrix that?does not exist. >> You should start with a PETSc example code. Test it and modify?it >> to suit your needs. >> >> Good luck, >> Mark >> >> On Fri, Jul 25, 2025 at 9:31?AM Alexis SALZMAN >> wrote: >> >> Hi, >> >> As I am relatively new to Petsc, I may have misunderstood how >> to use the >> MatCreateSubMatricesMPI function. The attached code is tuned >> for three >> processes and extracts one matrix for each colour of a >> subcommunicator >> that has been created using the MPI_Comm_split function from >> an? MPIAij >> matrix. The following error message appears when the code is >> set to its >> default configuration (i.e. when a rectangular matrix is >> extracted with >> more rows than columns for colour 0): >> >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: Argument out of range >> [0]PETSC ERROR: Column too large: col 4 max 3 >> [0]PETSC ERROR: See >> https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!ZqH097BZ0G0O3WI7RWrwIKFNpyk0czSWEqfusAeTlgEygAffwpgBUzsLw1TIoGkjZ3mYG-NRQxxFoxU4y8EyY0ofiz9I43Qwe0w$ >> for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.22.2, unknown >> >> ... petsc git hash 2a89477b25f compiled on a dell i9 computer >> with Gcc >> 14.3, mkl 2025.2, ..... >> [0]PETSC ERROR: #1 MatSetValues_SeqAIJ() at >> ...petsc/src/mat/impls/aij/seq/aij.c:426 >> [0]PETSC ERROR: #2 MatSetValues() at >> ...petsc/src/mat/interface/matrix.c:1543 >> [0]PETSC ERROR: #3 MatSetSeqMats_MPIAIJ() at >> .../petsc/src/mat/impls/aij/mpi/mpiov.c:2965 >> [0]PETSC ERROR: #4 MatCreateSubMatricesMPI_MPIXAIJ() at >> .../petsc/src/mat/impls/aij/mpi/mpiov.c:3163 >> [0]PETSC ERROR: #5 MatCreateSubMatricesMPI_MPIAIJ() at >> .../petsc/src/mat/impls/aij/mpi/mpiov.c:3196 >> [0]PETSC ERROR: #6 MatCreateSubMatricesMPI() at >> .../petsc/src/mat/interface/matrix.c:7293 >> [0]PETSC ERROR: #7 main() at sub.c:169 >> >> When the '-ok' option is selected, the code extracts a square >> matrix for >> colour 0, which runs smoothly in this case. Selecting the >> '-trans' >> option swaps the row and column selection indices, providing a >> transposed submatrix smoothly. For colour 1, which uses only >> one process >> and is therefore sequential, rectangular extraction is OK >> regardless of >> the shape. >> >> Is this dependency on the shape expected? Have I missed an >> important >> tuning step somewhere? >> >> Thank you in advance for any clarification. >> >> Regards >> >> A.S. >> >> P.S.: I'm sorry, but as I'm leaving my office for the >> following weeks >> this evening, I won't be very responsive during this period. >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: subnb.c Type: text/x-csrc Size: 6093 bytes Desc: not available URL: -------------- next part -------------- rstart 0 rend 4 Mat Object: 3 MPI processes type: mpiaij row 0: (0, 101.) (3, 104.) (6, 107.) (9, 110.) row 1: (2, 203.) (5, 206.) (8, 209.) (11, 212.) row 2: (1, 302.) (4, 305.) (7, 308.) (10, 311.) row 3: (0, 401.) (3, 404.) (6, 407.) (9, 410.) row 4: (2, 503.) (5, 506.) (8, 509.) (11, 512.) row 5: (1, 602.) (4, 605.) (7, 608.) (10, 611.) row 6: (0, 701.) (3, 704.) (6, 707.) (9, 710.) row 7: (2, 803.) (5, 806.) (8, 809.) (11, 812.) row 8: (1, 902.) (4, 905.) (7, 908.) (10, 911.) row 9: (0, 1001.) (3, 1004.) (6, 1007.) (9, 1010.) row 10: (2, 1103.) (5, 1106.) (8, 1109.) (11, 1112.) row 11: (1, 1202.) (4, 1205.) (7, 1208.) (10, 1211.) idxr proc IS Object: 2 MPI processes type: general [0] Number of indices in set 2 [0] 0 0 [0] 1 3 [1] Number of indices in set 2 [1] 0 6 [1] 1 9 idxc proc IS Object: 2 MPI processes type: general [0] Number of indices in set 2 [0] 0 0 [0] 1 3 [1] Number of indices in set 2 [1] 0 6 [1] 1 9 Mat Object: 2 MPI processes type: mpiaij row 0: (0, 101.) (1, 104.) (2, 107.) (3, 110.) row 1: (0, 401.) (1, 404.) (2, 407.) (3, 410.) row 2: (0, 701.) (1, 704.) (2, 707.) (3, 710.) row 3: (0, 1001.) (1, 1004.) (2, 1007.) (3, 1010.) rstart 0 rend 2 local row 0: ( 0 , 1.010000e+02) ( 1 , 1.040000e+02) ( 2 , 1.070000e+02) ( 3 , 1.100000e+02) local row 1: ( 0 , 4.010000e+02) ( 1 , 4.040000e+02) ( 2 , 4.070000e+02) ( 3 , 4.100000e+02) -------------- next part -------------- rstart 4 rend 8 idxr proc idxc proc rstart 2 rend 4 local row 2: ( 0 , 7.010000e+02) ( 1 , 7.040000e+02) ( 2 , 7.070000e+02) ( 3 , 7.100000e+02) local row 3: ( 0 , 1.001000e+03) ( 1 , 1.004000e+03) ( 2 , 1.007000e+03) ( 3 , 1.010000e+03) -------------- next part -------------- rstart 8 rend 12 idxr proc IS Object: 1 MPI process type: general Number of indices in set 12 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 11 11 idxc proc IS Object: 1 MPI process type: general Number of indices in set 4 0 0 1 1 2 8 3 9 Mat Object: 1 MPI process type: seqaij row 0: (0, 101.) (3, 110.) row 1: (2, 209.) row 2: (1, 302.) row 3: (0, 401.) (3, 410.) row 4: (2, 509.) row 5: (1, 602.) row 6: (0, 701.) (3, 710.) row 7: (2, 809.) row 8: (1, 902.) row 9: (0, 1001.) (3, 1010.) row 10: (2, 1109.) row 11: (1, 1202.) rstart 0 rend 12 local row 0: ( 0 , 1.010000e+02) ( 3 , 1.100000e+02) local row 1: ( 2 , 2.090000e+02) local row 2: ( 1 , 3.020000e+02) local row 3: ( 0 , 4.010000e+02) ( 3 , 4.100000e+02) local row 4: ( 2 , 5.090000e+02) local row 5: ( 1 , 6.020000e+02) local row 6: ( 0 , 7.010000e+02) ( 3 , 7.100000e+02) local row 7: ( 2 , 8.090000e+02) local row 8: ( 1 , 9.020000e+02) local row 9: ( 0 , 1.001000e+03) ( 3 , 1.010000e+03) local row 10: ( 2 , 1.109000e+03) local row 11: ( 1 , 1.202000e+03) From ali.ali_ahmad at utt.fr Mon Aug 25 17:02:12 2025 From: ali.ali_ahmad at utt.fr (Ali ALI AHMAD) Date: Tue, 26 Aug 2025 00:02:12 +0200 (CEST) Subject: [petsc-users] [petsc-maint] Question about NewtonTR / Inexact Newton with piecewise continuous functions In-Reply-To: References: <958788052.14717071.1756080960168.JavaMail.zimbra@utt.fr> Message-ID: <683850940.15267050.1756159332237.JavaMail.zimbra@utt.fr> Oh no, I was mistaken: I meant that the objective function is continuous, but its derivative is not. That is why I am using snesvinewtonssls. * However, the problem is that the nonlinear constraints I define in " ComputeBounds " are always inactive. As you can see in the output: 0 SNES Function norm 1.562978975444e+07 0 SNES VI Function norm 1.56298e+07 Active lower constraints 0/0 upper constraints 0/0 Percent of total 0. Percent of bounded 0. 0 KSP Residual norm 1.976350248286e+04 1 KSP Residual norm 1.483291315197e+04 2 KSP Residual norm 1.358944289069e+04 3 KSP Residual norm 1.276791956295e+04 4 KSP Residual norm 1.275078123378e+04 5 KSP Residual norm 1.253064296351e+04 6 KSP Residual norm 1.232954788760e+04 7 KSP Residual norm 1.232684700773e+04 8 KSP Residual norm 1.165265510468e+04 9 KSP Residual norm 1.159313964889e+04 10 KSP Residual norm 1.098241228864e+04 11 KSP Residual norm 1.063610434231e+04 12 KSP Residual norm 8.556981786904e+03 13 KSP Residual norm 7.792752698989e+03 14 KSP Residual norm 6.260039598596e+03 15 KSP Residual norm 3.025534878795e+03 16 KSP Residual norm 8.343990389000e+02 17 KSP Residual norm 2.591011944960e+02 18 KSP Residual norm 9.212497850148e+01 19 KSP Residual norm 4.137422005070e+01 20 KSP Residual norm 1.345189115925e+01 21 KSP Residual norm 2.795423327875e+00 22 KSP Residual norm 5.068536677416e-01 23 KSP Residual norm 1.167594367615e-01 Linear solve converged due to CONVERGED_RTOL iterations 23 [Non physique] pression n?gative ou NaN ? la cellule 61 : p = -1.66778e+14 Line search: objective function at lambdas = 1. is Inf or Nan, cutting lambda [Non physique] pression n?gative ou NaN ? la cellule 61 : p = -4.06293e+13 Line search: objective function at lambdas = 0.5 is Inf or Nan, cutting lambda [Non physique] pression n?gative ou NaN ? la cellule 61 : p = -9.63498e+12 Line search: objective function at lambdas = 0.25 is Inf or Nan, cutting lambda [Non physique] pression n?gative ou NaN ? la cellule 61 : p = -2.15792e+12 Line search: objective function at lambdas = 0.125 is Inf or Nan, cutting lambda [Non physique] pression n?gative ou NaN ? la cellule 61 : p = -1568.11 Line search: objective function at lambdas = 0.0625 is Inf or Nan, cutting lambda [Non physique] pression n?gative ou NaN ? la cellule 788 : p = -11.2882 Line search: objective function at lambdas = 0.03125 is Inf or Nan, cutting lambda Line search: Using full step: fnorm 1.562978975444e+07 gnorm 1.519904837982e+07 So none of my lower or upper bounds are active during the iterations. * Here is my ComputeBounds function, where I try to enforce positivity of density and pressure: PetscErrorCode ComputeBounds(SNES snes, Vec xl, Vec xu) { PetscFunctionBeginUser; // 1) Get user context (must have been set via SNESSetApplicationContext) void *ctx = NULL; PetscCall(SNESGetApplicationContext(snes, &ctx)); euler *e = (euler*)ctx; // Minimal positive thresholds const double eps_rho = 1e-8; // minimal density const double pmin = 1e-8; // minimal pressure // 2) Access current solution X (used here to build *dynamic* bounds) Vec X; const PetscScalar *x = NULL; PetscCall(SNESGetSolution(snes, &X)); PetscCall(VecGetArrayRead(X, &x)); // 3) Fill lower bounds 'xl' PetscScalar *l = NULL; PetscCall(VecGetArray(xl, &l)); for (int K = 0; K < e->N_cells; ++K) { const int i_rho = Ncomp*K + 0; // density index const int i_m = Ncomp*K + 1; // momentum (1D). In 2D use mx,my and adjust kinetic term. const int i_rhoE = Ncomp*K + 2; // total energy index const double rho = (double)x[i_rho]; const double m = (double)x[i_m]; // Use a safe density to avoid division by ~0 in kinetic energy const double rho_safe = PetscMax(rho, eps_rho); // Kinetic energy per unit volume: |m|^2/(2*rho) const double kin_vol = 0.5 * (m*m) / rho_safe; // Enforce p >= pmin via total energy lower bound: // rhoE >= kinetic + p/(gamma-1) const double rhoEmin = kin_vol + pmin/(_gamma - 1.0); // Set lower bounds l[i_rho] = eps_rho; // rho >= eps_rho l[i_m] = PETSC_NINFINITY; // no lower bound on momentum l[i_rhoE] = rhoEmin; // rhoE >= rhoEmin (to keep p >= pmin) } // 4) Restore arrays PetscCall(VecRestoreArrayRead(X, &x)); PetscCall(VecRestoreArray(xl, &l)); // 5) Set upper bounds to +infinity (no upper constraints) PetscCall(VecSet(xu, PETSC_INFINITY)); PetscFunctionReturn(PETSC_SUCCESS); } But when I run the solver, PETSc always reports Active lower constraints 0/0 . Why are my constraints not activated? Best regards, ALI ALI AHMAD De: "Matthew Knepley" ?: "Ali ALI AHMAD" Cc: "petsc-maint" , "petsc-users" , "Barry Smith" Envoy?: Lundi 25 Ao?t 2025 03:42:36 Objet: Re: [petsc-maint] Question about NewtonTR / Inexact Newton with piecewise continuous functions On Sun, Aug 24, 2025 at 8:16 PM Ali ALI AHMAD < [ mailto:ali.ali_ahmad at utt.fr | ali.ali_ahmad at utt.fr ] > wrote: Hello, I am currently working with PETSc to solve a nonlinear system. My function is piecewise continuous (so it can be discontinuous), and its derivative (Jacobian) is also piecewise continuous. I am not aware of any convergence framework for piecewise continuous functions. They are not technically computable, meaning you could have to compute for a very very long time, and still not get an accurate output (since you have the jump). This also means that the derivative is not computable, and you can have arbitrarily large errors near the discontinuity. I don't see how you could prove convergence here, but maybe someone else knows something I don't. Thanks, Matt BQ_BEGIN I implemented the residual and also compute the Jacobian analytically. When the discontinuities are small, both Inexact Newton and NewtonTR (with scaled Newton direction) converge without problem. However, for test cases with larger discontinuities, sometimes the solver converges, but in other cases it fails to converge. I initially tried , but I cannot use them in my case, because my problem involves nonlinear constraints. So my question is: how does PETSc handle such situations internally (piecewise continuous objective/residual functions)? And is there a recommended strategy within PETSc to deal with nonlinear solvers when and are discontinuous? I would like to continue working with PETSc and I am looking for a robust method to treat this type of problem. Thank you very much for your help and suggestions. Best regards, ALI ALI AHMAD BQ_END -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener [ https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!c59r-_cWhRyXgrv6zPsyXOKKCkQh6vxCZENKwjHfU9j8W-SbCNE9hvNJQKo-F6mc6MAmkFUmr5Ysk_S5VvLlpZfDMhUtWg$ | https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!c59r-_cWhRyXgrv6zPsyXOKKCkQh6vxCZENKwjHfU9j8W-SbCNE9hvNJQKo-F6mc6MAmkFUmr5Ysk_S5VvLlpZeH41xL_Q$ ] -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Mon Aug 25 20:47:39 2025 From: bsmith at petsc.dev (Barry Smith) Date: Mon, 25 Aug 2025 21:47:39 -0400 Subject: [petsc-users] [petsc-maint] Question about NewtonTR / Inexact Newton with piecewise continuous functions In-Reply-To: <683850940.15267050.1756159332237.JavaMail.zimbra@utt.fr> References: <958788052.14717071.1756080960168.JavaMail.zimbra@utt.fr> <683850940.15267050.1756159332237.JavaMail.zimbra@utt.fr> Message-ID: You can put a debugger breakpoint in the SNESVI code that checks the bounds to see why it is not finding violations. This is done in SNESSolve_VINEWTONRSLS() which calls SNESVIGetActiveSetIS(). So check th variables in SNESVIGetActiveSetIS. Note in even simple debuggers like gdb or lldb you can do call VecView(x,0) call VecView(xl,0) etc to examine the vectors quickly and verify if the values make sense. Barry > On Aug 25, 2025, at 6:02?PM, Ali ALI AHMAD wrote: > > Oh no, I was mistaken: I meant that the objective function is continuous, but its derivative is not. That is why I am using snesvinewtonssls. > > * However, the problem is that the nonlinear constraints I define in " ComputeBounds " are always inactive. As you can see in the output: > > 0 SNES Function norm 1.562978975444e+07 > 0 SNES VI Function norm 1.56298e+07 Active lower constraints 0/0 upper constraints 0/0 Percent of total 0. Percent of bounded 0. > 0 KSP Residual norm 1.976350248286e+04 > 1 KSP Residual norm 1.483291315197e+04 > 2 KSP Residual norm 1.358944289069e+04 > 3 KSP Residual norm 1.276791956295e+04 > 4 KSP Residual norm 1.275078123378e+04 > 5 KSP Residual norm 1.253064296351e+04 > 6 KSP Residual norm 1.232954788760e+04 > 7 KSP Residual norm 1.232684700773e+04 > 8 KSP Residual norm 1.165265510468e+04 > 9 KSP Residual norm 1.159313964889e+04 > 10 KSP Residual norm 1.098241228864e+04 > 11 KSP Residual norm 1.063610434231e+04 > 12 KSP Residual norm 8.556981786904e+03 > 13 KSP Residual norm 7.792752698989e+03 > 14 KSP Residual norm 6.260039598596e+03 > 15 KSP Residual norm 3.025534878795e+03 > 16 KSP Residual norm 8.343990389000e+02 > 17 KSP Residual norm 2.591011944960e+02 > 18 KSP Residual norm 9.212497850148e+01 > 19 KSP Residual norm 4.137422005070e+01 > 20 KSP Residual norm 1.345189115925e+01 > 21 KSP Residual norm 2.795423327875e+00 > 22 KSP Residual norm 5.068536677416e-01 > 23 KSP Residual norm 1.167594367615e-01 > Linear solve converged due to CONVERGED_RTOL iterations 23 > [Non physique] pression n?gative ou NaN ? la cellule 61 : p = -1.66778e+14 > Line search: objective function at lambdas = 1. is Inf or Nan, cutting lambda > [Non physique] pression n?gative ou NaN ? la cellule 61 : p = -4.06293e+13 > Line search: objective function at lambdas = 0.5 is Inf or Nan, cutting lambda > [Non physique] pression n?gative ou NaN ? la cellule 61 : p = -9.63498e+12 > Line search: objective function at lambdas = 0.25 is Inf or Nan, cutting lambda > [Non physique] pression n?gative ou NaN ? la cellule 61 : p = -2.15792e+12 > Line search: objective function at lambdas = 0.125 is Inf or Nan, cutting lambda > [Non physique] pression n?gative ou NaN ? la cellule 61 : p = -1568.11 > Line search: objective function at lambdas = 0.0625 is Inf or Nan, cutting lambda > [Non physique] pression n?gative ou NaN ? la cellule 788 : p = -11.2882 > Line search: objective function at lambdas = 0.03125 is Inf or Nan, cutting lambda > Line search: Using full step: fnorm 1.562978975444e+07 gnorm 1.519904837982e+07 > > So none of my lower or upper bounds are active during the iterations. > > * Here is my ComputeBounds function, where I try to enforce positivity of density and pressure: > > PetscErrorCode ComputeBounds(SNES snes, Vec xl, Vec xu) { > PetscFunctionBeginUser; > > // 1) Get user context (must have been set via SNESSetApplicationContext) > void *ctx = NULL; > PetscCall(SNESGetApplicationContext(snes, &ctx)); > euler *e = (euler*)ctx; > > // Minimal positive thresholds > const double eps_rho = 1e-8; // minimal density > const double pmin = 1e-8; // minimal pressure > > // 2) Access current solution X (used here to build *dynamic* bounds) > Vec X; > const PetscScalar *x = NULL; > PetscCall(SNESGetSolution(snes, &X)); > PetscCall(VecGetArrayRead(X, &x)); > > // 3) Fill lower bounds 'xl' > PetscScalar *l = NULL; > PetscCall(VecGetArray(xl, &l)); > > for (int K = 0; K < e->N_cells; ++K) { > const int i_rho = Ncomp*K + 0; // density index > const int i_m = Ncomp*K + 1; // momentum (1D). In 2D use mx,my and adjust kinetic term. > const int i_rhoE = Ncomp*K + 2; // total energy index > > const double rho = (double)x[i_rho]; > const double m = (double)x[i_m]; > > // Use a safe density to avoid division by ~0 in kinetic energy > const double rho_safe = PetscMax(rho, eps_rho); > > // Kinetic energy per unit volume: |m|^2/(2*rho) > const double kin_vol = 0.5 * (m*m) / rho_safe; > > // Enforce p >= pmin via total energy lower bound: > // rhoE >= kinetic + p/(gamma-1) > const double rhoEmin = kin_vol + pmin/(_gamma - 1.0); > > // Set lower bounds > l[i_rho] = eps_rho; // rho >= eps_rho > l[i_m] = PETSC_NINFINITY; // no lower bound on momentum > l[i_rhoE] = rhoEmin; // rhoE >= rhoEmin (to keep p >= pmin) > } > > // 4) Restore arrays > PetscCall(VecRestoreArrayRead(X, &x)); > PetscCall(VecRestoreArray(xl, &l)); > > // 5) Set upper bounds to +infinity (no upper constraints) > PetscCall(VecSet(xu, PETSC_INFINITY)); > > PetscFunctionReturn(PETSC_SUCCESS); > } > > But when I run the solver, PETSc always reports Active lower constraints 0/0. Why are my constraints not activated? > > Best regards, > ALI ALI AHMAD > > De: "Matthew Knepley" > ?: "Ali ALI AHMAD" > Cc: "petsc-maint" , "petsc-users" , "Barry Smith" > Envoy?: Lundi 25 Ao?t 2025 03:42:36 > Objet: Re: [petsc-maint] Question about NewtonTR / Inexact Newton with piecewise continuous functions > > On Sun, Aug 24, 2025 at 8:16?PM Ali ALI AHMAD > wrote: >> Hello, >> >> I am currently working with PETSc to solve a nonlinear system. My function is piecewise continuous (so it can be discontinuous), and its derivative (Jacobian) is also piecewise continuous. > > I am not aware of any convergence framework for piecewise continuous functions. They are not technically computable, meaning you could have to compute for a very very long time, and still not get an accurate output (since you have the jump). This also means that the derivative is not computable, and you can have arbitrarily large errors near the discontinuity. I don't see how you could prove convergence here, but maybe someone else knows something I don't. > > Thanks, > > Matt > >> I implemented the residual and also compute the Jacobian analytically. >> >> When the discontinuities are small, both Inexact Newton and NewtonTR (with scaled Newton direction) converge without problem. >> >> However, for test cases with larger discontinuities, sometimes the solver converges, but in other cases it fails to converge. >> >> I initially tried , but I cannot use them in my case, because my problem involves nonlinear constraints. >> >> So my question is: how does PETSc handle such situations internally (piecewise continuous objective/residual functions)? >> And is there a recommended strategy within PETSc to deal with nonlinear solvers when and are discontinuous? >> >> I would like to continue working with PETSc and I am looking for a robust method to treat this type of problem. >> >> Thank you very much for your help and suggestions. >> >> Best regards, >> ALI ALI AHMAD > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!eVe1OReScpoGqh7jJoAWZ73himymGGPZO4ZjvamIdb0EmslI9gdViRqCNUS8J8Y1cimxSWK184MgrZJnsDic0eY$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From aldo.bonfiglioli at unibas.it Tue Aug 26 03:21:23 2025 From: aldo.bonfiglioli at unibas.it (Aldo Bonfiglioli) Date: Tue, 26 Aug 2025 10:21:23 +0200 Subject: [petsc-users] Question about Pseudo Time Stepping Message-ID: <85e1d3ad-6a32-444e-94dd-b5ebb6cba819@unibas.it> Hi there, I am using -ts_type pseudo to find steady solutions of the Euler/NS PDEs. It looks like FormRHSFunction is called twice with the exactly same value of the dependent variable (u) before invoking TSRHSJacobianFn and then solving the linear system with KSP (u is then updated) > 1 TS dt 0.141197 time 0.141197 > Calling FormRHSFunction; ||u|| = ???60.489405479528003 > ???????463.15635675129079 ???????62.813532026019146 > ???????5.9789502719351155 > Calling FormRHSFunction; ||u|| = ???60.489405479528003 > ???????463.1563567512907962.8135320260191465.9789502719351155 > Calling TSRHSJacobianFn with ||u|| = ???60.489405479528003 > ???????463.15635675129079 ???????62.813532026019146 > ???????5.9789502719351155 > ?????Linear solve converged due to CONVERGED_RTOL iterations 9 > Calling FormRHSFunction; ||u|| = > ???60.489405476878737463.1563567560218662.8135320316169655.9789502710519811 > 2 TS dt 0.155917 time 0.297115 > Calling FormRHSFunction; ||u|| = > ???60.489405476878737463.1563567560218662.8135320316169655.9789502710519811 > Calling FormRHSFunction; ||u|| = > ???60.489405476878737463.1563567560218662.8135320316169655.9789502710519811 > Calling TSRHSJacobianFn with ||u|| = > ???60.489405476878737463.1563567560218662.8135320316169655.9789502710519811 > Why is it so? Am I possibly misusing (or missing) some options? My .petscrc file is attached. Thank you, Aldo -- Dr. Aldo Bonfiglioli Associate professor of Fluid Mechanics Dipartimento di Ingegneria Universita' della Basilicata V.le dell'Ateneo Lucano, 10 85100 Potenza ITALY tel:+39.0971.205203 fax:+39.0971.205215 web:https://urldefense.us/v3/__http://docenti.unibas.it/site/home/docente.html?m=002423__;!!G_uCfscf7eWS!ci_aySHdF2Qx_B-37IOzELFDtcQRthJ2NItK-JNbfXB0M2_21B_z97WIfjxZUyKCVRRhd1mzL6b-rnCed_i8LLA56JvFdz7JBrw$ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- -dm_plex_dim 2 -dm_plex_filename /home/abonfi/grids/2D/aerofoils/naca0012/euler/gmsh/naca0012.msh #-dm_plex_filename /home/abonfi/grids/2D/aerofoils/naca0012/euler/gmsh/mesh.h5 -testcase 3 ##-dm_view ##-dm_plex_view_labels "marker" ##-dm_plex_view_labels "Face Sets" -freestream_Mach_number .5 -flow_angles 1.,0.,0. -dm_mat_type baij ##-Reynolds 1.e39 -colors 4,5,-1,-1,-1,-1 -dm_plex_simplex true -dm_plex_interpolate -dm_plex_check_all -petscpartitioner_view ##-dm_petscsection_view ##-ts_dt 100. -ts_max_steps 100 -ts_type pseudo -pc_factor_mat_ordering_type rcm -pc_factor_reuse_ordering -ksp_converged_reason -options_left ##-ksp_view From alexis.salzman at ec-nantes.fr Tue Aug 26 05:50:10 2025 From: alexis.salzman at ec-nantes.fr (Alexis SALZMAN) Date: Tue, 26 Aug 2025 12:50:10 +0200 Subject: [petsc-users] MatCreateSubMatricesMPI strange behavior In-Reply-To: <113ece7d-25cf-4b16-b29b-bdd1ad2a004a@ec-nantes.fr> References: <9d648a1d-72d2-44d0-8a3b-a9d64b01604f@ec-nantes.fr> <113ece7d-25cf-4b16-b29b-bdd1ad2a004a@ec-nantes.fr> Message-ID: <82d06db6-ab86-49b8-9e26-000745ae2134@ec-nantes.fr> Mark, you were right and I was wrong about the dense matrix. Adding explicit zeros to the distributed matrix used to extract the sub-matrices (making it dense) in my test does not change the behaviour: there is still an error. I am finding it increasingly difficult to understand the logic of the row and column 'IS' creation. I ran many tests to achieve the desired result: a rectangular sub-matrix (so a rectangular or square sub-matrix appears to be possible). However, many others resulted in the same kind of error. From what I observed, the test only works if the column selection contribution (size_c in the test) has a specific value related to the row selection contribution (size_r in the test) for proc 0 (rank for both communicator and sub-communicator): * if size_r==2 then if size_c<=2 it works. * if size_r>=3 and size_r<=5 then size_c==size_r is the only working case. This occurs "regardless" of what is requested in proc 1 and in selr/selc (It can't be a dummy setting, though). In any case, it's certainly not an exhaustive analysis. Many thanks to anyone who can explain to me the logic behind the construction of row and column 'IS'. Regards A.S. Le 25/08/2025 ? 20:00, Alexis SALZMAN a ?crit?: > > Thanks Mark for your attention. > > The uncleaned error message, compared to my post in July, is as follows: > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Argument out of range > [0]PETSC ERROR: Column too large: col 4 max 3 > [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!dWBkCu100EMuxu8ooVUnqSFN7OhzOBoNHAiwDYEQ5cJ921sU5hdFb-G24ounZFeUQgZkfWqGRX4iIHyQ-xLQElJst5RbKa2pGnk$ for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.22.2, unknown > [0]PETSC ERROR: subnb with 3 MPI process(es) and PETSC_ARCH ?on > pc-str97.ec-nantes.fr by salzman Mon Aug 25 19:11:37 2025 > [0]PETSC ERROR: Configure options: PETSC_ARCH=real_fc41_Release_gcc_i4 > PETSC_DIR=/home/salzman/devel/ExternalLib/build/PETSC/petsc > --doCleanup=1 --with-scalar-type=real --known-level1-dcach > e-linesize=64 --with-cc=gcc --CFLAGS="-fPIC " > --CC_LINKER_FLAGS=-fopenmp --with-cxx=g++ --with-cxx-dialect=c++20 > --CXXFLAGS="-fPIC " --CXX_LINKER_FLAGS=-fopenmp --with-fc=gfortran > --FFLAGS= > "-fPIC " --FC_LINKER_FLAGS=-fopenmp --with-debugging=0 > --with-fortran-bindings=0 --with-fortran-kernels=1 > --with-mpi-compilers=0 --with-mpi-include=/usr/include/openmpi-x86_64 > --with-mpi-li > b="[/usr/lib64/openmpi/lib/libmpi.so,/usr/lib64/openmpi/lib/libmpi.so,/usr/lib64/openmpi/lib/libmpi_mpifh.so]" > --with-blas-lib="[/opt/intel/oneapi/mkl/latest/lib/libmkl_intel_lp64.so,/opt/i > ntel/oneapi/mkl/latest/lib/libmkl_gnu_thread.so,/opt/intel/oneapi/mkl/latest/lib/libmkl_core.so]" > --with-lapack-lib="[/opt/intel/oneapi/mkl/latest/lib/libmkl_intel_lp64.so,/opt/intel/oneapi > /mkl/latest/lib/libmkl_gnu_thread.so,/opt/intel/oneapi/mkl/latest/lib/libmkl_core.so]" > --with-mumps=1 --with-mumps-include=/home/salzman/local/i4_gcc/include > --with-mumps-lib="[/home/salzma > n/local/i4_gcc/lib/libdmumps.so,/home/salzman/local/i4_gcc/lib/libmumps_common.so,/home/salzman/local/i4_gcc/lib/libpord.so]" > --with-scalapack-lib="[/opt/intel/oneapi/mkl/latest/lib/libmkl_ > scalapack_lp64.so,/opt/intel/oneapi/mkl/latest/lib/libmkl_blacs_openmpi_lp64.so]" > --with-mkl_pardiso=1 > --with-mkl_pardiso-include=/opt/intel/oneapi/mkl/latest/include > --with-mkl_pardiso-lib > ="[/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_intel_lp64.so]" > --with-hdf5=1 --with-hdf5-include=/usr/include/openmpi-x86_64 > --with-hdf5-lib="[/usr/lib64/openmpi/lib/libhdf5.so]" --with > -pastix=0 --download-pastix=no --with-hwloc=1 > --with-hwloc-dir=/home/salzman/local/i4_gcc --download-hwloc=no > --with-ptscotch-include=/home/salzman/local/i4_gcc/include > --with-ptscotch-lib= > "[/home/salzman/local/i4_gcc/lib/libptscotch.a,/home/salzman/local/i4_gcc/lib/libptscotcherr.a,/home/salzman/local/i4_gcc/lib/libptscotcherrexit.a,/home/salzman/local/i4_gcc/lib/libscotch.a > ,/home/salzman/local/i4_gcc/lib/libscotcherr.a,/home/salzman/local/i4_gcc/lib/libscotcherrexit.a]" > --with-hypre=1 --download-hypre=yes --with-suitesparse=1 > --with-suitesparse-include=/home/ > salzman/local/i4_gcc/include > --with-suitesparse-lib="[/home/salzman/local/i4_gcc/lib/libsuitesparseconfig.so,/home/salzman/local/i4_gcc/lib/libumfpack.so,/home/salzman/local/i4_gcc/lib/libk > lu.so,/home/salzman/local/i4_gcc/lib/libcholmod.so,/home/salzman/local/i4_gcc/lib/libspqr.so,/home/salzman/local/i4_gcc/lib/libcolamd.so,/home/salzman/local/i4_gcc/lib/libccolamd.so,/home/s > alzman/local/i4_gcc/lib/libcamd.so,/home/salzman/local/i4_gcc/lib/libamd.so,/home/salzman/local/i4_gcc/lib/libmetis.so]" > --download-suitesparse=no --with-python-exec=python3.12 --have-numpy > =1 ---with-petsc4py=1 ---with-petsc4py-test-np=4 ---with-mpi4py=1 > --prefix=/home/salzman/local/i4_gcc/real_arithmetic COPTFLAGS="-O3 -g > " CXXOPTFLAGS="-O3 -g " FOPTFLAGS="-O3 -g " > [0]PETSC ERROR: #1 MatSetValues_SeqAIJ() at > /home/salzman/devel/PETSc/petsc/src/mat/impls/aij/seq/aij.c:426 > [0]PETSC ERROR: #2 MatSetValues() at > /home/salzman/devel/PETSc/petsc/src/mat/interface/matrix.c:1543 > [0]PETSC ERROR: #3 MatSetSeqMats_MPIAIJ() at > /home/salzman/devel/PETSc/petsc/src/mat/impls/aij/mpi/mpiov.c:2965 > [0]PETSC ERROR: #4 MatCreateSubMatricesMPI_MPIXAIJ() at > /home/salzman/devel/PETSc/petsc/src/mat/impls/aij/mpi/mpiov.c:3163 > [0]PETSC ERROR: #5 MatCreateSubMatricesMPI_MPIAIJ() at > /home/salzman/devel/PETSc/petsc/src/mat/impls/aij/mpi/mpiov.c:3196 > [0]PETSC ERROR: #6 MatCreateSubMatricesMPI() at > /home/salzman/devel/PETSc/petsc/src/mat/interface/matrix.c:7293 > [0]PETSC ERROR: #7 main() at subnb.c:181 > [0]PETSC ERROR: No PETSc Option Table entries > [0]PETSC ERROR: ----------------End of Error Message -------send > entire error message to petsc-maint at mcs.anl.gov---------- > -------------------------------------------------------------------------- > > This message comes from executing the attached test (I simplified the > test by removing the block size from the matrix used for extraction, > compared to the July test). In proc_xx_output.txt, you will find the > output from the code execution with the -ok option (i.e. irow/idxr and > icol/idxc are the same, i.e. a square sub-block for colour 0 > distributed across the first two processes). > > Has expected in this case we obtain the 0,3,6,9 sub-block terms, which > are distributed across processes 0 and 1 (two rows per proc). > > When asking for rectangular sub-block (i.e. with no option) it crash > with column to large on process 0: 4 col max 3 ??? I ask for 4 rows > and 2 columns in this process ??? > > Otherwise, I mention the dense aspect of the matrix in ex183.c, > because, in this case, no matter what selection is requested, all > terms are non-null. If there is an issue with the way the selection is > coded in the user program, I think it will be masked thanks to the > full graph representation. However, this may not be the case ? I > should test it. > > I'll take a look at ex23.c. > > Thanks, > > A.S. > > > > Le 25/08/2025 ? 17:55, Mark Adams a ?crit?: >> Ah, OK, never say never. >> >> MatCreateSubMatrices seems to support creating a new matrix with the >> communicator of the IS. >> It just needs to read from the input matrix and does not use it for >> communication, so it can do that. >> >> As far as rectangular matrices, there is no reason not to support >> that (the row IS and column IS can be distinct). >> Can you send the whole error message? >> There may not be a test that does this, but?src/mat/tests/ex23.c >> looks like it may be a rectangular?matrix output. >> >> And, it should not matter if the input matrix has a 100% full sparse >> matrix. It is still MatAIJ. >> The semantics and API is the same for sparse or dense matrices. >> >> Thanks, >> Mark >> >> On Mon, Aug 25, 2025 at 7:31?AM Alexis SALZMAN >> wrote: >> >> Hi, >> >> Thanks for your answer, Mark. Perhaps MatCreateSubMatricesMPI is >> the only PETSc function that acts on a sub-communicator ? I'm not >> sure ? but it's clear that there's no ambiguity on that point. >> The first line of the documentation for that function states that >> it 'may live on subcomms'. This is confirmed by the >> 'src/mat/tests/ex183.c' test case. I used this test case to >> understand the function, which helped me with my code and the >> example I provided in my initial post. Unfortunately, in this >> example, the matrix from which the sub-matrices are extracted is >> dense, even though it uses a sparse structure. This does not >> clarify how to define sub-matrices when extracting from a sparse >> distributed matrix. Since my initial post, I have discovered that >> having more columns than rows can also result in the same error >> message. >> >> So, my questions boil down to: >> >> Can MatCreateSubMatricesMPI extract rectangular matrices from a >> square distributed sparse matrix? >> >> If not, the fact that only square matrices can be extracted in >> this context should perhaps be mentioned in the documentation. >> >> If so, I would be very grateful for any assistance in defining an >> IS pair in this context. >> >> Regards >> >> A.S. >> >> Le 27/07/2025 ? 00:15, Mark Adams a ?crit?: >>> First, you can not mix communicators in PETSc calls in general >>> (ever?), but this error looks like you?might be?asking for a row >>> from the matrix that?does not exist. >>> You should start with a PETSc example code. Test it and >>> modify?it to suit your needs. >>> >>> Good luck, >>> Mark >>> >>> On Fri, Jul 25, 2025 at 9:31?AM Alexis SALZMAN >>> wrote: >>> >>> Hi, >>> >>> As I am relatively new to Petsc, I may have misunderstood >>> how to use the >>> MatCreateSubMatricesMPI function. The attached code is tuned >>> for three >>> processes and extracts one matrix for each colour of a >>> subcommunicator >>> that has been created using the MPI_Comm_split function from >>> an? MPIAij >>> matrix. The following error message appears when the code is >>> set to its >>> default configuration (i.e. when a rectangular matrix is >>> extracted with >>> more rows than columns for colour 0): >>> >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: Argument out of range >>> [0]PETSC ERROR: Column too large: col 4 max 3 >>> [0]PETSC ERROR: See >>> https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!ZqH097BZ0G0O3WI7RWrwIKFNpyk0czSWEqfusAeTlgEygAffwpgBUzsLw1TIoGkjZ3mYG-NRQxxFoxU4y8EyY0ofiz9I43Qwe0w$ >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Release Version 3.22.2, unknown >>> >>> ... petsc git hash 2a89477b25f compiled on a dell i9 >>> computer with Gcc >>> 14.3, mkl 2025.2, ..... >>> [0]PETSC ERROR: #1 MatSetValues_SeqAIJ() at >>> ...petsc/src/mat/impls/aij/seq/aij.c:426 >>> [0]PETSC ERROR: #2 MatSetValues() at >>> ...petsc/src/mat/interface/matrix.c:1543 >>> [0]PETSC ERROR: #3 MatSetSeqMats_MPIAIJ() at >>> .../petsc/src/mat/impls/aij/mpi/mpiov.c:2965 >>> [0]PETSC ERROR: #4 MatCreateSubMatricesMPI_MPIXAIJ() at >>> .../petsc/src/mat/impls/aij/mpi/mpiov.c:3163 >>> [0]PETSC ERROR: #5 MatCreateSubMatricesMPI_MPIAIJ() at >>> .../petsc/src/mat/impls/aij/mpi/mpiov.c:3196 >>> [0]PETSC ERROR: #6 MatCreateSubMatricesMPI() at >>> .../petsc/src/mat/interface/matrix.c:7293 >>> [0]PETSC ERROR: #7 main() at sub.c:169 >>> >>> When the '-ok' option is selected, the code extracts a >>> square matrix for >>> colour 0, which runs smoothly in this case. Selecting the >>> '-trans' >>> option swaps the row and column selection indices, providing a >>> transposed submatrix smoothly. For colour 1, which uses only >>> one process >>> and is therefore sequential, rectangular extraction is OK >>> regardless of >>> the shape. >>> >>> Is this dependency on the shape expected? Have I missed an >>> important >>> tuning step somewhere? >>> >>> Thank you in advance for any clarification. >>> >>> Regards >>> >>> A.S. >>> >>> P.S.: I'm sorry, but as I'm leaving my office for the >>> following weeks >>> this evening, I won't be very responsive during this period. >>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierre at joliv.et Tue Aug 26 08:18:32 2025 From: pierre at joliv.et (Pierre Jolivet) Date: Tue, 26 Aug 2025 15:18:32 +0200 Subject: [petsc-users] MatCreateSubMatricesMPI strange behavior In-Reply-To: <82d06db6-ab86-49b8-9e26-000745ae2134@ec-nantes.fr> References: <9d648a1d-72d2-44d0-8a3b-a9d64b01604f@ec-nantes.fr> <113ece7d-25cf-4b16-b29b-bdd1ad2a004a@ec-nantes.fr> <82d06db6-ab86-49b8-9e26-000745ae2134@ec-nantes.fr> Message-ID: <8A46BABF-E664-4923-87DF-A407B29163A8@joliv.et> > On 26 Aug 2025, at 12:50?PM, Alexis SALZMAN wrote: > > Mark, you were right and I was wrong about the dense matrix. Adding explicit zeros to the distributed matrix used to extract the sub-matrices (making it dense) in my test does not change the behaviour: there is still an error. > > I am finding it increasingly difficult to understand the logic of the row and column 'IS' creation. I ran many tests to achieve the desired result: a rectangular sub-matrix (so a rectangular or square sub-matrix appears to be possible). However, many others resulted in the same kind of error. > This may be a PETSc bug in MatSetSeqMats_MPIAIJ(). -> 2965 PetscCall(MatSetValues(aij->B, 1, &row, 1, &col, &v, INSERT_VALUES)); col has a value of 4, which doesn?t make sense since the output Mat has 4 columns (thus, has the error message suggests, the value should be lower than or equal to 3). Thanks, Pierre > From what I observed, the test only works if the column selection contribution (size_c in the test) has a specific value related to the row selection contribution (size_r in the test) for proc 0 (rank for both communicator and sub-communicator): > > if size_r==2 then if size_c<=2 it works. > if size_r>=3 and size_r<=5 then size_c==size_r is the only working case. > This occurs "regardless" of what is requested in proc 1 and in selr/selc (It can't be a dummy setting, though). In any case, it's certainly not an exhaustive analysis. > > Many thanks to anyone who can explain to me the logic behind the construction of row and column 'IS'. > > Regards > > A.S. > > > > Le 25/08/2025 ? 20:00, Alexis SALZMAN a ?crit : >> Thanks Mark for your attention. >> >> The uncleaned error message, compared to my post in July, is as follows: >> >> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> [0]PETSC ERROR: Argument out of range >> [0]PETSC ERROR: Column too large: col 4 max 3 >> [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!dQLn3dzkbI887Zoyl2HUdmbl2gPS0L9k5B1aeBRkfKj82LGaBD5jiWw1KisbOuNwOmEBwqHDa6wo1du-6sKBtA$ for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.22.2, unknown >> [0]PETSC ERROR: subnb with 3 MPI process(es) and PETSC_ARCH on pc-str97.ec-nantes.fr by salzman Mon Aug 25 19:11:37 2025 >> [0]PETSC ERROR: Configure options: PETSC_ARCH=real_fc41_Release_gcc_i4 PETSC_DIR=/home/salzman/devel/ExternalLib/build/PETSC/petsc --doCleanup=1 --with-scalar-type=real --known-level1-dcach >> e-linesize=64 --with-cc=gcc --CFLAGS="-fPIC " --CC_LINKER_FLAGS=-fopenmp --with-cxx=g++ --with-cxx-dialect=c++20 --CXXFLAGS="-fPIC " --CXX_LINKER_FLAGS=-fopenmp --with-fc=gfortran --FFLAGS= >> "-fPIC " --FC_LINKER_FLAGS=-fopenmp --with-debugging=0 --with-fortran-bindings=0 --with-fortran-kernels=1 --with-mpi-compilers=0 --with-mpi-include=/usr/include/openmpi-x86_64 --with-mpi-li >> b="[/usr/lib64/openmpi/lib/libmpi.so,/usr/lib64/openmpi/lib/libmpi.so,/usr/lib64/openmpi/lib/libmpi_mpifh.so]" --with-blas-lib="[/opt/intel/oneapi/mkl/latest/lib/libmkl_intel_lp64.so,/opt/i >> ntel/oneapi/mkl/latest/lib/libmkl_gnu_thread.so,/opt/intel/oneapi/mkl/latest/lib/libmkl_core.so]" --with-lapack-lib="[/opt/intel/oneapi/mkl/latest/lib/libmkl_intel_lp64.so,/opt/intel/oneapi >> /mkl/latest/lib/libmkl_gnu_thread.so,/opt/intel/oneapi/mkl/latest/lib/libmkl_core.so]" --with-mumps=1 --with-mumps-include=/home/salzman/local/i4_gcc/include --with-mumps-lib="[/home/salzma >> n/local/i4_gcc/lib/libdmumps.so,/home/salzman/local/i4_gcc/lib/libmumps_common.so,/home/salzman/local/i4_gcc/lib/libpord.so]" --with-scalapack-lib="[/opt/intel/oneapi/mkl/latest/lib/libmkl_ >> scalapack_lp64.so,/opt/intel/oneapi/mkl/latest/lib/libmkl_blacs_openmpi_lp64.so]" --with-mkl_pardiso=1 --with-mkl_pardiso-include=/opt/intel/oneapi/mkl/latest/include --with-mkl_pardiso-lib >> ="[/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_intel_lp64.so]" --with-hdf5=1 --with-hdf5-include=/usr/include/openmpi-x86_64 --with-hdf5-lib="[/usr/lib64/openmpi/lib/libhdf5.so]" --with >> -pastix=0 --download-pastix=no --with-hwloc=1 --with-hwloc-dir=/home/salzman/local/i4_gcc --download-hwloc=no --with-ptscotch-include=/home/salzman/local/i4_gcc/include --with-ptscotch-lib= >> "[/home/salzman/local/i4_gcc/lib/libptscotch.a,/home/salzman/local/i4_gcc/lib/libptscotcherr.a,/home/salzman/local/i4_gcc/lib/libptscotcherrexit.a,/home/salzman/local/i4_gcc/lib/libscotch.a >> ,/home/salzman/local/i4_gcc/lib/libscotcherr.a,/home/salzman/local/i4_gcc/lib/libscotcherrexit.a]" --with-hypre=1 --download-hypre=yes --with-suitesparse=1 --with-suitesparse-include=/home/ >> salzman/local/i4_gcc/include --with-suitesparse-lib="[/home/salzman/local/i4_gcc/lib/libsuitesparseconfig.so,/home/salzman/local/i4_gcc/lib/libumfpack.so,/home/salzman/local/i4_gcc/lib/libk >> lu.so,/home/salzman/local/i4_gcc/lib/libcholmod.so,/home/salzman/local/i4_gcc/lib/libspqr.so,/home/salzman/local/i4_gcc/lib/libcolamd.so,/home/salzman/local/i4_gcc/lib/libccolamd.so,/home/s >> alzman/local/i4_gcc/lib/libcamd.so,/home/salzman/local/i4_gcc/lib/libamd.so,/home/salzman/local/i4_gcc/lib/libmetis.so]" --download-suitesparse=no --with-python-exec=python3.12 --have-numpy >> =1 ---with-petsc4py=1 ---with-petsc4py-test-np=4 ---with-mpi4py=1 --prefix=/home/salzman/local/i4_gcc/real_arithmetic COPTFLAGS="-O3 -g " CXXOPTFLAGS="-O3 -g " FOPTFLAGS="-O3 -g " >> [0]PETSC ERROR: #1 MatSetValues_SeqAIJ() at /home/salzman/devel/PETSc/petsc/src/mat/impls/aij/seq/aij.c:426 >> [0]PETSC ERROR: #2 MatSetValues() at /home/salzman/devel/PETSc/petsc/src/mat/interface/matrix.c:1543 >> [0]PETSC ERROR: #3 MatSetSeqMats_MPIAIJ() at /home/salzman/devel/PETSc/petsc/src/mat/impls/aij/mpi/mpiov.c:2965 >> [0]PETSC ERROR: #4 MatCreateSubMatricesMPI_MPIXAIJ() at /home/salzman/devel/PETSc/petsc/src/mat/impls/aij/mpi/mpiov.c:3163 >> [0]PETSC ERROR: #5 MatCreateSubMatricesMPI_MPIAIJ() at /home/salzman/devel/PETSc/petsc/src/mat/impls/aij/mpi/mpiov.c:3196 >> [0]PETSC ERROR: #6 MatCreateSubMatricesMPI() at /home/salzman/devel/PETSc/petsc/src/mat/interface/matrix.c:7293 >> [0]PETSC ERROR: #7 main() at subnb.c:181 >> [0]PETSC ERROR: No PETSc Option Table entries >> [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov ---------- >> -------------------------------------------------------------------------- >> >> This message comes from executing the attached test (I simplified the test by removing the block size from the matrix used for extraction, compared to the July test). In proc_xx_output.txt, you will find the output from the code execution with the -ok option (i.e. irow/idxr and icol/idxc are the same, i.e. a square sub-block for colour 0 distributed across the first two processes). >> >> Has expected in this case we obtain the 0,3,6,9 sub-block terms, which are distributed across processes 0 and 1 (two rows per proc). >> >> When asking for rectangular sub-block (i.e. with no option) it crash with column to large on process 0: 4 col max 3 ??? I ask for 4 rows and 2 columns in this process ??? >> >> Otherwise, I mention the dense aspect of the matrix in ex183.c, because, in this case, no matter what selection is requested, all terms are non-null. If there is an issue with the way the selection is coded in the user program, I think it will be masked thanks to the full graph representation. However, this may not be the case ? I should test it. >> >> I'll take a look at ex23.c. >> >> Thanks, >> >> A.S. >> >> >> >> >> >> Le 25/08/2025 ? 17:55, Mark Adams a ?crit : >>> Ah, OK, never say never. >>> >>> MatCreateSubMatrices seems to support creating a new matrix with the communicator of the IS. >>> It just needs to read from the input matrix and does not use it for communication, so it can do that. >>> >>> As far as rectangular matrices, there is no reason not to support that (the row IS and column IS can be distinct). >>> Can you send the whole error message? >>> There may not be a test that does this, but src/mat/tests/ex23.c looks like it may be a rectangular matrix output. >>> >>> And, it should not matter if the input matrix has a 100% full sparse matrix. It is still MatAIJ. >>> The semantics and API is the same for sparse or dense matrices. >>> >>> Thanks, >>> Mark >>> >>> On Mon, Aug 25, 2025 at 7:31?AM Alexis SALZMAN > wrote: >>>> Hi, >>>> >>>> Thanks for your answer, Mark. Perhaps MatCreateSubMatricesMPI is the only PETSc function that acts on a sub-communicator ? I'm not sure ? but it's clear that there's no ambiguity on that point. The first line of the documentation for that function states that it 'may live on subcomms'. This is confirmed by the 'src/mat/tests/ex183.c' test case. I used this test case to understand the function, which helped me with my code and the example I provided in my initial post. Unfortunately, in this example, the matrix from which the sub-matrices are extracted is dense, even though it uses a sparse structure. This does not clarify how to define sub-matrices when extracting from a sparse distributed matrix. Since my initial post, I have discovered that having more columns than rows can also result in the same error message. >>>> So, my questions boil down to: >>>> >>>> Can MatCreateSubMatricesMPI extract rectangular matrices from a square distributed sparse matrix? >>>> >>>> If not, the fact that only square matrices can be extracted in this context should perhaps be mentioned in the documentation. >>>> >>>> If so, I would be very grateful for any assistance in defining an IS pair in this context. >>>> >>>> Regards >>>> >>>> A.S. >>>> >>>> Le 27/07/2025 ? 00:15, Mark Adams a ?crit : >>>>> First, you can not mix communicators in PETSc calls in general (ever?), but this error looks like you might be asking for a row from the matrix that does not exist. >>>>> You should start with a PETSc example code. Test it and modify it to suit your needs. >>>>> >>>>> Good luck, >>>>> Mark >>>>> >>>>> On Fri, Jul 25, 2025 at 9:31?AM Alexis SALZMAN > wrote: >>>>>> Hi, >>>>>> >>>>>> As I am relatively new to Petsc, I may have misunderstood how to use the >>>>>> MatCreateSubMatricesMPI function. The attached code is tuned for three >>>>>> processes and extracts one matrix for each colour of a subcommunicator >>>>>> that has been created using the MPI_Comm_split function from an MPIAij >>>>>> matrix. The following error message appears when the code is set to its >>>>>> default configuration (i.e. when a rectangular matrix is extracted with >>>>>> more rows than columns for colour 0): >>>>>> >>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>> -------------------------------------------------------------- >>>>>> [0]PETSC ERROR: Argument out of range >>>>>> [0]PETSC ERROR: Column too large: col 4 max 3 >>>>>> [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!ZqH097BZ0G0O3WI7RWrwIKFNpyk0czSWEqfusAeTlgEygAffwpgBUzsLw1TIoGkjZ3mYG-NRQxxFoxU4y8EyY0ofiz9I43Qwe0w$ for trouble shooting. >>>>>> [0]PETSC ERROR: Petsc Release Version 3.22.2, unknown >>>>>> >>>>>> ... petsc git hash 2a89477b25f compiled on a dell i9 computer with Gcc >>>>>> 14.3, mkl 2025.2, ..... >>>>>> [0]PETSC ERROR: #1 MatSetValues_SeqAIJ() at >>>>>> ...petsc/src/mat/impls/aij/seq/aij.c:426 >>>>>> [0]PETSC ERROR: #2 MatSetValues() at >>>>>> ...petsc/src/mat/interface/matrix.c:1543 >>>>>> [0]PETSC ERROR: #3 MatSetSeqMats_MPIAIJ() at >>>>>> .../petsc/src/mat/impls/aij/mpi/mpiov.c:2965 >>>>>> [0]PETSC ERROR: #4 MatCreateSubMatricesMPI_MPIXAIJ() at >>>>>> .../petsc/src/mat/impls/aij/mpi/mpiov.c:3163 >>>>>> [0]PETSC ERROR: #5 MatCreateSubMatricesMPI_MPIAIJ() at >>>>>> .../petsc/src/mat/impls/aij/mpi/mpiov.c:3196 >>>>>> [0]PETSC ERROR: #6 MatCreateSubMatricesMPI() at >>>>>> .../petsc/src/mat/interface/matrix.c:7293 >>>>>> [0]PETSC ERROR: #7 main() at sub.c:169 >>>>>> >>>>>> When the '-ok' option is selected, the code extracts a square matrix for >>>>>> colour 0, which runs smoothly in this case. Selecting the '-trans' >>>>>> option swaps the row and column selection indices, providing a >>>>>> transposed submatrix smoothly. For colour 1, which uses only one process >>>>>> and is therefore sequential, rectangular extraction is OK regardless of >>>>>> the shape. >>>>>> >>>>>> Is this dependency on the shape expected? Have I missed an important >>>>>> tuning step somewhere? >>>>>> >>>>>> Thank you in advance for any clarification. >>>>>> >>>>>> Regards >>>>>> >>>>>> A.S. >>>>>> >>>>>> P.S.: I'm sorry, but as I'm leaving my office for the following weeks >>>>>> this evening, I won't be very responsive during this period. >>>>>> >>>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Aug 26 08:37:34 2025 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 26 Aug 2025 09:37:34 -0400 Subject: [petsc-users] MatCreateSubMatricesMPI strange behavior In-Reply-To: <8A46BABF-E664-4923-87DF-A407B29163A8@joliv.et> References: <9d648a1d-72d2-44d0-8a3b-a9d64b01604f@ec-nantes.fr> <113ece7d-25cf-4b16-b29b-bdd1ad2a004a@ec-nantes.fr> <82d06db6-ab86-49b8-9e26-000745ae2134@ec-nantes.fr> <8A46BABF-E664-4923-87DF-A407B29163A8@joliv.et> Message-ID: On Tue, Aug 26, 2025 at 9:24?AM Pierre Jolivet wrote: > > On 26 Aug 2025, at 12:50?PM, Alexis SALZMAN > wrote: > > Mark, you were right and I was wrong about the dense matrix. Adding > explicit zeros to the distributed matrix used to extract the sub-matrices > (making it dense) in my test does not change the behaviour: there is still > an error. > > I am finding it increasingly difficult to understand the logic of the row > and column 'IS' creation. I ran many tests to achieve the desired result: a > rectangular sub-matrix (so a rectangular or square sub-matrix appears to be > possible). However, many others resulted in the same kind of error. > > This may be a PETSc bug in MatSetSeqMats_MPIAIJ(). > -> 2965 PetscCall(MatSetValues(aij->B, 1, &row, 1, &col, &v, > INSERT_VALUES)); > col has a value of 4, which doesn?t make sense since the output Mat has 4 > columns (thus, has the error message suggests, the value should be lower > than or equal to 3). > This is Dmitry's code to make GASM work I believe, so not the most extensively tested. If you can give us a small example of it failing, we can fix it. Thanks! Matt > Thanks, > Pierre > > From what I observed, the test only works if the column selection > contribution (size_c in the test) has a specific value related to the row > selection contribution (size_r in the test) for proc 0 (rank for both > communicator and sub-communicator): > > - if size_r==2 then if size_c<=2 it works. > - if size_r>=3 and size_r<=5 then size_c==size_r is the only working > case. > > This occurs "regardless" of what is requested in proc 1 and in selr/selc > (It can't be a dummy setting, though). In any case, it's certainly not an > exhaustive analysis. > > Many thanks to anyone who can explain to me the logic behind the > construction of row and column 'IS'. > > Regards > > A.S. > > > Le 25/08/2025 ? 20:00, Alexis SALZMAN a ?crit : > > Thanks Mark for your attention. > > The uncleaned error message, compared to my post in July, is as follows: > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Argument out of range > [0]PETSC ERROR: Column too large: col 4 max 3 > [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!fBY9e0RmPNyVN9uPJ14k4sUmw0SCKHmNlb2PHAD_AML_6-p3fRQsua7ZkuJAltkbamfuDJ3hmV63V52V_vQ2$ > > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.22.2, unknown > [0]PETSC ERROR: subnb with 3 MPI process(es) and PETSC_ARCH on > pc-str97.ec-nantes.fr by salzman Mon Aug 25 19:11:37 2025 > [0]PETSC ERROR: Configure options: PETSC_ARCH=real_fc41_Release_gcc_i4 > PETSC_DIR=/home/salzman/devel/ExternalLib/build/PETSC/petsc --doCleanup=1 > --with-scalar-type=real --known-level1-dcach > e-linesize=64 --with-cc=gcc --CFLAGS="-fPIC " --CC_LINKER_FLAGS=-fopenmp > --with-cxx=g++ --with-cxx-dialect=c++20 --CXXFLAGS="-fPIC " > --CXX_LINKER_FLAGS=-fopenmp --with-fc=gfortran --FFLAGS= > "-fPIC " --FC_LINKER_FLAGS=-fopenmp --with-debugging=0 > --with-fortran-bindings=0 --with-fortran-kernels=1 --with-mpi-compilers=0 > --with-mpi-include=/usr/include/openmpi-x86_64 --with-mpi-li > b="[/usr/lib64/openmpi/lib/libmpi.so,/usr/lib64/openmpi/lib/libmpi.so,/usr/lib64/openmpi/lib/libmpi_mpifh.so]" > --with-blas-lib="[/opt/intel/oneapi/mkl/latest/lib/libmkl_intel_lp64.so,/opt/i > ntel/oneapi/mkl/latest/lib/libmkl_gnu_thread.so,/opt/intel/oneapi/mkl/latest/lib/libmkl_core.so]" > --with-lapack-lib="[/opt/intel/oneapi/mkl/latest/lib/libmkl_intel_lp64.so,/opt/intel/oneapi > /mkl/latest/lib/libmkl_gnu_thread.so,/opt/intel/oneapi/mkl/latest/lib/libmkl_core.so]" > --with-mumps=1 --with-mumps-include=/home/salzman/local/i4_gcc/include > --with-mumps-lib="[/home/salzma > n/local/i4_gcc/lib/libdmumps.so,/home/salzman/local/i4_gcc/lib/libmumps_common.so,/home/salzman/local/i4_gcc/lib/libpord.so]" > --with-scalapack-lib="[/opt/intel/oneapi/mkl/latest/lib/libmkl_ > scalapack_lp64.so,/opt/intel/oneapi/mkl/latest/lib/libmkl_blacs_openmpi_lp64.so]" > --with-mkl_pardiso=1 > --with-mkl_pardiso-include=/opt/intel/oneapi/mkl/latest/include > --with-mkl_pardiso-lib > ="[/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_intel_lp64.so]" > --with-hdf5=1 --with-hdf5-include=/usr/include/openmpi-x86_64 > --with-hdf5-lib="[/usr/lib64/openmpi/lib/libhdf5.so]" --with > -pastix=0 --download-pastix=no --with-hwloc=1 > --with-hwloc-dir=/home/salzman/local/i4_gcc --download-hwloc=no > --with-ptscotch-include=/home/salzman/local/i4_gcc/include > --with-ptscotch-lib= > > "[/home/salzman/local/i4_gcc/lib/libptscotch.a,/home/salzman/local/i4_gcc/lib/libptscotcherr.a,/home/salzman/local/i4_gcc/lib/libptscotcherrexit.a,/home/salzman/local/i4_gcc/lib/libscotch.a > ,/home/salzman/local/i4_gcc/lib/libscotcherr.a,/home/salzman/local/i4_gcc/lib/libscotcherrexit.a]" > --with-hypre=1 --download-hypre=yes --with-suitesparse=1 > --with-suitesparse-include=/home/ > salzman/local/i4_gcc/include > --with-suitesparse-lib="[/home/salzman/local/i4_gcc/lib/libsuitesparseconfig.so,/home/salzman/local/i4_gcc/lib/libumfpack.so,/home/salzman/local/i4_gcc/lib/libk > > lu.so,/home/salzman/local/i4_gcc/lib/libcholmod.so,/home/salzman/local/i4_gcc/lib/libspqr.so,/home/salzman/local/i4_gcc/lib/libcolamd.so,/home/salzman/local/i4_gcc/lib/libccolamd.so,/home/s > alzman/local/i4_gcc/lib/libcamd.so,/home/salzman/local/i4_gcc/lib/libamd.so,/home/salzman/local/i4_gcc/lib/libmetis.so]" > --download-suitesparse=no --with-python-exec=python3.12 --have-numpy > =1 ---with-petsc4py=1 ---with-petsc4py-test-np=4 ---with-mpi4py=1 > --prefix=/home/salzman/local/i4_gcc/real_arithmetic COPTFLAGS="-O3 -g " > CXXOPTFLAGS="-O3 -g " FOPTFLAGS="-O3 -g " > [0]PETSC ERROR: #1 MatSetValues_SeqAIJ() at > /home/salzman/devel/PETSc/petsc/src/mat/impls/aij/seq/aij.c:426 > [0]PETSC ERROR: #2 MatSetValues() at > /home/salzman/devel/PETSc/petsc/src/mat/interface/matrix.c:1543 > [0]PETSC ERROR: #3 MatSetSeqMats_MPIAIJ() at > /home/salzman/devel/PETSc/petsc/src/mat/impls/aij/mpi/mpiov.c:2965 > [0]PETSC ERROR: #4 MatCreateSubMatricesMPI_MPIXAIJ() at > /home/salzman/devel/PETSc/petsc/src/mat/impls/aij/mpi/mpiov.c:3163 > [0]PETSC ERROR: #5 MatCreateSubMatricesMPI_MPIAIJ() at > /home/salzman/devel/PETSc/petsc/src/mat/impls/aij/mpi/mpiov.c:3196 > [0]PETSC ERROR: #6 MatCreateSubMatricesMPI() at > /home/salzman/devel/PETSc/petsc/src/mat/interface/matrix.c:7293 > [0]PETSC ERROR: #7 main() at subnb.c:181 > [0]PETSC ERROR: No PETSc Option Table entries > [0]PETSC ERROR: ----------------End of Error Message -------send entire > error message to petsc-maint at mcs.anl.gov---------- > -------------------------------------------------------------------------- > > This message comes from executing the attached test (I simplified the test > by removing the block size from the matrix used for extraction, compared to > the July test). In proc_xx_output.txt, you will find the output from the > code execution with the -ok option (i.e. irow/idxr and icol/idxc are the > same, i.e. a square sub-block for colour 0 distributed across the first two > processes). > > Has expected in this case we obtain the 0,3,6,9 sub-block terms, which are > distributed across processes 0 and 1 (two rows per proc). > > When asking for rectangular sub-block (i.e. with no option) it crash with > column to large on process 0: 4 col max 3 ??? I ask for 4 rows and 2 > columns in this process ??? > > Otherwise, I mention the dense aspect of the matrix in ex183.c, because, > in this case, no matter what selection is requested, all terms are > non-null. If there is an issue with the way the selection is coded in the > user program, I think it will be masked thanks to the full graph > representation. However, this may not be the case ? I should test it. > > I'll take a look at ex23.c. > > Thanks, > > A.S. > > > > Le 25/08/2025 ? 17:55, Mark Adams a ?crit : > > Ah, OK, never say never. > > MatCreateSubMatrices seems to support creating a new matrix with the > communicator of the IS. > It just needs to read from the input matrix and does not use it for > communication, so it can do that. > > As far as rectangular matrices, there is no reason not to support that > (the row IS and column IS can be distinct). > Can you send the whole error message? > There may not be a test that does this, but src/mat/tests/ex23.c looks > like it may be a rectangular matrix output. > > And, it should not matter if the input matrix has a 100% full sparse > matrix. It is still MatAIJ. > The semantics and API is the same for sparse or dense matrices. > > Thanks, > Mark > > On Mon, Aug 25, 2025 at 7:31?AM Alexis SALZMAN < > alexis.salzman at ec-nantes.fr> wrote: > >> Hi, >> >> Thanks for your answer, Mark. Perhaps MatCreateSubMatricesMPI is the only >> PETSc function that acts on a sub-communicator ? I'm not sure ? but it's >> clear that there's no ambiguity on that point. The first line of the >> documentation for that function states that it 'may live on subcomms'. This >> is confirmed by the 'src/mat/tests/ex183.c' test case. I used this test >> case to understand the function, which helped me with my code and the >> example I provided in my initial post. Unfortunately, in this example, the >> matrix from which the sub-matrices are extracted is dense, even though it >> uses a sparse structure. This does not clarify how to define sub-matrices >> when extracting from a sparse distributed matrix. Since my initial post, I >> have discovered that having more columns than rows can also result in the >> same error message. >> >> So, my questions boil down to: >> >> Can MatCreateSubMatricesMPI extract rectangular matrices from a square >> distributed sparse matrix? >> >> If not, the fact that only square matrices can be extracted in this >> context should perhaps be mentioned in the documentation. >> >> If so, I would be very grateful for any assistance in defining an IS pair >> in this context. >> >> Regards >> >> A.S. >> Le 27/07/2025 ? 00:15, Mark Adams a ?crit : >> >> First, you can not mix communicators in PETSc calls in general (ever?), >> but this error looks like you might be asking for a row from the matrix >> that does not exist. >> You should start with a PETSc example code. Test it and modify it to suit >> your needs. >> >> Good luck, >> Mark >> >> On Fri, Jul 25, 2025 at 9:31?AM Alexis SALZMAN < >> alexis.salzman at ec-nantes.fr> wrote: >> >>> Hi, >>> >>> As I am relatively new to Petsc, I may have misunderstood how to use the >>> MatCreateSubMatricesMPI function. The attached code is tuned for three >>> processes and extracts one matrix for each colour of a subcommunicator >>> that has been created using the MPI_Comm_split function from an MPIAij >>> matrix. The following error message appears when the code is set to its >>> default configuration (i.e. when a rectangular matrix is extracted with >>> more rows than columns for colour 0): >>> >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: Argument out of range >>> [0]PETSC ERROR: Column too large: col 4 max 3 >>> [0]PETSC ERROR: See >>> https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!ZqH097BZ0G0O3WI7RWrwIKFNpyk0czSWEqfusAeTlgEygAffwpgBUzsLw1TIoGkjZ3mYG-NRQxxFoxU4y8EyY0ofiz9I43Qwe0w$ >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Release Version 3.22.2, unknown >>> >>> ... petsc git hash 2a89477b25f compiled on a dell i9 computer with Gcc >>> 14.3, mkl 2025.2, ..... >>> [0]PETSC ERROR: #1 MatSetValues_SeqAIJ() at >>> ...petsc/src/mat/impls/aij/seq/aij.c:426 >>> [0]PETSC ERROR: #2 MatSetValues() at >>> ...petsc/src/mat/interface/matrix.c:1543 >>> [0]PETSC ERROR: #3 MatSetSeqMats_MPIAIJ() at >>> .../petsc/src/mat/impls/aij/mpi/mpiov.c:2965 >>> [0]PETSC ERROR: #4 MatCreateSubMatricesMPI_MPIXAIJ() at >>> .../petsc/src/mat/impls/aij/mpi/mpiov.c:3163 >>> [0]PETSC ERROR: #5 MatCreateSubMatricesMPI_MPIAIJ() at >>> .../petsc/src/mat/impls/aij/mpi/mpiov.c:3196 >>> [0]PETSC ERROR: #6 MatCreateSubMatricesMPI() at >>> .../petsc/src/mat/interface/matrix.c:7293 >>> [0]PETSC ERROR: #7 main() at sub.c:169 >>> >>> When the '-ok' option is selected, the code extracts a square matrix for >>> colour 0, which runs smoothly in this case. Selecting the '-trans' >>> option swaps the row and column selection indices, providing a >>> transposed submatrix smoothly. For colour 1, which uses only one process >>> and is therefore sequential, rectangular extraction is OK regardless of >>> the shape. >>> >>> Is this dependency on the shape expected? Have I missed an important >>> tuning step somewhere? >>> >>> Thank you in advance for any clarification. >>> >>> Regards >>> >>> A.S. >>> >>> P.S.: I'm sorry, but as I'm leaving my office for the following weeks >>> this evening, I won't be very responsive during this period. >>> >>> >>> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fBY9e0RmPNyVN9uPJ14k4sUmw0SCKHmNlb2PHAD_AML_6-p3fRQsua7ZkuJAltkbamfuDJ3hmV63Vwm7-4zV$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierre at joliv.et Tue Aug 26 08:36:57 2025 From: pierre at joliv.et (Pierre Jolivet) Date: Tue, 26 Aug 2025 15:36:57 +0200 Subject: [petsc-users] MatCreateSubMatricesMPI strange behavior In-Reply-To: <8A46BABF-E664-4923-87DF-A407B29163A8@joliv.et> References: <9d648a1d-72d2-44d0-8a3b-a9d64b01604f@ec-nantes.fr> <113ece7d-25cf-4b16-b29b-bdd1ad2a004a@ec-nantes.fr> <82d06db6-ab86-49b8-9e26-000745ae2134@ec-nantes.fr> <8A46BABF-E664-4923-87DF-A407B29163A8@joliv.et> Message-ID: <37AC2024-4D4C-41BD-A543-07221FDFB2C3@joliv.et> It?s indeed very suspicious (to me) that we are using rmap to change a column index. Switching to cmap gets your code running, but I?ll need to see if this triggers regressions. Thanks for the report, Pierre diff --git a/src/mat/impls/aij/mpi/mpiov.c b/src/mat/impls/aij/mpi/mpiov.c index d1037d7d817..051981ebe9a 100644 --- a/src/mat/impls/aij/mpi/mpiov.c +++ b/src/mat/impls/aij/mpi/mpiov.c @@ -2948,3 +2948,3 @@ PetscErrorCode MatSetSeqMats_MPIAIJ(Mat C, IS rowemb, IS dcolemb, IS ocolemb, Ma - PetscCall(PetscLayoutGetRange(C->rmap, &rstart, &rend)); + PetscCall(PetscLayoutGetRange(C->cmap, &rstart, &rend)); shift = rend - rstart; $ cat proc_0_output.txt rstart 0 rend 4 Mat Object: 3 MPI processes type: mpiaij row 0: (0, 101.) (3, 104.) (6, 107.) (9, 110.) row 1: (2, 203.) (5, 206.) (8, 209.) (11, 212.) row 2: (1, 302.) (4, 305.) (7, 308.) (10, 311.) row 3: (0, 401.) (3, 404.) (6, 407.) (9, 410.) row 4: (2, 503.) (5, 506.) (8, 509.) (11, 512.) row 5: (1, 602.) (4, 605.) (7, 608.) (10, 611.) row 6: (0, 701.) (3, 704.) (6, 707.) (9, 710.) row 7: (2, 803.) (5, 806.) (8, 809.) (11, 812.) row 8: (1, 902.) (4, 905.) (7, 908.) (10, 911.) row 9: (0, 1001.) (3, 1004.) (6, 1007.) (9, 1010.) row 10: (2, 1103.) (5, 1106.) (8, 1109.) (11, 1112.) row 11: (1, 1202.) (4, 1205.) (7, 1208.) (10, 1211.) idxr proc IS Object: 2 MPI processes type: general [0] Number of indices in set 4 [0] 0 0 [0] 1 1 [0] 2 2 [0] 3 3 [1] Number of indices in set 4 [1] 0 4 [1] 1 5 [1] 2 6 [1] 3 7 idxc proc IS Object: 2 MPI processes type: general [0] Number of indices in set 2 [0] 0 0 [0] 1 1 [1] Number of indices in set 2 [1] 0 6 [1] 1 7 Mat Object: 2 MPI processes type: mpiaij row 0: (0, 101.) (2, 107.) row 1: row 2: (1, 302.) (3, 308.) row 3: (0, 401.) (2, 407.) row 4: row 5: (1, 602.) (3, 608.) row 6: (0, 701.) (2, 707.) row 7: rstart 0 rend 4 local row 0: ( 0 , 1.010000e+02) ( 2 , 1.070000e+02) local row 1: local row 2: ( 1 , 3.020000e+02) ( 3 , 3.080000e+02) local row 3: ( 0 , 4.010000e+02) ( 2 , 4.070000e+02) > On 26 Aug 2025, at 3:18?PM, Pierre Jolivet wrote: > > >> On 26 Aug 2025, at 12:50?PM, Alexis SALZMAN wrote: >> >> Mark, you were right and I was wrong about the dense matrix. Adding explicit zeros to the distributed matrix used to extract the sub-matrices (making it dense) in my test does not change the behaviour: there is still an error. >> >> I am finding it increasingly difficult to understand the logic of the row and column 'IS' creation. I ran many tests to achieve the desired result: a rectangular sub-matrix (so a rectangular or square sub-matrix appears to be possible). However, many others resulted in the same kind of error. >> > This may be a PETSc bug in MatSetSeqMats_MPIAIJ(). > -> 2965 PetscCall(MatSetValues(aij->B, 1, &row, 1, &col, &v, INSERT_VALUES)); > col has a value of 4, which doesn?t make sense since the output Mat has 4 columns (thus, has the error message suggests, the value should be lower than or equal to 3). > > Thanks, > Pierre >> From what I observed, the test only works if the column selection contribution (size_c in the test) has a specific value related to the row selection contribution (size_r in the test) for proc 0 (rank for both communicator and sub-communicator): >> >> if size_r==2 then if size_c<=2 it works. >> if size_r>=3 and size_r<=5 then size_c==size_r is the only working case. >> This occurs "regardless" of what is requested in proc 1 and in selr/selc (It can't be a dummy setting, though). In any case, it's certainly not an exhaustive analysis. >> >> Many thanks to anyone who can explain to me the logic behind the construction of row and column 'IS'. >> >> Regards >> >> A.S. >> >> >> >> Le 25/08/2025 ? 20:00, Alexis SALZMAN a ?crit : >>> Thanks Mark for your attention. >>> >>> The uncleaned error message, compared to my post in July, is as follows: >>> >>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>> [0]PETSC ERROR: Argument out of range >>> [0]PETSC ERROR: Column too large: col 4 max 3 >>> [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!fO8_CzT5tLNg4AugvsuMRWr-Lw8VlCt0VQgyHJ7BNmCJijU6SQdg9V02uTCc7FhyO65Ks2UyTNAqbbamtSViDQ$ for trouble shooting. >>> [0]PETSC ERROR: Petsc Release Version 3.22.2, unknown >>> [0]PETSC ERROR: subnb with 3 MPI process(es) and PETSC_ARCH on pc-str97.ec-nantes.fr by salzman Mon Aug 25 19:11:37 2025 >>> [0]PETSC ERROR: Configure options: PETSC_ARCH=real_fc41_Release_gcc_i4 PETSC_DIR=/home/salzman/devel/ExternalLib/build/PETSC/petsc --doCleanup=1 --with-scalar-type=real --known-level1-dcach >>> e-linesize=64 --with-cc=gcc --CFLAGS="-fPIC " --CC_LINKER_FLAGS=-fopenmp --with-cxx=g++ --with-cxx-dialect=c++20 --CXXFLAGS="-fPIC " --CXX_LINKER_FLAGS=-fopenmp --with-fc=gfortran --FFLAGS= >>> "-fPIC " --FC_LINKER_FLAGS=-fopenmp --with-debugging=0 --with-fortran-bindings=0 --with-fortran-kernels=1 --with-mpi-compilers=0 --with-mpi-include=/usr/include/openmpi-x86_64 --with-mpi-li >>> b="[/usr/lib64/openmpi/lib/libmpi.so,/usr/lib64/openmpi/lib/libmpi.so,/usr/lib64/openmpi/lib/libmpi_mpifh.so]" --with-blas-lib="[/opt/intel/oneapi/mkl/latest/lib/libmkl_intel_lp64.so,/opt/i >>> ntel/oneapi/mkl/latest/lib/libmkl_gnu_thread.so,/opt/intel/oneapi/mkl/latest/lib/libmkl_core.so]" --with-lapack-lib="[/opt/intel/oneapi/mkl/latest/lib/libmkl_intel_lp64.so,/opt/intel/oneapi >>> /mkl/latest/lib/libmkl_gnu_thread.so,/opt/intel/oneapi/mkl/latest/lib/libmkl_core.so]" --with-mumps=1 --with-mumps-include=/home/salzman/local/i4_gcc/include --with-mumps-lib="[/home/salzma >>> n/local/i4_gcc/lib/libdmumps.so,/home/salzman/local/i4_gcc/lib/libmumps_common.so,/home/salzman/local/i4_gcc/lib/libpord.so]" --with-scalapack-lib="[/opt/intel/oneapi/mkl/latest/lib/libmkl_ >>> scalapack_lp64.so,/opt/intel/oneapi/mkl/latest/lib/libmkl_blacs_openmpi_lp64.so]" --with-mkl_pardiso=1 --with-mkl_pardiso-include=/opt/intel/oneapi/mkl/latest/include --with-mkl_pardiso-lib >>> ="[/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_intel_lp64.so]" --with-hdf5=1 --with-hdf5-include=/usr/include/openmpi-x86_64 --with-hdf5-lib="[/usr/lib64/openmpi/lib/libhdf5.so]" --with >>> -pastix=0 --download-pastix=no --with-hwloc=1 --with-hwloc-dir=/home/salzman/local/i4_gcc --download-hwloc=no --with-ptscotch-include=/home/salzman/local/i4_gcc/include --with-ptscotch-lib= >>> "[/home/salzman/local/i4_gcc/lib/libptscotch.a,/home/salzman/local/i4_gcc/lib/libptscotcherr.a,/home/salzman/local/i4_gcc/lib/libptscotcherrexit.a,/home/salzman/local/i4_gcc/lib/libscotch.a >>> ,/home/salzman/local/i4_gcc/lib/libscotcherr.a,/home/salzman/local/i4_gcc/lib/libscotcherrexit.a]" --with-hypre=1 --download-hypre=yes --with-suitesparse=1 --with-suitesparse-include=/home/ >>> salzman/local/i4_gcc/include --with-suitesparse-lib="[/home/salzman/local/i4_gcc/lib/libsuitesparseconfig.so,/home/salzman/local/i4_gcc/lib/libumfpack.so,/home/salzman/local/i4_gcc/lib/libk >>> lu.so,/home/salzman/local/i4_gcc/lib/libcholmod.so,/home/salzman/local/i4_gcc/lib/libspqr.so,/home/salzman/local/i4_gcc/lib/libcolamd.so,/home/salzman/local/i4_gcc/lib/libccolamd.so,/home/s >>> alzman/local/i4_gcc/lib/libcamd.so,/home/salzman/local/i4_gcc/lib/libamd.so,/home/salzman/local/i4_gcc/lib/libmetis.so]" --download-suitesparse=no --with-python-exec=python3.12 --have-numpy >>> =1 ---with-petsc4py=1 ---with-petsc4py-test-np=4 ---with-mpi4py=1 --prefix=/home/salzman/local/i4_gcc/real_arithmetic COPTFLAGS="-O3 -g " CXXOPTFLAGS="-O3 -g " FOPTFLAGS="-O3 -g " >>> [0]PETSC ERROR: #1 MatSetValues_SeqAIJ() at /home/salzman/devel/PETSc/petsc/src/mat/impls/aij/seq/aij.c:426 >>> [0]PETSC ERROR: #2 MatSetValues() at /home/salzman/devel/PETSc/petsc/src/mat/interface/matrix.c:1543 >>> [0]PETSC ERROR: #3 MatSetSeqMats_MPIAIJ() at /home/salzman/devel/PETSc/petsc/src/mat/impls/aij/mpi/mpiov.c:2965 >>> [0]PETSC ERROR: #4 MatCreateSubMatricesMPI_MPIXAIJ() at /home/salzman/devel/PETSc/petsc/src/mat/impls/aij/mpi/mpiov.c:3163 >>> [0]PETSC ERROR: #5 MatCreateSubMatricesMPI_MPIAIJ() at /home/salzman/devel/PETSc/petsc/src/mat/impls/aij/mpi/mpiov.c:3196 >>> [0]PETSC ERROR: #6 MatCreateSubMatricesMPI() at /home/salzman/devel/PETSc/petsc/src/mat/interface/matrix.c:7293 >>> [0]PETSC ERROR: #7 main() at subnb.c:181 >>> [0]PETSC ERROR: No PETSc Option Table entries >>> [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov ---------- >>> -------------------------------------------------------------------------- >>> >>> This message comes from executing the attached test (I simplified the test by removing the block size from the matrix used for extraction, compared to the July test). In proc_xx_output.txt, you will find the output from the code execution with the -ok option (i.e. irow/idxr and icol/idxc are the same, i.e. a square sub-block for colour 0 distributed across the first two processes). >>> >>> Has expected in this case we obtain the 0,3,6,9 sub-block terms, which are distributed across processes 0 and 1 (two rows per proc). >>> >>> When asking for rectangular sub-block (i.e. with no option) it crash with column to large on process 0: 4 col max 3 ??? I ask for 4 rows and 2 columns in this process ??? >>> >>> Otherwise, I mention the dense aspect of the matrix in ex183.c, because, in this case, no matter what selection is requested, all terms are non-null. If there is an issue with the way the selection is coded in the user program, I think it will be masked thanks to the full graph representation. However, this may not be the case ? I should test it. >>> >>> I'll take a look at ex23.c. >>> >>> Thanks, >>> >>> A.S. >>> >>> >>> >>> >>> >>> Le 25/08/2025 ? 17:55, Mark Adams a ?crit : >>>> Ah, OK, never say never. >>>> >>>> MatCreateSubMatrices seems to support creating a new matrix with the communicator of the IS. >>>> It just needs to read from the input matrix and does not use it for communication, so it can do that. >>>> >>>> As far as rectangular matrices, there is no reason not to support that (the row IS and column IS can be distinct). >>>> Can you send the whole error message? >>>> There may not be a test that does this, but src/mat/tests/ex23.c looks like it may be a rectangular matrix output. >>>> >>>> And, it should not matter if the input matrix has a 100% full sparse matrix. It is still MatAIJ. >>>> The semantics and API is the same for sparse or dense matrices. >>>> >>>> Thanks, >>>> Mark >>>> >>>> On Mon, Aug 25, 2025 at 7:31?AM Alexis SALZMAN > wrote: >>>>> Hi, >>>>> >>>>> Thanks for your answer, Mark. Perhaps MatCreateSubMatricesMPI is the only PETSc function that acts on a sub-communicator ? I'm not sure ? but it's clear that there's no ambiguity on that point. The first line of the documentation for that function states that it 'may live on subcomms'. This is confirmed by the 'src/mat/tests/ex183.c' test case. I used this test case to understand the function, which helped me with my code and the example I provided in my initial post. Unfortunately, in this example, the matrix from which the sub-matrices are extracted is dense, even though it uses a sparse structure. This does not clarify how to define sub-matrices when extracting from a sparse distributed matrix. Since my initial post, I have discovered that having more columns than rows can also result in the same error message. >>>>> So, my questions boil down to: >>>>> >>>>> Can MatCreateSubMatricesMPI extract rectangular matrices from a square distributed sparse matrix? >>>>> >>>>> If not, the fact that only square matrices can be extracted in this context should perhaps be mentioned in the documentation. >>>>> >>>>> If so, I would be very grateful for any assistance in defining an IS pair in this context. >>>>> >>>>> Regards >>>>> >>>>> A.S. >>>>> >>>>> Le 27/07/2025 ? 00:15, Mark Adams a ?crit : >>>>>> First, you can not mix communicators in PETSc calls in general (ever?), but this error looks like you might be asking for a row from the matrix that does not exist. >>>>>> You should start with a PETSc example code. Test it and modify it to suit your needs. >>>>>> >>>>>> Good luck, >>>>>> Mark >>>>>> >>>>>> On Fri, Jul 25, 2025 at 9:31?AM Alexis SALZMAN > wrote: >>>>>>> Hi, >>>>>>> >>>>>>> As I am relatively new to Petsc, I may have misunderstood how to use the >>>>>>> MatCreateSubMatricesMPI function. The attached code is tuned for three >>>>>>> processes and extracts one matrix for each colour of a subcommunicator >>>>>>> that has been created using the MPI_Comm_split function from an MPIAij >>>>>>> matrix. The following error message appears when the code is set to its >>>>>>> default configuration (i.e. when a rectangular matrix is extracted with >>>>>>> more rows than columns for colour 0): >>>>>>> >>>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>>> -------------------------------------------------------------- >>>>>>> [0]PETSC ERROR: Argument out of range >>>>>>> [0]PETSC ERROR: Column too large: col 4 max 3 >>>>>>> [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!ZqH097BZ0G0O3WI7RWrwIKFNpyk0czSWEqfusAeTlgEygAffwpgBUzsLw1TIoGkjZ3mYG-NRQxxFoxU4y8EyY0ofiz9I43Qwe0w$ for trouble shooting. >>>>>>> [0]PETSC ERROR: Petsc Release Version 3.22.2, unknown >>>>>>> >>>>>>> ... petsc git hash 2a89477b25f compiled on a dell i9 computer with Gcc >>>>>>> 14.3, mkl 2025.2, ..... >>>>>>> [0]PETSC ERROR: #1 MatSetValues_SeqAIJ() at >>>>>>> ...petsc/src/mat/impls/aij/seq/aij.c:426 >>>>>>> [0]PETSC ERROR: #2 MatSetValues() at >>>>>>> ...petsc/src/mat/interface/matrix.c:1543 >>>>>>> [0]PETSC ERROR: #3 MatSetSeqMats_MPIAIJ() at >>>>>>> .../petsc/src/mat/impls/aij/mpi/mpiov.c:2965 >>>>>>> [0]PETSC ERROR: #4 MatCreateSubMatricesMPI_MPIXAIJ() at >>>>>>> .../petsc/src/mat/impls/aij/mpi/mpiov.c:3163 >>>>>>> [0]PETSC ERROR: #5 MatCreateSubMatricesMPI_MPIAIJ() at >>>>>>> .../petsc/src/mat/impls/aij/mpi/mpiov.c:3196 >>>>>>> [0]PETSC ERROR: #6 MatCreateSubMatricesMPI() at >>>>>>> .../petsc/src/mat/interface/matrix.c:7293 >>>>>>> [0]PETSC ERROR: #7 main() at sub.c:169 >>>>>>> >>>>>>> When the '-ok' option is selected, the code extracts a square matrix for >>>>>>> colour 0, which runs smoothly in this case. Selecting the '-trans' >>>>>>> option swaps the row and column selection indices, providing a >>>>>>> transposed submatrix smoothly. For colour 1, which uses only one process >>>>>>> and is therefore sequential, rectangular extraction is OK regardless of >>>>>>> the shape. >>>>>>> >>>>>>> Is this dependency on the shape expected? Have I missed an important >>>>>>> tuning step somewhere? >>>>>>> >>>>>>> Thank you in advance for any clarification. >>>>>>> >>>>>>> Regards >>>>>>> >>>>>>> A.S. >>>>>>> >>>>>>> P.S.: I'm sorry, but as I'm leaving my office for the following weeks >>>>>>> this evening, I won't be very responsive during this period. >>>>>>> >>>>>>> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Aug 26 08:46:11 2025 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 26 Aug 2025 09:46:11 -0400 Subject: [petsc-users] MatCreateSubMatricesMPI strange behavior In-Reply-To: <37AC2024-4D4C-41BD-A543-07221FDFB2C3@joliv.et> References: <9d648a1d-72d2-44d0-8a3b-a9d64b01604f@ec-nantes.fr> <113ece7d-25cf-4b16-b29b-bdd1ad2a004a@ec-nantes.fr> <82d06db6-ab86-49b8-9e26-000745ae2134@ec-nantes.fr> <8A46BABF-E664-4923-87DF-A407B29163A8@joliv.et> <37AC2024-4D4C-41BD-A543-07221FDFB2C3@joliv.et> Message-ID: On Tue, Aug 26, 2025 at 9:42?AM Pierre Jolivet wrote: > It?s indeed very suspicious (to me) that we are using rmap to change a > column index. > Switching to cmap gets your code running, but I?ll need to see if this > triggers regressions. > That looks right to me. I am sure this has only been tested for GASM, which would be symmetric. Thanks, Matt > Thanks for the report, > Pierre > > diff --git a/src/mat/impls/aij/mpi/mpiov.c b/src/mat/impls/aij/mpi/mpiov.c > index d1037d7d817..051981ebe9a 100644 > --- a/src/mat/impls/aij/mpi/mpiov.c > +++ b/src/mat/impls/aij/mpi/mpiov.c > @@ -2948,3 +2948,3 @@ PetscErrorCode MatSetSeqMats_MPIAIJ(Mat C, IS > rowemb, IS dcolemb, IS ocolemb, Ma > > - PetscCall(PetscLayoutGetRange(C->rmap, &rstart, &rend)); > + PetscCall(PetscLayoutGetRange(C->cmap, &rstart, &rend)); > shift = rend - rstart; > > $ cat proc_0_output.txt > rstart 0 rend 4 > Mat Object: 3 MPI processes > type: mpiaij > row 0: (0, 101.) (3, 104.) (6, 107.) (9, 110.) > row 1: (2, 203.) (5, 206.) (8, 209.) (11, 212.) > row 2: (1, 302.) (4, 305.) (7, 308.) (10, 311.) > row 3: (0, 401.) (3, 404.) (6, 407.) (9, 410.) > row 4: (2, 503.) (5, 506.) (8, 509.) (11, 512.) > row 5: (1, 602.) (4, 605.) (7, 608.) (10, 611.) > row 6: (0, 701.) (3, 704.) (6, 707.) (9, 710.) > row 7: (2, 803.) (5, 806.) (8, 809.) (11, 812.) > row 8: (1, 902.) (4, 905.) (7, 908.) (10, 911.) > row 9: (0, 1001.) (3, 1004.) (6, 1007.) (9, 1010.) > row 10: (2, 1103.) (5, 1106.) (8, 1109.) (11, 1112.) > row 11: (1, 1202.) (4, 1205.) (7, 1208.) (10, 1211.) > idxr proc > IS Object: 2 MPI processes > type: general > [0] Number of indices in set 4 > [0] 0 0 > [0] 1 1 > [0] 2 2 > [0] 3 3 > [1] Number of indices in set 4 > [1] 0 4 > [1] 1 5 > [1] 2 6 > [1] 3 7 > idxc proc > IS Object: 2 MPI processes > type: general > [0] Number of indices in set 2 > [0] 0 0 > [0] 1 1 > [1] Number of indices in set 2 > [1] 0 6 > [1] 1 7 > Mat Object: 2 MPI processes > type: mpiaij > row 0: (0, 101.) (2, 107.) > row 1: > row 2: (1, 302.) (3, 308.) > row 3: (0, 401.) (2, 407.) > row 4: > row 5: (1, 602.) (3, 608.) > row 6: (0, 701.) (2, 707.) > row 7: > rstart 0 rend 4 > local row 0: ( 0 , 1.010000e+02) ( 2 , 1.070000e+02) > local row 1: > local row 2: ( 1 , 3.020000e+02) ( 3 , 3.080000e+02) > local row 3: ( 0 , 4.010000e+02) ( 2 , 4.070000e+02) > > On 26 Aug 2025, at 3:18?PM, Pierre Jolivet wrote: > > > On 26 Aug 2025, at 12:50?PM, Alexis SALZMAN > wrote: > > Mark, you were right and I was wrong about the dense matrix. Adding > explicit zeros to the distributed matrix used to extract the sub-matrices > (making it dense) in my test does not change the behaviour: there is still > an error. > > I am finding it increasingly difficult to understand the logic of the row > and column 'IS' creation. I ran many tests to achieve the desired result: a > rectangular sub-matrix (so a rectangular or square sub-matrix appears to be > possible). However, many others resulted in the same kind of error. > > This may be a PETSc bug in MatSetSeqMats_MPIAIJ(). > -> 2965 PetscCall(MatSetValues(aij->B, 1, &row, 1, &col, &v, > INSERT_VALUES)); > col has a value of 4, which doesn?t make sense since the output Mat has 4 > columns (thus, has the error message suggests, the value should be lower > than or equal to 3). > > Thanks, > Pierre > > From what I observed, the test only works if the column selection > contribution (size_c in the test) has a specific value related to the row > selection contribution (size_r in the test) for proc 0 (rank for both > communicator and sub-communicator): > > - if size_r==2 then if size_c<=2 it works. > - if size_r>=3 and size_r<=5 then size_c==size_r is the only working > case. > > This occurs "regardless" of what is requested in proc 1 and in selr/selc > (It can't be a dummy setting, though). In any case, it's certainly not an > exhaustive analysis. > > Many thanks to anyone who can explain to me the logic behind the > construction of row and column 'IS'. > > Regards > > A.S. > > > Le 25/08/2025 ? 20:00, Alexis SALZMAN a ?crit : > > Thanks Mark for your attention. > > The uncleaned error message, compared to my post in July, is as follows: > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Argument out of range > [0]PETSC ERROR: Column too large: col 4 max 3 > [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!bHTBIkomOzv2YXBZD7vNmmrb0Ijc6Xd4lvZJ15CTTDE3_ewzI1zbkHE8LP2mjO3qm6VWQ8yao8uuUJllLwYe$ > > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.22.2, unknown > [0]PETSC ERROR: subnb with 3 MPI process(es) and PETSC_ARCH on > pc-str97.ec-nantes.fr by salzman Mon Aug 25 19:11:37 2025 > [0]PETSC ERROR: Configure options: PETSC_ARCH=real_fc41_Release_gcc_i4 > PETSC_DIR=/home/salzman/devel/ExternalLib/build/PETSC/petsc --doCleanup=1 > --with-scalar-type=real --known-level1-dcach > e-linesize=64 --with-cc=gcc --CFLAGS="-fPIC " --CC_LINKER_FLAGS=-fopenmp > --with-cxx=g++ --with-cxx-dialect=c++20 --CXXFLAGS="-fPIC " > --CXX_LINKER_FLAGS=-fopenmp --with-fc=gfortran --FFLAGS= > "-fPIC " --FC_LINKER_FLAGS=-fopenmp --with-debugging=0 > --with-fortran-bindings=0 --with-fortran-kernels=1 --with-mpi-compilers=0 > --with-mpi-include=/usr/include/openmpi-x86_64 --with-mpi-li > b="[/usr/lib64/openmpi/lib/libmpi.so,/usr/lib64/openmpi/lib/libmpi.so,/usr/lib64/openmpi/lib/libmpi_mpifh.so]" > --with-blas-lib="[/opt/intel/oneapi/mkl/latest/lib/libmkl_intel_lp64.so,/opt/i > ntel/oneapi/mkl/latest/lib/libmkl_gnu_thread.so,/opt/intel/oneapi/mkl/latest/lib/libmkl_core.so]" > --with-lapack-lib="[/opt/intel/oneapi/mkl/latest/lib/libmkl_intel_lp64.so,/opt/intel/oneapi > /mkl/latest/lib/libmkl_gnu_thread.so,/opt/intel/oneapi/mkl/latest/lib/libmkl_core.so]" > --with-mumps=1 --with-mumps-include=/home/salzman/local/i4_gcc/include > --with-mumps-lib="[/home/salzma > n/local/i4_gcc/lib/libdmumps.so,/home/salzman/local/i4_gcc/lib/libmumps_common.so,/home/salzman/local/i4_gcc/lib/libpord.so]" > --with-scalapack-lib="[/opt/intel/oneapi/mkl/latest/lib/libmkl_ > scalapack_lp64.so,/opt/intel/oneapi/mkl/latest/lib/libmkl_blacs_openmpi_lp64.so]" > --with-mkl_pardiso=1 > --with-mkl_pardiso-include=/opt/intel/oneapi/mkl/latest/include > --with-mkl_pardiso-lib > ="[/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_intel_lp64.so]" > --with-hdf5=1 --with-hdf5-include=/usr/include/openmpi-x86_64 > --with-hdf5-lib="[/usr/lib64/openmpi/lib/libhdf5.so]" --with > -pastix=0 --download-pastix=no --with-hwloc=1 > --with-hwloc-dir=/home/salzman/local/i4_gcc --download-hwloc=no > --with-ptscotch-include=/home/salzman/local/i4_gcc/include > --with-ptscotch-lib= > > "[/home/salzman/local/i4_gcc/lib/libptscotch.a,/home/salzman/local/i4_gcc/lib/libptscotcherr.a,/home/salzman/local/i4_gcc/lib/libptscotcherrexit.a,/home/salzman/local/i4_gcc/lib/libscotch.a > ,/home/salzman/local/i4_gcc/lib/libscotcherr.a,/home/salzman/local/i4_gcc/lib/libscotcherrexit.a]" > --with-hypre=1 --download-hypre=yes --with-suitesparse=1 > --with-suitesparse-include=/home/ > salzman/local/i4_gcc/include > --with-suitesparse-lib="[/home/salzman/local/i4_gcc/lib/libsuitesparseconfig.so,/home/salzman/local/i4_gcc/lib/libumfpack.so,/home/salzman/local/i4_gcc/lib/libk > > lu.so,/home/salzman/local/i4_gcc/lib/libcholmod.so,/home/salzman/local/i4_gcc/lib/libspqr.so,/home/salzman/local/i4_gcc/lib/libcolamd.so,/home/salzman/local/i4_gcc/lib/libccolamd.so,/home/s > alzman/local/i4_gcc/lib/libcamd.so,/home/salzman/local/i4_gcc/lib/libamd.so,/home/salzman/local/i4_gcc/lib/libmetis.so]" > --download-suitesparse=no --with-python-exec=python3.12 --have-numpy > =1 ---with-petsc4py=1 ---with-petsc4py-test-np=4 ---with-mpi4py=1 > --prefix=/home/salzman/local/i4_gcc/real_arithmetic COPTFLAGS="-O3 -g " > CXXOPTFLAGS="-O3 -g " FOPTFLAGS="-O3 -g " > [0]PETSC ERROR: #1 MatSetValues_SeqAIJ() at > /home/salzman/devel/PETSc/petsc/src/mat/impls/aij/seq/aij.c:426 > [0]PETSC ERROR: #2 MatSetValues() at > /home/salzman/devel/PETSc/petsc/src/mat/interface/matrix.c:1543 > [0]PETSC ERROR: #3 MatSetSeqMats_MPIAIJ() at > /home/salzman/devel/PETSc/petsc/src/mat/impls/aij/mpi/mpiov.c:2965 > [0]PETSC ERROR: #4 MatCreateSubMatricesMPI_MPIXAIJ() at > /home/salzman/devel/PETSc/petsc/src/mat/impls/aij/mpi/mpiov.c:3163 > [0]PETSC ERROR: #5 MatCreateSubMatricesMPI_MPIAIJ() at > /home/salzman/devel/PETSc/petsc/src/mat/impls/aij/mpi/mpiov.c:3196 > [0]PETSC ERROR: #6 MatCreateSubMatricesMPI() at > /home/salzman/devel/PETSc/petsc/src/mat/interface/matrix.c:7293 > [0]PETSC ERROR: #7 main() at subnb.c:181 > [0]PETSC ERROR: No PETSc Option Table entries > [0]PETSC ERROR: ----------------End of Error Message -------send entire > error message to petsc-maint at mcs.anl.gov---------- > -------------------------------------------------------------------------- > > This message comes from executing the attached test (I simplified the test > by removing the block size from the matrix used for extraction, compared to > the July test). In proc_xx_output.txt, you will find the output from the > code execution with the -ok option (i.e. irow/idxr and icol/idxc are the > same, i.e. a square sub-block for colour 0 distributed across the first two > processes). > > Has expected in this case we obtain the 0,3,6,9 sub-block terms, which are > distributed across processes 0 and 1 (two rows per proc). > > When asking for rectangular sub-block (i.e. with no option) it crash with > column to large on process 0: 4 col max 3 ??? I ask for 4 rows and 2 > columns in this process ??? > > Otherwise, I mention the dense aspect of the matrix in ex183.c, because, > in this case, no matter what selection is requested, all terms are > non-null. If there is an issue with the way the selection is coded in the > user program, I think it will be masked thanks to the full graph > representation. However, this may not be the case ? I should test it. > > I'll take a look at ex23.c. > > Thanks, > > A.S. > > > > Le 25/08/2025 ? 17:55, Mark Adams a ?crit : > > Ah, OK, never say never. > > MatCreateSubMatrices seems to support creating a new matrix with the > communicator of the IS. > It just needs to read from the input matrix and does not use it for > communication, so it can do that. > > As far as rectangular matrices, there is no reason not to support that > (the row IS and column IS can be distinct). > Can you send the whole error message? > There may not be a test that does this, but src/mat/tests/ex23.c looks > like it may be a rectangular matrix output. > > And, it should not matter if the input matrix has a 100% full sparse > matrix. It is still MatAIJ. > The semantics and API is the same for sparse or dense matrices. > > Thanks, > Mark > > On Mon, Aug 25, 2025 at 7:31?AM Alexis SALZMAN < > alexis.salzman at ec-nantes.fr> wrote: > >> Hi, >> >> Thanks for your answer, Mark. Perhaps MatCreateSubMatricesMPI is the only >> PETSc function that acts on a sub-communicator ? I'm not sure ? but it's >> clear that there's no ambiguity on that point. The first line of the >> documentation for that function states that it 'may live on subcomms'. This >> is confirmed by the 'src/mat/tests/ex183.c' test case. I used this test >> case to understand the function, which helped me with my code and the >> example I provided in my initial post. Unfortunately, in this example, the >> matrix from which the sub-matrices are extracted is dense, even though it >> uses a sparse structure. This does not clarify how to define sub-matrices >> when extracting from a sparse distributed matrix. Since my initial post, I >> have discovered that having more columns than rows can also result in the >> same error message. >> >> So, my questions boil down to: >> >> Can MatCreateSubMatricesMPI extract rectangular matrices from a square >> distributed sparse matrix? >> >> If not, the fact that only square matrices can be extracted in this >> context should perhaps be mentioned in the documentation. >> >> If so, I would be very grateful for any assistance in defining an IS pair >> in this context. >> >> Regards >> >> A.S. >> Le 27/07/2025 ? 00:15, Mark Adams a ?crit : >> >> First, you can not mix communicators in PETSc calls in general (ever?), >> but this error looks like you might be asking for a row from the matrix >> that does not exist. >> You should start with a PETSc example code. Test it and modify it to suit >> your needs. >> >> Good luck, >> Mark >> >> On Fri, Jul 25, 2025 at 9:31?AM Alexis SALZMAN < >> alexis.salzman at ec-nantes.fr> wrote: >> >>> Hi, >>> >>> As I am relatively new to Petsc, I may have misunderstood how to use the >>> MatCreateSubMatricesMPI function. The attached code is tuned for three >>> processes and extracts one matrix for each colour of a subcommunicator >>> that has been created using the MPI_Comm_split function from an MPIAij >>> matrix. The following error message appears when the code is set to its >>> default configuration (i.e. when a rectangular matrix is extracted with >>> more rows than columns for colour 0): >>> >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: Argument out of range >>> [0]PETSC ERROR: Column too large: col 4 max 3 >>> [0]PETSC ERROR: See >>> https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!ZqH097BZ0G0O3WI7RWrwIKFNpyk0czSWEqfusAeTlgEygAffwpgBUzsLw1TIoGkjZ3mYG-NRQxxFoxU4y8EyY0ofiz9I43Qwe0w$ >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Release Version 3.22.2, unknown >>> >>> ... petsc git hash 2a89477b25f compiled on a dell i9 computer with Gcc >>> 14.3, mkl 2025.2, ..... >>> [0]PETSC ERROR: #1 MatSetValues_SeqAIJ() at >>> ...petsc/src/mat/impls/aij/seq/aij.c:426 >>> [0]PETSC ERROR: #2 MatSetValues() at >>> ...petsc/src/mat/interface/matrix.c:1543 >>> [0]PETSC ERROR: #3 MatSetSeqMats_MPIAIJ() at >>> .../petsc/src/mat/impls/aij/mpi/mpiov.c:2965 >>> [0]PETSC ERROR: #4 MatCreateSubMatricesMPI_MPIXAIJ() at >>> .../petsc/src/mat/impls/aij/mpi/mpiov.c:3163 >>> [0]PETSC ERROR: #5 MatCreateSubMatricesMPI_MPIAIJ() at >>> .../petsc/src/mat/impls/aij/mpi/mpiov.c:3196 >>> [0]PETSC ERROR: #6 MatCreateSubMatricesMPI() at >>> .../petsc/src/mat/interface/matrix.c:7293 >>> [0]PETSC ERROR: #7 main() at sub.c:169 >>> >>> When the '-ok' option is selected, the code extracts a square matrix for >>> colour 0, which runs smoothly in this case. Selecting the '-trans' >>> option swaps the row and column selection indices, providing a >>> transposed submatrix smoothly. For colour 1, which uses only one process >>> and is therefore sequential, rectangular extraction is OK regardless of >>> the shape. >>> >>> Is this dependency on the shape expected? Have I missed an important >>> tuning step somewhere? >>> >>> Thank you in advance for any clarification. >>> >>> Regards >>> >>> A.S. >>> >>> P.S.: I'm sorry, but as I'm leaving my office for the following weeks >>> this evening, I won't be very responsive during this period. >>> >>> >>> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bHTBIkomOzv2YXBZD7vNmmrb0Ijc6Xd4lvZJ15CTTDE3_ewzI1zbkHE8LP2mjO3qm6VWQ8yao8uuUIZe95ad$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From lzou at anl.gov Tue Aug 26 09:24:00 2025 From: lzou at anl.gov (Zou, Ling) Date: Tue, 26 Aug 2025 14:24:00 +0000 Subject: [petsc-users] Understanding 'snes' object In-Reply-To: <3D5FD262-12FC-4022-89A5-DF86DEE5C691@petsc.dev> References: <3D5FD262-12FC-4022-89A5-DF86DEE5C691@petsc.dev> Message-ID: Barry, thanks very much for the insight, and you are right! For the purpose of sharing information, in case anyone is interested, here are some details. The problem at hand is a *nearly* incompressible flow problem, such that part of the diagonal Jacobian term (d(rho)_dp) is very close to but not truly zero. I have noticed that Option-2 complained about zero pivot, and I can see that the diagonal Jacobian term is zero by outputting the Jacobian. Option-1 was Ok, and I can see that the diagonal Jacobian term is very small but non-zero. So you are right, they behave very similarly but produces a tiny bit of difference. Best, -Ling From: Barry Smith Date: Sunday, August 24, 2025 at 3:30?PM To: Zou, Ling Cc: Stefano Zampini , PETSc Subject: Re: [petsc-users] Understanding 'snes' object You should have very similar behavior, but not identical behavior. The Jacobian-vector product will produce slightly different values with the two approaches, which means that the computed residual norms displayed with -ksp_monitor -snes_monitor ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd You should have very similar behavior, but not identical behavior. The Jacobian-vector product will produce slightly different values with the two approaches, which means that the computed residual norms displayed with -ksp_monitor -snes_monitor etc will be somewhat different. With long runs that involve many nonlinear solves the differences in the norms computed will become nontrivial and you will start getting different iteration counts, but the solutions computed should remain very similar. Barry On Aug 23, 2025, at 10:03?AM, Zou, Ling via petsc-users wrote: Thank you both Jed and Stefano. Then, somehow, I may have introduced inconsistence in my code, because I expected they have exactly the same behavior. Best, -Ling From: Stefano Zampini > Date: Friday, August 22, 2025 at 4:42?PM To: Zou, Ling > Cc: PETSc > Subject: Re: [petsc-users] Understanding 'snes' object Il giorno sab 23 ago 2025 alle ore 00:?25 Zou, Ling via petsc-users ha scritto: Hi all, Can someone please help me understand the output of the following two outputs using ?-snes_view?? Option 1: SNES Object: 1 ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Il giorno sab 23 ago 2025 alle ore 00:25 Zou, Ling via petsc-users > ha scritto: Hi all, Can someone please help me understand the output of the following two outputs using ?-snes_view?? Option 1: SNES Object: 1 MPI process type: newtonls maximum iterations=15, maximum function evaluations=10000 tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 total number of linear solver iterations=5 total number of function evaluations=14 norm schedule ALWAYS Jacobian is built using finite differences with coloring Option 2: SNES Object: 1 MPI process type: newtonls maximum iterations=50, maximum function evaluations=10000 tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 total number of linear solver iterations=8 total number of function evaluations=17 norm schedule ALWAYS Jacobian is applied matrix-free with differencing Preconditioning Jacobian is built using finite differences with coloring Does it mean that * Option 1 uses Newton?s method, such that J du = b is solved with J being explicitly constructed? yes * * Option 2 uses matrix free method, such that J du = b is solved with J v ~ (F(u + dv) ? F(u)) / h when needed, e.g., in a Krylov method. yes, and the preconditioner is constructed using an explicitly constructed Jacobian matrix * Thanks, -Ling -- Stefano -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Tue Aug 26 09:35:51 2025 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 26 Aug 2025 10:35:51 -0400 Subject: [petsc-users] Question about Pseudo Time Stepping In-Reply-To: <85e1d3ad-6a32-444e-94dd-b5ebb6cba819@unibas.it> References: <85e1d3ad-6a32-444e-94dd-b5ebb6cba819@unibas.it> Message-ID: <5136EADB-E4F4-4A34-872D-EC8297244DA8@petsc.dev> What code are you running? So we can run a reproducer. > On Aug 26, 2025, at 4:21?AM, Aldo Bonfiglioli wrote: > > Hi there, > > I am using -ts_type pseudo to find steady solutions of the Euler/NS PDEs. > > It looks like FormRHSFunction is called twice with the exactly same value of the dependent variable (u) > > before invoking TSRHSJacobianFn and then solving the linear system with KSP (u is then updated) > > >> 1 TS dt 0.141197 time 0.141197 >> Calling FormRHSFunction; ||u|| = 60.489405479528003 463.15635675129079 62.813532026019146 5.9789502719351155 >> Calling FormRHSFunction; ||u|| = 60.489405479528003 463.15635675129079 62.813532026019146 5.9789502719351155 >> Calling TSRHSJacobianFn with ||u|| = 60.489405479528003 463.15635675129079 62.813532026019146 5.9789502719351155 >> Linear solve converged due to CONVERGED_RTOL iterations 9 >> Calling FormRHSFunction; ||u|| = 60.489405476878737 463.15635675602186 62.813532031616965 5.9789502710519811 >> 2 TS dt 0.155917 time 0.297115 >> Calling FormRHSFunction; ||u|| = 60.489405476878737 463.15635675602186 62.813532031616965 5.9789502710519811 >> Calling FormRHSFunction; ||u|| = 60.489405476878737 463.15635675602186 62.813532031616965 5.9789502710519811 >> Calling TSRHSJacobianFn with ||u|| = 60.489405476878737 463.15635675602186 62.813532031616965 5.9789502710519811 >> > Why is it so? Am I possibly misusing (or missing) some options? > > My .petscrc file is attached. > > Thank you, > > Aldo > > -- > Dr. Aldo Bonfiglioli > Associate professor of Fluid Mechanics > Dipartimento di Ingegneria > Universita' della Basilicata > V.le dell'Ateneo Lucano, 10 85100 Potenza ITALY > tel:+39.0971.205203 fax:+39.0971.205215 > web: https://urldefense.us/v3/__http://docenti.unibas.it/site/home/docente.html?m=002423__;!!G_uCfscf7eWS!dbaOWMZSge8uqvR9TNizV3K7TdZt-gKWocxl5mk3p-3muUHHd2_c4QfE4CFLhjnmSJhr_VhecH-vFUZs6mKLlSY$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From popov at uni-mainz.de Tue Aug 26 11:20:22 2025 From: popov at uni-mainz.de (Anton Popov) Date: Tue, 26 Aug 2025 18:20:22 +0200 Subject: [petsc-users] subdomain solver options Message-ID: <210a91d0-9d0e-44a5-8827-09f74861450d@uni-mainz.de> Hi all, I have a strange behavior setting the solver options for asm and bjacobi blocks. Irrespective of what I set with -sub_ksp_ -sub_pc_ it just selects the default preonly and ilu options. This behavior does not change no matter how many local blocks per cpu I try to use, for example with the default one block/cpu it does not work as well. Is there anything wrong? Best, Anton From popov at uni-mainz.de Tue Aug 26 11:36:18 2025 From: popov at uni-mainz.de (Anton Popov) Date: Tue, 26 Aug 2025 18:36:18 +0200 Subject: [petsc-users] subdomain solver options In-Reply-To: <210a91d0-9d0e-44a5-8827-09f74861450d@uni-mainz.de> References: <210a91d0-9d0e-44a5-8827-09f74861450d@uni-mainz.de> Message-ID: There is an important addition, I use asm and bjacobi as smoothers on multigrid levels. So I probably need to set the options with -mg_levels_sub_, or -mg_levels_3_sub_(if they are different on different levels), instead of just -sub_. Correct? Anton On 26.08.25 18:20, Anton Popov wrote: > Hi all, > > I have a strange behavior setting the solver options for asm and > bjacobi blocks. > > Irrespective of what I set with -sub_ksp_ -sub_pc_ it just selects the > default preonly and ilu options. > > This behavior does not change no matter how many local blocks per cpu > I try to use, for example with the default one block/cpu it does not > work as well. > > Is there anything wrong? > > Best, > > Anton > From mfadams at lbl.gov Tue Aug 26 13:10:36 2025 From: mfadams at lbl.gov (Mark Adams) Date: Tue, 26 Aug 2025 14:10:36 -0400 Subject: [petsc-users] subdomain solver options In-Reply-To: References: <210a91d0-9d0e-44a5-8827-09f74861450d@uni-mainz.de> Message-ID: Run with -options_left and -ksp_view You will see options left, not used, printed at the end and in the KSP view output you will see the option to use like: PC Object: (mg_levels_1_) 128 MPI processes type: redundant On Tue, Aug 26, 2025 at 12:36?PM Anton Popov wrote: > There is an important addition, I use asm and bjacobi as smoothers on > multigrid levels. > > So I probably need to set the options with -mg_levels_sub_, or > -mg_levels_3_sub_(if they are different on different levels), instead of > just -sub_. > > Correct? > > Anton > > > > On 26.08.25 18:20, Anton Popov wrote: > > Hi all, > > > > I have a strange behavior setting the solver options for asm and > > bjacobi blocks. > > > > Irrespective of what I set with -sub_ksp_ -sub_pc_ it just selects the > > default preonly and ilu options. > > > > This behavior does not change no matter how many local blocks per cpu > > I try to use, for example with the default one block/cpu it does not > > work as well. > > > > Is there anything wrong? > > > > Best, > > > > Anton > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Aug 26 15:34:32 2025 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 26 Aug 2025 16:34:32 -0400 Subject: [petsc-users] subdomain solver options In-Reply-To: References: <210a91d0-9d0e-44a5-8827-09f74861450d@uni-mainz.de> Message-ID: On Tue, Aug 26, 2025 at 12:45?PM Anton Popov wrote: > There is an important addition, I use asm and bjacobi as smoothers on > multigrid levels. > > So I probably need to set the options with -mg_levels_sub_, or > -mg_levels_3_sub_(if they are different on different levels), instead of > just -sub_. > > Correct? > Yes, and as Mark says, you can see the prefix if you View the solver. Thanks, Matt > Anton > > > > On 26.08.25 18:20, Anton Popov wrote: > > Hi all, > > > > I have a strange behavior setting the solver options for asm and > > bjacobi blocks. > > > > Irrespective of what I set with -sub_ksp_ -sub_pc_ it just selects the > > default preonly and ilu options. > > > > This behavior does not change no matter how many local blocks per cpu > > I try to use, for example with the default one block/cpu it does not > > work as well. > > > > Is there anything wrong? > > > > Best, > > > > Anton > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YJ62X-g3HWHOyVCSc8dsKNAO6uNehYJ2i4JwXTMP6lr_dBNvcvnnAgqOWC_c6IGw3_mqX47OtDZpkBsDOJyR$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ali.ali_ahmad at utt.fr Tue Aug 26 20:56:35 2025 From: ali.ali_ahmad at utt.fr (Ali ALI AHMAD) Date: Wed, 27 Aug 2025 03:56:35 +0200 (CEST) Subject: [petsc-users] [petsc-maint] Question about NewtonTR / Inexact Newton with piecewise continuous functions In-Reply-To: References: <958788052.14717071.1756080960168.JavaMail.zimbra@utt.fr> <683850940.15267050.1756159332237.JavaMail.zimbra@utt.fr> Message-ID: <1881636397.15804938.1756259795051.JavaMail.zimbra@utt.fr> Thanks again for your help. * SNESVI is now running and I do see active constraints in some iterations. With a feasible initial guess, the constraints are respected and the linear solve converges each step. However, during the line search PETSc evaluates trial points that violate the (nonlinear) physical constraints (eg. p<0) leading to Inf/NaN and "line search ? diverged", the line search should search within the feasible/physical region (the reduced space) defined by the bounds ! For example, SNES VI Function norm 3.59545e+07 Active lower constraints 0/0 upper constraints 0/0 Percent of total 0. Percent of bounded 0. Linear solve converged due to CONVERGED_RTOL iterations 1 [Non physique] pression n?gative ou NaN ? la cellule 695 : p = -3.87338 . . . 1 SNES VI Function norm 2.62898e+07 Active lower constraints 1/6 upper constraints 0/0 Percent of total 0.000108015 Percent of bounded 0. Linear solve converged due to CONVERGED_RTOL iterations 1 [Non physique] pression n?gative ou NaN ? la cellule 1099 : p = -17.4556 etc * I expected the line search to stay in the feasible set. Is this behavior expected when bounds are computed from the current iterate via " SNESVISetComputeVariableBounds " ? If so, what?s the recommended way to keep trial points feasible? * I?m currently using " vinewtonrsls with -snes_linesearch_type bt " . For now " vinewtonssls" is not working ! * For clarity: my feasibility constraint is nonlinear. Best regards, Ali ALI AHMAD De: "Barry Smith" ?: "Ali ALI AHMAD" Cc: "petsc-users" Envoy?: Mardi 26 Ao?t 2025 03:47:39 Objet: Re: [petsc-maint] Question about NewtonTR / Inexact Newton with piecewise continuous functions You can put a debugger breakpoint in the SNESVI code that checks the bounds to see why it is not finding violations. This is done in SNESSolve_VINEWTONRSLS() which calls SNESVIGetActiveSetIS(). So check th variables in SNESVIGetActiveSetIS. Note in even simple debuggers like gdb or lldb you can do call VecView(x,0) call VecView(xl,0) etc to examine the vectors quickly and verify if the values make sense. Barry On Aug 25, 2025, at 6:02 PM, Ali ALI AHMAD wrote: Oh no, I was mistaken: I meant that the objective function is continuous, but its derivative is not. That is why I am using snesvinewtonssls. * However, the problem is that the nonlinear constraints I define in " ComputeBounds " are always inactive. As you can see in the output: 0 SNES Function norm 1.562978975444e+07 0 SNES VI Function norm 1.56298e+07 Active lower constraints 0/0 upper constraints 0/0 Percent of total 0. Percent of bounded 0. 0 KSP Residual norm 1.976350248286e+04 1 KSP Residual norm 1.483291315197e+04 2 KSP Residual norm 1.358944289069e+04 3 KSP Residual norm 1.276791956295e+04 4 KSP Residual norm 1.275078123378e+04 5 KSP Residual norm 1.253064296351e+04 6 KSP Residual norm 1.232954788760e+04 7 KSP Residual norm 1.232684700773e+04 8 KSP Residual norm 1.165265510468e+04 9 KSP Residual norm 1.159313964889e+04 10 KSP Residual norm 1.098241228864e+04 11 KSP Residual norm 1.063610434231e+04 12 KSP Residual norm 8.556981786904e+03 13 KSP Residual norm 7.792752698989e+03 14 KSP Residual norm 6.260039598596e+03 15 KSP Residual norm 3.025534878795e+03 16 KSP Residual norm 8.343990389000e+02 17 KSP Residual norm 2.591011944960e+02 18 KSP Residual norm 9.212497850148e+01 19 KSP Residual norm 4.137422005070e+01 20 KSP Residual norm 1.345189115925e+01 21 KSP Residual norm 2.795423327875e+00 22 KSP Residual norm 5.068536677416e-01 23 KSP Residual norm 1.167594367615e-01 Linear solve converged due to CONVERGED_RTOL iterations 23 [Non physique] pression n?gative ou NaN ? la cellule 61 : p = -1.66778e+14 Line search: objective function at lambdas = 1. is Inf or Nan, cutting lambda [Non physique] pression n?gative ou NaN ? la cellule 61 : p = -4.06293e+13 Line search: objective function at lambdas = 0.5 is Inf or Nan, cutting lambda [Non physique] pression n?gative ou NaN ? la cellule 61 : p = -9.63498e+12 Line search: objective function at lambdas = 0.25 is Inf or Nan, cutting lambda [Non physique] pression n?gative ou NaN ? la cellule 61 : p = -2.15792e+12 Line search: objective function at lambdas = 0.125 is Inf or Nan, cutting lambda [Non physique] pression n?gative ou NaN ? la cellule 61 : p = -1568.11 Line search: objective function at lambdas = 0.0625 is Inf or Nan, cutting lambda [Non physique] pression n?gative ou NaN ? la cellule 788 : p = -11.2882 Line search: objective function at lambdas = 0.03125 is Inf or Nan, cutting lambda Line search: Using full step: fnorm 1.562978975444e+07 gnorm 1.519904837982e+07 So none of my lower or upper bounds are active during the iterations. * Here is my ComputeBounds function, where I try to enforce positivity of density and pressure: PetscErrorCode ComputeBounds(SNES snes, Vec xl, Vec xu) { PetscFunctionBeginUser; // 1) Get user context (must have been set via SNESSetApplicationContext) void *ctx = NULL; PetscCall(SNESGetApplicationContext(snes, &ctx)); euler *e = (euler*)ctx; // Minimal positive thresholds const double eps_rho = 1e-8; // minimal density const double pmin = 1e-8; // minimal pressure // 2) Access current solution X (used here to build *dynamic* bounds) Vec X; const PetscScalar *x = NULL; PetscCall(SNESGetSolution(snes, &X)); PetscCall(VecGetArrayRead(X, &x)); // 3) Fill lower bounds 'xl' PetscScalar *l = NULL; PetscCall(VecGetArray(xl, &l)); for (int K = 0; K < e->N_cells; ++K) { const int i_rho = Ncomp*K + 0; // density index const int i_m = Ncomp*K + 1; // momentum (1D). In 2D use mx,my and adjust kinetic term. const int i_rhoE = Ncomp*K + 2; // total energy index const double rho = (double)x[i_rho]; const double m = (double)x[i_m]; // Use a safe density to avoid division by ~0 in kinetic energy const double rho_safe = PetscMax(rho, eps_rho); // Kinetic energy per unit volume: |m|^2/(2*rho) const double kin_vol = 0.5 * (m*m) / rho_safe; // Enforce p >= pmin via total energy lower bound: // rhoE >= kinetic + p/(gamma-1) const double rhoEmin = kin_vol + pmin/(_gamma - 1.0); // Set lower bounds l[i_rho] = eps_rho; // rho >= eps_rho l[i_m] = PETSC_NINFINITY; // no lower bound on momentum l[i_rhoE] = rhoEmin; // rhoE >= rhoEmin (to keep p >= pmin) } // 4) Restore arrays PetscCall(VecRestoreArrayRead(X, &x)); PetscCall(VecRestoreArray(xl, &l)); // 5) Set upper bounds to +infinity (no upper constraints) PetscCall(VecSet(xu, PETSC_INFINITY)); PetscFunctionReturn(PETSC_SUCCESS); } But when I run the solver, PETSc always reports Active lower constraints 0/0 . Why are my constraints not activated? Best regards, ALI ALI AHMAD De: "Matthew Knepley" ?: "Ali ALI AHMAD" Cc: "petsc-maint" , "petsc-users" , "Barry Smith" Envoy?: Lundi 25 Ao?t 2025 03:42:36 Objet: Re: [petsc-maint] Question about NewtonTR / Inexact Newton with piecewise continuous functions On Sun, Aug 24, 2025 at 8:16 PM Ali ALI AHMAD < [ mailto:ali.ali_ahmad at utt.fr | ali.ali_ahmad at utt.fr ] > wrote: BQ_BEGIN Hello, I am currently working with PETSc to solve a nonlinear system. My function is piecewise continuous (so it can be discontinuous), and its derivative (Jacobian) is also piecewise continuous. I am not aware of any convergence framework for piecewise continuous functions. They are not technically computable, meaning you could have to compute for a very very long time, and still not get an accurate output (since you have the jump). This also means that the derivative is not computable, and you can have arbitrarily large errors near the discontinuity. I don't see how you could prove convergence here, but maybe someone else knows something I don't. Thanks, Matt BQ_BEGIN I implemented the residual and also compute the Jacobian analytically. When the discontinuities are small, both Inexact Newton and NewtonTR (with scaled Newton direction) converge without problem. However, for test cases with larger discontinuities, sometimes the solver converges, but in other cases it fails to converge. I initially tried , but I cannot use them in my case, because my problem involves nonlinear constraints. So my question is: how does PETSc handle such situations internally (piecewise continuous objective/residual functions)? And is there a recommended strategy within PETSc to deal with nonlinear solvers when and are discontinuous? I would like to continue working with PETSc and I am looking for a robust method to treat this type of problem. Thank you very much for your help and suggestions. Best regards, ALI ALI AHMAD BQ_END -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener [ https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!ZjM8gfx4v5nBjyk-nU5i7HZ1B5UUQuoz_BakQWoIhuxmFhO8c_Mmt496I-YoSv6ecHQpp489fhIP64CVPnmqli6q0mvmLw$ | https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!ZjM8gfx4v5nBjyk-nU5i7HZ1B5UUQuoz_BakQWoIhuxmFhO8c_Mmt496I-YoSv6ecHQpp489fhIP64CVPnmqli5vQ01ZfQ$ ] BQ_END -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Tue Aug 26 21:16:48 2025 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 26 Aug 2025 22:16:48 -0400 Subject: [petsc-users] [petsc-maint] Question about NewtonTR / Inexact Newton with piecewise continuous functions In-Reply-To: <1881636397.15804938.1756259795051.JavaMail.zimbra@utt.fr> References: <958788052.14717071.1756080960168.JavaMail.zimbra@utt.fr> <683850940.15267050.1756159332237.JavaMail.zimbra@utt.fr> <1881636397.15804938.1756259795051.JavaMail.zimbra@utt.fr> Message-ID: <2D9B3DB1-B752-4FCE-A021-FDCBCE62DB3A@petsc.dev> > On Aug 26, 2025, at 9:56?PM, Ali ALI AHMAD wrote: > > Thanks again for your help. > > * SNESVI is now running and I do see active constraints in some iterations. With a feasible initial guess, the constraints are respected and the linear solve converges each step. However, during the line search PETSc evaluates trial points that violate the (nonlinear) physical constraints (eg. p<0) leading to Inf/NaN and "line search ? diverged", the line search should search within the feasible/physical region (the reduced space) defined by the bounds ! Yes this is how it is suppose to work for __bounded__ inequality constraints. > > For example, > > SNES VI Function norm 3.59545e+07 Active lower constraints 0/0 upper constraints 0/0 Percent of total 0. Percent of bounded 0. > Linear solve converged due to CONVERGED_RTOL iterations 1 > [Non physique] pression n?gative ou NaN ? la cellule 695 : p = -3.87338 > > . > . > . > > 1 SNES VI Function norm 2.62898e+07 Active lower constraints 1/6 upper constraints 0/0 Percent of total 0.000108015 Percent of bounded 0. > Linear solve converged due to CONVERGED_RTOL iterations 1 > [Non physique] pression n?gative ou NaN ? la cellule 1099 : p = -17.4556 > > etc > > > * I expected the line search to stay in the feasible set. Is this behavior expected when bounds are computed from the current iterate via " SNESVISetComputeVariableBounds " ? If so, what?s the recommended way to keep trial points feasible? ' Are you changing the bounds on each SNES iteration? Are the bound constraints on the actual variables in your solution or are they on some other values you compute from your actual solution variables? If so, I don't think that will work. I think more sophisticated algorithms are needed for more general inequality constraints. I know nothing on that topic and suggest you do a little research on appropriate algorithmic approaches and let us know what you discover. > > * I?m currently using " vinewtonrsls with -snes_linesearch_type bt ". For now "vinewtonssls" is not working ! > > * For clarity: my feasibility constraint is nonlinear. > > Best regards, > Ali ALI AHMAD > > De: "Barry Smith" > ?: "Ali ALI AHMAD" > Cc: "petsc-users" > Envoy?: Mardi 26 Ao?t 2025 03:47:39 > Objet: Re: [petsc-maint] Question about NewtonTR / Inexact Newton with piecewise continuous functions > > You can put a debugger breakpoint in the SNESVI code that checks the bounds to see why it is not finding violations. This is done in SNESSolve_VINEWTONRSLS() which calls SNESVIGetActiveSetIS(). So check th variables in SNESVIGetActiveSetIS. > Note in even simple debuggers like gdb or lldb you can do > > call VecView(x,0) > call VecView(xl,0) > etc > > to examine the vectors quickly and verify if the values make sense. > > Barry > > > > On Aug 25, 2025, at 6:02?PM, Ali ALI AHMAD wrote: > > Oh no, I was mistaken: I meant that the objective function is continuous, but its derivative is not. That is why I am using snesvinewtonssls. > > * However, the problem is that the nonlinear constraints I define in " ComputeBounds " are always inactive. As you can see in the output: > > 0 SNES Function norm 1.562978975444e+07 > 0 SNES VI Function norm 1.56298e+07 Active lower constraints 0/0 upper constraints 0/0 Percent of total 0. Percent of bounded 0. > 0 KSP Residual norm 1.976350248286e+04 > 1 KSP Residual norm 1.483291315197e+04 > 2 KSP Residual norm 1.358944289069e+04 > 3 KSP Residual norm 1.276791956295e+04 > 4 KSP Residual norm 1.275078123378e+04 > 5 KSP Residual norm 1.253064296351e+04 > 6 KSP Residual norm 1.232954788760e+04 > 7 KSP Residual norm 1.232684700773e+04 > 8 KSP Residual norm 1.165265510468e+04 > 9 KSP Residual norm 1.159313964889e+04 > 10 KSP Residual norm 1.098241228864e+04 > 11 KSP Residual norm 1.063610434231e+04 > 12 KSP Residual norm 8.556981786904e+03 > 13 KSP Residual norm 7.792752698989e+03 > 14 KSP Residual norm 6.260039598596e+03 > 15 KSP Residual norm 3.025534878795e+03 > 16 KSP Residual norm 8.343990389000e+02 > 17 KSP Residual norm 2.591011944960e+02 > 18 KSP Residual norm 9.212497850148e+01 > 19 KSP Residual norm 4.137422005070e+01 > 20 KSP Residual norm 1.345189115925e+01 > 21 KSP Residual norm 2.795423327875e+00 > 22 KSP Residual norm 5.068536677416e-01 > 23 KSP Residual norm 1.167594367615e-01 > Linear solve converged due to CONVERGED_RTOL iterations 23 > [Non physique] pression n?gative ou NaN ? la cellule 61 : p = -1.66778e+14 > Line search: objective function at lambdas = 1. is Inf or Nan, cutting lambda > [Non physique] pression n?gative ou NaN ? la cellule 61 : p = -4.06293e+13 > Line search: objective function at lambdas = 0.5 is Inf or Nan, cutting lambda > [Non physique] pression n?gative ou NaN ? la cellule 61 : p = -9.63498e+12 > Line search: objective function at lambdas = 0.25 is Inf or Nan, cutting lambda > [Non physique] pression n?gative ou NaN ? la cellule 61 : p = -2.15792e+12 > Line search: objective function at lambdas = 0.125 is Inf or Nan, cutting lambda > [Non physique] pression n?gative ou NaN ? la cellule 61 : p = -1568.11 > Line search: objective function at lambdas = 0.0625 is Inf or Nan, cutting lambda > [Non physique] pression n?gative ou NaN ? la cellule 788 : p = -11.2882 > Line search: objective function at lambdas = 0.03125 is Inf or Nan, cutting lambda > Line search: Using full step: fnorm 1.562978975444e+07 gnorm 1.519904837982e+07 > > So none of my lower or upper bounds are active during the iterations. > > * Here is my ComputeBounds function, where I try to enforce positivity of density and pressure: > > PetscErrorCode ComputeBounds(SNES snes, Vec xl, Vec xu) { > PetscFunctionBeginUser; > > // 1) Get user context (must have been set via SNESSetApplicationContext) > void *ctx = NULL; > PetscCall(SNESGetApplicationContext(snes, &ctx)); > euler *e = (euler*)ctx; > > // Minimal positive thresholds > const double eps_rho = 1e-8; // minimal density > const double pmin = 1e-8; // minimal pressure > > // 2) Access current solution X (used here to build *dynamic* bounds) > Vec X; > const PetscScalar *x = NULL; > PetscCall(SNESGetSolution(snes, &X)); > PetscCall(VecGetArrayRead(X, &x)); > > // 3) Fill lower bounds 'xl' > PetscScalar *l = NULL; > PetscCall(VecGetArray(xl, &l)); > > for (int K = 0; K < e->N_cells; ++K) { > const int i_rho = Ncomp*K + 0; // density index > const int i_m = Ncomp*K + 1; // momentum (1D). In 2D use mx,my and adjust kinetic term. > const int i_rhoE = Ncomp*K + 2; // total energy index > > const double rho = (double)x[i_rho]; > const double m = (double)x[i_m]; > > // Use a safe density to avoid division by ~0 in kinetic energy > const double rho_safe = PetscMax(rho, eps_rho); > > // Kinetic energy per unit volume: |m|^2/(2*rho) > const double kin_vol = 0.5 * (m*m) / rho_safe; > > // Enforce p >= pmin via total energy lower bound: > // rhoE >= kinetic + p/(gamma-1) > const double rhoEmin = kin_vol + pmin/(_gamma - 1.0); > > // Set lower bounds > l[i_rho] = eps_rho; // rho >= eps_rho > l[i_m] = PETSC_NINFINITY; // no lower bound on momentum > l[i_rhoE] = rhoEmin; // rhoE >= rhoEmin (to keep p >= pmin) > } > > // 4) Restore arrays > PetscCall(VecRestoreArrayRead(X, &x)); > PetscCall(VecRestoreArray(xl, &l)); > > // 5) Set upper bounds to +infinity (no upper constraints) > PetscCall(VecSet(xu, PETSC_INFINITY)); > > PetscFunctionReturn(PETSC_SUCCESS); > } > > But when I run the solver, PETSc always reports Active lower constraints 0/0. Why are my constraints not activated? > > Best regards, > ALI ALI AHMAD > > De: "Matthew Knepley" > ?: "Ali ALI AHMAD" > Cc: "petsc-maint" , "petsc-users" , "Barry Smith" > Envoy?: Lundi 25 Ao?t 2025 03:42:36 > Objet: Re: [petsc-maint] Question about NewtonTR / Inexact Newton with piecewise continuous functions > > On Sun, Aug 24, 2025 at 8:16?PM Ali ALI AHMAD > wrote: >> Hello, >> >> I am currently working with PETSc to solve a nonlinear system. My function is piecewise continuous (so it can be discontinuous), and its derivative (Jacobian) is also piecewise continuous. > > I am not aware of any convergence framework for piecewise continuous functions. They are not technically computable, meaning you could have to compute for a very very long time, and still not get an accurate output (since you have the jump). This also means that the derivative is not computable, and you can have arbitrarily large errors near the discontinuity. I don't see how you could prove convergence here, but maybe someone else knows something I don't. > > Thanks, > > Matt > >> I implemented the residual and also compute the Jacobian analytically. >> >> When the discontinuities are small, both Inexact Newton and NewtonTR (with scaled Newton direction) converge without problem. >> >> However, for test cases with larger discontinuities, sometimes the solver converges, but in other cases it fails to converge. >> >> I initially tried , but I cannot use them in my case, because my problem involves nonlinear constraints. >> >> So my question is: how does PETSc handle such situations internally (piecewise continuous objective/residual functions)? >> And is there a recommended strategy within PETSc to deal with nonlinear solvers when and are discontinuous? >> >> I would like to continue working with PETSc and I am looking for a robust method to treat this type of problem. >> >> Thank you very much for your help and suggestions. >> >> Best regards, >> ALI ALI AHMAD > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YaTmdAEobm7MGKyVHIM0JqC-5CY3xeZpfBEXMX5rp8-ue1P1oJJjPmKRHZVVSNP64ZZKp2XvSB0qFVp2xNKDOls$ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From popov at uni-mainz.de Wed Aug 27 02:39:42 2025 From: popov at uni-mainz.de (Anton Popov) Date: Wed, 27 Aug 2025 09:39:42 +0200 Subject: [petsc-users] subdomain solver options In-Reply-To: References: <210a91d0-9d0e-44a5-8827-09f74861450d@uni-mainz.de> Message-ID: <39a89c20-0927-48f3-9ba1-caee3ba3e31b@uni-mainz.de> Thanks Matt, it works! Anton On 26.08.25 22:34, Matthew Knepley wrote: > On Tue, Aug 26, 2025 at 12:45?PM Anton Popov wrote: > > There is an important addition, I use asm and bjacobi as smoothers on > multigrid levels. > > So I probably need to set the options with -mg_levels_sub_, or > -mg_levels_3_sub_(if they are different on different levels), > instead of > just -sub_. > > Correct? > > > Yes, and as Mark says, you can see the prefix if you View the solver. > > ? Thanks, > > ? ? ?Matt > > Anton > > > > On 26.08.25 18:20, Anton Popov wrote: > > Hi all, > > > > I have a strange behavior setting the solver options for asm and > > bjacobi blocks. > > > > Irrespective of what I set with -sub_ksp_ -sub_pc_ it just > selects the > > default preonly and ilu options. > > > > This behavior does not change no matter how many local blocks > per cpu > > I try to use, for example with the default one block/cpu it does > not > > work as well. > > > > Is there anything wrong? > > > > Best, > > > > Anton > > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!dduznZr2wz7YJHJJczD2Qkrwr8f7vwu56e_Z3HlM5b0coSWG33ZtoI9QE275Yki2aVtsa8vtSODfzsxo-uZkERvq$ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexis.salzman at ec-nantes.fr Wed Aug 27 04:14:54 2025 From: alexis.salzman at ec-nantes.fr (Alexis SALZMAN) Date: Wed, 27 Aug 2025 11:14:54 +0200 Subject: [petsc-users] MatCreateSubMatricesMPI strange behavior In-Reply-To: References: <9d648a1d-72d2-44d0-8a3b-a9d64b01604f@ec-nantes.fr> <113ece7d-25cf-4b16-b29b-bdd1ad2a004a@ec-nantes.fr> <82d06db6-ab86-49b8-9e26-000745ae2134@ec-nantes.fr> <8A46BABF-E664-4923-87DF-A407B29163A8@joliv.et> <37AC2024-4D4C-41BD-A543-07221FDFB2C3@joliv.et> Message-ID: <3b6ce54f-1c62-4494-b162-eec2f90ee31f@ec-nantes.fr> Hello Pierre, After applying your patch to my local version of PETSc, all of the cases that previously caused the provided test to fail are now running smoothly. In a more complex context (with more processes and colors in my application), no errors are found and the sub-matrices look OK. Thank you very much for your time. This debugged function will greatly simplify the development of my application. Best regards A.S. Le 26/08/2025 ? 15:46, Matthew Knepley a ?crit?: > On Tue, Aug 26, 2025 at 9:42?AM Pierre Jolivet wrote: > > It?s indeed very suspicious (to me) that we are using rmap to > change a column index. > Switching to cmap gets your code running, but I?ll need to see if > this triggers regressions. > > > That looks right to me. I am sure this has only been tested for?GASM, > which would be symmetric. > > ? Thanks, > > ? ? ?Matt > > Thanks for the report, > Pierre > > diff --git a/src/mat/impls/aij/mpi/mpiov.c > b/src/mat/impls/aij/mpi/mpiov.c > index d1037d7d817..051981ebe9a 100644 > --- a/src/mat/impls/aij/mpi/mpiov.c > +++ b/src/mat/impls/aij/mpi/mpiov.c > @@ -2948,3 +2948,3 @@ PetscErrorCode MatSetSeqMats_MPIAIJ(Mat C, > IS rowemb, IS dcolemb, IS ocolemb, Ma > - ? ?PetscCall(PetscLayoutGetRange(C->rmap, &rstart, &rend)); > + ? ?PetscCall(PetscLayoutGetRange(C->cmap, &rstart, &rend)); > ? ? ?shift ? ? ?= rend - rstart; > > $ cat proc_0_output.txt > rstart 0 rend 4 > Mat Object: 3 MPI processes > ? type: mpiaij > ? row 0: ? (0, 101.) ? ?(3, 104.) ? ?(6, 107.) ?(9, 110.) > ? row 1: ? (2, 203.) ? ?(5, 206.) ? ?(8, 209.) ?(11, 212.) > ? row 2: ? (1, 302.) ? ?(4, 305.) ? ?(7, 308.) ?(10, 311.) > ? row 3: ? (0, 401.) ? ?(3, 404.) ? ?(6, 407.) ?(9, 410.) > ? row 4: ? (2, 503.) ? ?(5, 506.) ? ?(8, 509.) ?(11, 512.) > ? row 5: ? (1, 602.) ? ?(4, 605.) ? ?(7, 608.) ?(10, 611.) > ? row 6: ? (0, 701.) ? ?(3, 704.) ? ?(6, 707.) ?(9, 710.) > ? row 7: ? (2, 803.) ? ?(5, 806.) ? ?(8, 809.) ?(11, 812.) > ? row 8: ? (1, 902.) ? ?(4, 905.) ? ?(7, 908.) ?(10, 911.) > ? row 9: ? (0, 1001.) ? ?(3, 1004.) ? ?(6, 1007.) ?(9, 1010.) > ? row 10: ? (2, 1103.) ? ?(5, 1106.) ? ?(8, 1109.) ? ?(11, 1112.) > ? row 11: ? (1, 1202.) ? ?(4, 1205.) ? ?(7, 1208.) ? ?(10, 1211.) > idxr proc > IS Object: 2 MPI processes > ? type: general > [0] Number of indices in set 4 > [0] 0 0 > [0] 1 1 > [0] 2 2 > [0] 3 3 > [1] Number of indices in set 4 > [1] 0 4 > [1] 1 5 > [1] 2 6 > [1] 3 7 > idxc proc > IS Object: 2 MPI processes > ? type: general > [0] Number of indices in set 2 > [0] 0 0 > [0] 1 1 > [1] Number of indices in set 2 > [1] 0 6 > [1] 1 7 > Mat Object: 2 MPI processes > ? type: mpiaij > ? row 0: ? (0, 101.) ? ?(2, 107.) > ? row 1: > ? row 2: ? (1, 302.) ? ?(3, 308.) > ? row 3: ? (0, 401.) ? ?(2, 407.) > ? row 4: > ? row 5: ? (1, 602.) ? ?(3, 608.) > ? row 6: ? (0, 701.) ? ?(2, 707.) > ? row 7: > rstart 0 rend 4 > local row 0: ( 0 , 1.010000e+02) ( 2 , 1.070000e+02) > local row 1: > local row 2: ( 1 , 3.020000e+02) ( 3 , 3.080000e+02) > local row 3: ( 0 , 4.010000e+02) ( 2 , 4.070000e+02) > >> On 26 Aug 2025, at 3:18?PM, Pierre Jolivet wrote: >> >> >>> On 26 Aug 2025, at 12:50?PM, Alexis SALZMAN >>> wrote: >>> >>> Mark, you were right and I was wrong about the dense matrix. >>> Adding explicit zeros to the distributed matrix used to extract >>> the sub-matrices (making it dense) in my test does not change >>> the behaviour: there is still an error. >>> >>> I am finding it increasingly difficult to understand the logic >>> of the row and column 'IS' creation. I ran many tests to achieve >>> the desired result: a rectangular sub-matrix (so a rectangular >>> or square sub-matrix appears to be possible). However, many >>> others resulted in the same kind of error. >>> >> This may be a PETSc bug in?MatSetSeqMats_MPIAIJ(). >> -> 2965? ?PetscCall(MatSetValues(aij->B, 1, &row, 1, &col, &v, >> INSERT_VALUES)); >> col has a value of 4, which doesn?t make sense since the output >> Mat has 4 columns (thus, has the error message suggests, the >> value should be lower than or equal to 3). >> >> Thanks, >> Pierre >>> >>> From what I observed, the test only works if the column >>> selection contribution (size_c in the test) has a specific value >>> related to the row selection contribution (size_r in the test) >>> for proc 0 (rank for both communicator and sub-communicator): >>> >>> * if size_r==2 then if size_c<=2 it works. >>> * if size_r>=3 and size_r<=5 then size_c==size_r is the only >>> working case. >>> >>> This occurs "regardless" of what is requested in proc 1 and in >>> selr/selc (It can't be a dummy setting, though). In any case, >>> it's certainly not an exhaustive analysis. >>> >>> Many thanks to anyone who can explain to me the logic behind the >>> construction of row and column 'IS'. >>> >>> Regards >>> >>> A.S. >>> >>> >>> Le 25/08/2025 ? 20:00, Alexis SALZMAN a ?crit?: >>>> >>>> Thanks Mark for your attention. >>>> >>>> The uncleaned error message, compared to my post in July, is as >>>> follows: >>>> >>>> [0]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> [0]PETSC ERROR: Argument out of range >>>> [0]PETSC ERROR: Column too large: col 4 max 3 >>>> [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!eQa_exf2PCgmMQ0L4h9al-nkWsWRJJ1Zwkjm_qHJsqT0zwLzW7eMjKlkRssc6loRju6u04y4yp9L0U39POoDIvyQKcfmQGUDmdo$ >>>> >>>> for trouble shooting. >>>> [0]PETSC ERROR: Petsc Release Version 3.22.2, unknown >>>> [0]PETSC ERROR: subnb with 3 MPI process(es) and PETSC_ARCH ?on >>>> pc-str97.ec-nantes.fr by salzman >>>> Mon Aug 25 19:11:37 2025 >>>> [0]PETSC ERROR: Configure options: >>>> PETSC_ARCH=real_fc41_Release_gcc_i4 >>>> PETSC_DIR=/home/salzman/devel/ExternalLib/build/PETSC/petsc >>>> --doCleanup=1 --with-scalar-type=real --known-level1-dcach >>>> e-linesize=64 --with-cc=gcc --CFLAGS="-fPIC " >>>> --CC_LINKER_FLAGS=-fopenmp --with-cxx=g++ >>>> --with-cxx-dialect=c++20 --CXXFLAGS="-fPIC " >>>> --CXX_LINKER_FLAGS=-fopenmp --with-fc=gfortran --FFLAGS= >>>> "-fPIC " --FC_LINKER_FLAGS=-fopenmp --with-debugging=0 >>>> --with-fortran-bindings=0 --with-fortran-kernels=1 >>>> --with-mpi-compilers=0 >>>> --with-mpi-include=/usr/include/openmpi-x86_64 --with-mpi-li >>>> b="[/usr/lib64/openmpi/lib/libmpi.so,/usr/lib64/openmpi/lib/libmpi.so,/usr/lib64/openmpi/lib/libmpi_mpifh.so]" >>>> --with-blas-lib="[/opt/intel/oneapi/mkl/latest/lib/libmkl_intel_lp64.so,/opt/i >>>> ntel/oneapi/mkl/latest/lib/libmkl_gnu_thread.so,/opt/intel/oneapi/mkl/latest/lib/libmkl_core.so]" >>>> --with-lapack-lib="[/opt/intel/oneapi/mkl/latest/lib/libmkl_intel_lp64.so,/opt/intel/oneapi >>>> /mkl/latest/lib/libmkl_gnu_thread.so,/opt/intel/oneapi/mkl/latest/lib/libmkl_core.so]" >>>> --with-mumps=1 >>>> --with-mumps-include=/home/salzman/local/i4_gcc/include >>>> --with-mumps-lib="[/home/salzma >>>> n/local/i4_gcc/lib/libdmumps.so,/home/salzman/local/i4_gcc/lib/libmumps_common.so,/home/salzman/local/i4_gcc/lib/libpord.so]" >>>> --with-scalapack-lib="[/opt/intel/oneapi/mkl/latest/lib/libmkl_ >>>> scalapack_lp64.so,/opt/intel/oneapi/mkl/latest/lib/libmkl_blacs_openmpi_lp64.so]" >>>> --with-mkl_pardiso=1 >>>> --with-mkl_pardiso-include=/opt/intel/oneapi/mkl/latest/include >>>> --with-mkl_pardiso-lib >>>> ="[/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_intel_lp64.so]" >>>> --with-hdf5=1 --with-hdf5-include=/usr/include/openmpi-x86_64 >>>> --with-hdf5-lib="[/usr/lib64/openmpi/lib/libhdf5.so]" --with >>>> -pastix=0 --download-pastix=no --with-hwloc=1 >>>> --with-hwloc-dir=/home/salzman/local/i4_gcc --download-hwloc=no >>>> --with-ptscotch-include=/home/salzman/local/i4_gcc/include >>>> --with-ptscotch-lib= >>>> "[/home/salzman/local/i4_gcc/lib/libptscotch.a,/home/salzman/local/i4_gcc/lib/libptscotcherr.a,/home/salzman/local/i4_gcc/lib/libptscotcherrexit.a,/home/salzman/local/i4_gcc/lib/libscotch.a >>>> ,/home/salzman/local/i4_gcc/lib/libscotcherr.a,/home/salzman/local/i4_gcc/lib/libscotcherrexit.a]" >>>> --with-hypre=1 --download-hypre=yes --with-suitesparse=1 >>>> --with-suitesparse-include=/home/ >>>> salzman/local/i4_gcc/include >>>> --with-suitesparse-lib="[/home/salzman/local/i4_gcc/lib/libsuitesparseconfig.so,/home/salzman/local/i4_gcc/lib/libumfpack.so,/home/salzman/local/i4_gcc/lib/libk >>>> lu.so,/home/salzman/local/i4_gcc/lib/libcholmod.so,/home/salzman/local/i4_gcc/lib/libspqr.so,/home/salzman/local/i4_gcc/lib/libcolamd.so,/home/salzman/local/i4_gcc/lib/libccolamd.so,/home/s >>>> alzman/local/i4_gcc/lib/libcamd.so,/home/salzman/local/i4_gcc/lib/libamd.so,/home/salzman/local/i4_gcc/lib/libmetis.so]" >>>> --download-suitesparse=no --with-python-exec=python3.12 >>>> --have-numpy >>>> =1 ---with-petsc4py=1 ---with-petsc4py-test-np=4 >>>> ---with-mpi4py=1 >>>> --prefix=/home/salzman/local/i4_gcc/real_arithmetic >>>> COPTFLAGS="-O3 -g " CXXOPTFLAGS="-O3 -g " FOPTFLAGS="-O3 -g " >>>> [0]PETSC ERROR: #1 MatSetValues_SeqAIJ() at >>>> /home/salzman/devel/PETSc/petsc/src/mat/impls/aij/seq/aij.c:426 >>>> [0]PETSC ERROR: #2 MatSetValues() at >>>> /home/salzman/devel/PETSc/petsc/src/mat/interface/matrix.c:1543 >>>> [0]PETSC ERROR: #3 MatSetSeqMats_MPIAIJ() at >>>> /home/salzman/devel/PETSc/petsc/src/mat/impls/aij/mpi/mpiov.c:2965 >>>> [0]PETSC ERROR: #4 MatCreateSubMatricesMPI_MPIXAIJ() at >>>> /home/salzman/devel/PETSc/petsc/src/mat/impls/aij/mpi/mpiov.c:3163 >>>> [0]PETSC ERROR: #5 MatCreateSubMatricesMPI_MPIAIJ() at >>>> /home/salzman/devel/PETSc/petsc/src/mat/impls/aij/mpi/mpiov.c:3196 >>>> [0]PETSC ERROR: #6 MatCreateSubMatricesMPI() at >>>> /home/salzman/devel/PETSc/petsc/src/mat/interface/matrix.c:7293 >>>> [0]PETSC ERROR: #7 main() at subnb.c:181 >>>> [0]PETSC ERROR: No PETSc Option Table entries >>>> [0]PETSC ERROR: ----------------End of Error Message >>>> -------send entire error message to >>>> petsc-maint at mcs.anl.gov---------- >>>> -------------------------------------------------------------------------- >>>> >>>> This message comes from executing the attached test (I >>>> simplified the test by removing the block size from the matrix >>>> used for extraction, compared to the July test). In >>>> proc_xx_output.txt, you will find the output from the code >>>> execution with the -ok option (i.e. irow/idxr and icol/idxc are >>>> the same, i.e. a square sub-block for colour 0 distributed >>>> across the first two processes). >>>> >>>> Has expected in this case we obtain the 0,3,6,9 sub-block >>>> terms, which are distributed across processes 0 and 1 (two rows >>>> per proc). >>>> >>>> When asking for rectangular sub-block (i.e. with no option) it >>>> crash with column to large on process 0: 4 col max 3 ??? I ask >>>> for 4 rows and 2 columns in this process ??? >>>> >>>> Otherwise, I mention the dense aspect of the matrix in ex183.c, >>>> because, in this case, no matter what selection is requested, >>>> all terms are non-null. If there is an issue with the way the >>>> selection is coded in the user program, I think it will be >>>> masked thanks to the full graph representation. However, this >>>> may not be the case ? I should test it. >>>> >>>> I'll take a look at ex23.c. >>>> >>>> Thanks, >>>> >>>> A.S. >>>> >>>> >>>> >>>> Le 25/08/2025 ? 17:55, Mark Adams a ?crit?: >>>>> Ah, OK, never say never. >>>>> >>>>> MatCreateSubMatrices seems to support creating a new matrix >>>>> with the communicator of the IS. >>>>> It just needs to read from the input matrix and does not use >>>>> it for communication, so it can do that. >>>>> >>>>> As far as rectangular matrices, there is no reason not to >>>>> support that (the row IS and column IS can be distinct). >>>>> Can you send the whole error message? >>>>> There may not be a test that does this, >>>>> but?src/mat/tests/ex23.c looks like it may be a >>>>> rectangular?matrix output. >>>>> >>>>> And, it should not matter if the input matrix has a 100% full >>>>> sparse matrix. It is still MatAIJ. >>>>> The semantics and API is the same for sparse or dense matrices. >>>>> >>>>> Thanks, >>>>> Mark >>>>> >>>>> On Mon, Aug 25, 2025 at 7:31?AM Alexis SALZMAN >>>>> wrote: >>>>> >>>>> Hi, >>>>> >>>>> Thanks for your answer, Mark. Perhaps >>>>> MatCreateSubMatricesMPI is the only PETSc function that >>>>> acts on a sub-communicator ? I'm not sure ? but it's clear >>>>> that there's no ambiguity on that point. The first line of >>>>> the documentation for that function states that it 'may >>>>> live on subcomms'. This is confirmed by the >>>>> 'src/mat/tests/ex183.c' test case. I used this test case >>>>> to understand the function, which helped me with my code >>>>> and the example I provided in my initial post. >>>>> Unfortunately, in this example, the matrix from which the >>>>> sub-matrices are extracted is dense, even though it uses a >>>>> sparse structure. This does not clarify how to define >>>>> sub-matrices when extracting from a sparse distributed >>>>> matrix. Since my initial post, I have discovered that >>>>> having more columns than rows can also result in the same >>>>> error message. >>>>> >>>>> So, my questions boil down to: >>>>> >>>>> Can MatCreateSubMatricesMPI extract rectangular matrices >>>>> from a square distributed sparse matrix? >>>>> >>>>> If not, the fact that only square matrices can be >>>>> extracted in this context should perhaps be mentioned in >>>>> the documentation. >>>>> >>>>> If so, I would be very grateful for any assistance in >>>>> defining an IS pair in this context. >>>>> >>>>> Regards >>>>> >>>>> A.S. >>>>> >>>>> Le 27/07/2025 ? 00:15, Mark Adams a ?crit?: >>>>>> First, you can not mix communicators in PETSc calls in >>>>>> general (ever?), but this error looks like you?might >>>>>> be?asking for a row from the matrix that?does not exist. >>>>>> You should start with a PETSc example code. Test it and >>>>>> modify?it to suit your needs. >>>>>> >>>>>> Good luck, >>>>>> Mark >>>>>> >>>>>> On Fri, Jul 25, 2025 at 9:31?AM Alexis SALZMAN >>>>>> wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> As I am relatively new to Petsc, I may have >>>>>> misunderstood how to use the >>>>>> MatCreateSubMatricesMPI function. The attached code >>>>>> is tuned for three >>>>>> processes and extracts one matrix for each colour of >>>>>> a subcommunicator >>>>>> that has been created using the MPI_Comm_split >>>>>> function from an? MPIAij >>>>>> matrix. The following error message appears when the >>>>>> code is set to its >>>>>> default configuration (i.e. when a rectangular matrix >>>>>> is extracted with >>>>>> more rows than columns for colour 0): >>>>>> >>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>> -------------------------------------------------------------- >>>>>> [0]PETSC ERROR: Argument out of range >>>>>> [0]PETSC ERROR: Column too large: col 4 max 3 >>>>>> [0]PETSC ERROR: See >>>>>> https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!ZqH097BZ0G0O3WI7RWrwIKFNpyk0czSWEqfusAeTlgEygAffwpgBUzsLw1TIoGkjZ3mYG-NRQxxFoxU4y8EyY0ofiz9I43Qwe0w$ >>>>>> for trouble shooting. >>>>>> [0]PETSC ERROR: Petsc Release Version 3.22.2, unknown >>>>>> >>>>>> ... petsc git hash 2a89477b25f compiled on a dell i9 >>>>>> computer with Gcc >>>>>> 14.3, mkl 2025.2, ..... >>>>>> [0]PETSC ERROR: #1 MatSetValues_SeqAIJ() at >>>>>> ...petsc/src/mat/impls/aij/seq/aij.c:426 >>>>>> [0]PETSC ERROR: #2 MatSetValues() at >>>>>> ...petsc/src/mat/interface/matrix.c:1543 >>>>>> [0]PETSC ERROR: #3 MatSetSeqMats_MPIAIJ() at >>>>>> .../petsc/src/mat/impls/aij/mpi/mpiov.c:2965 >>>>>> [0]PETSC ERROR: #4 MatCreateSubMatricesMPI_MPIXAIJ() at >>>>>> .../petsc/src/mat/impls/aij/mpi/mpiov.c:3163 >>>>>> [0]PETSC ERROR: #5 MatCreateSubMatricesMPI_MPIAIJ() at >>>>>> .../petsc/src/mat/impls/aij/mpi/mpiov.c:3196 >>>>>> [0]PETSC ERROR: #6 MatCreateSubMatricesMPI() at >>>>>> .../petsc/src/mat/interface/matrix.c:7293 >>>>>> [0]PETSC ERROR: #7 main() at sub.c:169 >>>>>> >>>>>> When the '-ok' option is selected, the code extracts >>>>>> a square matrix for >>>>>> colour 0, which runs smoothly in this case. Selecting >>>>>> the '-trans' >>>>>> option swaps the row and column selection indices, >>>>>> providing a >>>>>> transposed submatrix smoothly. For colour 1, which >>>>>> uses only one process >>>>>> and is therefore sequential, rectangular extraction >>>>>> is OK regardless of >>>>>> the shape. >>>>>> >>>>>> Is this dependency on the shape expected? Have I >>>>>> missed an important >>>>>> tuning step somewhere? >>>>>> >>>>>> Thank you in advance for any clarification. >>>>>> >>>>>> Regards >>>>>> >>>>>> A.S. >>>>>> >>>>>> P.S.: I'm sorry, but as I'm leaving my office for the >>>>>> following weeks >>>>>> this evening, I won't be very responsive during this >>>>>> period. >>>>>> >>>>>> >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!eQa_exf2PCgmMQ0L4h9al-nkWsWRJJ1Zwkjm_qHJsqT0zwLzW7eMjKlkRssc6loRju6u04y4yp9L0U39POoDIvyQKcfmxLsZ-Is$ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Aug 27 07:04:50 2025 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 27 Aug 2025 08:04:50 -0400 Subject: [petsc-users] MatCreateSubMatricesMPI strange behavior In-Reply-To: <3b6ce54f-1c62-4494-b162-eec2f90ee31f@ec-nantes.fr> References: <9d648a1d-72d2-44d0-8a3b-a9d64b01604f@ec-nantes.fr> <113ece7d-25cf-4b16-b29b-bdd1ad2a004a@ec-nantes.fr> <82d06db6-ab86-49b8-9e26-000745ae2134@ec-nantes.fr> <8A46BABF-E664-4923-87DF-A407B29163A8@joliv.et> <37AC2024-4D4C-41BD-A543-07221FDFB2C3@joliv.et> <3b6ce54f-1c62-4494-b162-eec2f90ee31f@ec-nantes.fr> Message-ID: On Wed, Aug 27, 2025 at 5:15?AM Alexis SALZMAN wrote: > Hello Pierre, > > After applying your patch to my local version of PETSc, all of the cases > that previously caused the provided test to fail are now running smoothly. > In a more complex context (with more processes and colors in my > application), no errors are found and the sub-matrices look OK. > > Thank you very much for your time. This debugged function will greatly > simplify the development of my application. > > Thanks for taking the time to report the error in a clear fashion. We can't make the code better without this kind of cooperative effort. Matt > Best regards > > A.S. > Le 26/08/2025 ? 15:46, Matthew Knepley a ?crit : > > On Tue, Aug 26, 2025 at 9:42?AM Pierre Jolivet wrote: > >> It?s indeed very suspicious (to me) that we are using rmap to change a >> column index. >> Switching to cmap gets your code running, but I?ll need to see if this >> triggers regressions. >> > > That looks right to me. I am sure this has only been tested for GASM, > which would be symmetric. > > Thanks, > > Matt > > >> Thanks for the report, >> Pierre >> >> diff --git a/src/mat/impls/aij/mpi/mpiov.c b/src/mat/impls/aij/mpi/mpiov.c >> index d1037d7d817..051981ebe9a 100644 >> --- a/src/mat/impls/aij/mpi/mpiov.c >> +++ b/src/mat/impls/aij/mpi/mpiov.c >> @@ -2948,3 +2948,3 @@ PetscErrorCode MatSetSeqMats_MPIAIJ(Mat C, IS >> rowemb, IS dcolemb, IS ocolemb, Ma >> >> - PetscCall(PetscLayoutGetRange(C->rmap, &rstart, &rend)); >> + PetscCall(PetscLayoutGetRange(C->cmap, &rstart, &rend)); >> shift = rend - rstart; >> >> $ cat proc_0_output.txt >> rstart 0 rend 4 >> Mat Object: 3 MPI processes >> type: mpiaij >> row 0: (0, 101.) (3, 104.) (6, 107.) (9, 110.) >> row 1: (2, 203.) (5, 206.) (8, 209.) (11, 212.) >> row 2: (1, 302.) (4, 305.) (7, 308.) (10, 311.) >> row 3: (0, 401.) (3, 404.) (6, 407.) (9, 410.) >> row 4: (2, 503.) (5, 506.) (8, 509.) (11, 512.) >> row 5: (1, 602.) (4, 605.) (7, 608.) (10, 611.) >> row 6: (0, 701.) (3, 704.) (6, 707.) (9, 710.) >> row 7: (2, 803.) (5, 806.) (8, 809.) (11, 812.) >> row 8: (1, 902.) (4, 905.) (7, 908.) (10, 911.) >> row 9: (0, 1001.) (3, 1004.) (6, 1007.) (9, 1010.) >> row 10: (2, 1103.) (5, 1106.) (8, 1109.) (11, 1112.) >> row 11: (1, 1202.) (4, 1205.) (7, 1208.) (10, 1211.) >> idxr proc >> IS Object: 2 MPI processes >> type: general >> [0] Number of indices in set 4 >> [0] 0 0 >> [0] 1 1 >> [0] 2 2 >> [0] 3 3 >> [1] Number of indices in set 4 >> [1] 0 4 >> [1] 1 5 >> [1] 2 6 >> [1] 3 7 >> idxc proc >> IS Object: 2 MPI processes >> type: general >> [0] Number of indices in set 2 >> [0] 0 0 >> [0] 1 1 >> [1] Number of indices in set 2 >> [1] 0 6 >> [1] 1 7 >> Mat Object: 2 MPI processes >> type: mpiaij >> row 0: (0, 101.) (2, 107.) >> row 1: >> row 2: (1, 302.) (3, 308.) >> row 3: (0, 401.) (2, 407.) >> row 4: >> row 5: (1, 602.) (3, 608.) >> row 6: (0, 701.) (2, 707.) >> row 7: >> rstart 0 rend 4 >> local row 0: ( 0 , 1.010000e+02) ( 2 , 1.070000e+02) >> local row 1: >> local row 2: ( 1 , 3.020000e+02) ( 3 , 3.080000e+02) >> local row 3: ( 0 , 4.010000e+02) ( 2 , 4.070000e+02) >> >> On 26 Aug 2025, at 3:18?PM, Pierre Jolivet wrote: >> >> >> On 26 Aug 2025, at 12:50?PM, Alexis SALZMAN >> wrote: >> >> Mark, you were right and I was wrong about the dense matrix. Adding >> explicit zeros to the distributed matrix used to extract the sub-matrices >> (making it dense) in my test does not change the behaviour: there is still >> an error. >> >> I am finding it increasingly difficult to understand the logic of the row >> and column 'IS' creation. I ran many tests to achieve the desired result: a >> rectangular sub-matrix (so a rectangular or square sub-matrix appears to be >> possible). However, many others resulted in the same kind of error. >> >> This may be a PETSc bug in MatSetSeqMats_MPIAIJ(). >> -> 2965 PetscCall(MatSetValues(aij->B, 1, &row, 1, &col, &v, >> INSERT_VALUES)); >> col has a value of 4, which doesn?t make sense since the output Mat has 4 >> columns (thus, has the error message suggests, the value should be lower >> than or equal to 3). >> >> Thanks, >> Pierre >> >> From what I observed, the test only works if the column selection >> contribution (size_c in the test) has a specific value related to the row >> selection contribution (size_r in the test) for proc 0 (rank for both >> communicator and sub-communicator): >> >> - if size_r==2 then if size_c<=2 it works. >> - if size_r>=3 and size_r<=5 then size_c==size_r is the only working >> case. >> >> This occurs "regardless" of what is requested in proc 1 and in selr/selc >> (It can't be a dummy setting, though). In any case, it's certainly not an >> exhaustive analysis. >> >> Many thanks to anyone who can explain to me the logic behind the >> construction of row and column 'IS'. >> >> Regards >> >> A.S. >> >> >> Le 25/08/2025 ? 20:00, Alexis SALZMAN a ?crit : >> >> Thanks Mark for your attention. >> >> The uncleaned error message, compared to my post in July, is as follows: >> >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: Argument out of range >> [0]PETSC ERROR: Column too large: col 4 max 3 >> [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!eHMhc5L1tNjyBrEpH0recWwhp5pSwBnZuh09zH8p-ZQyM8m3kxz8ryLVwi7nXr0NNKedHWcHfY5etKLEhuGb$ >> >> for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.22.2, unknown >> [0]PETSC ERROR: subnb with 3 MPI process(es) and PETSC_ARCH on >> pc-str97.ec-nantes.fr >> >> by salzman Mon Aug 25 19:11:37 2025 >> [0]PETSC ERROR: Configure options: PETSC_ARCH=real_fc41_Release_gcc_i4 >> PETSC_DIR=/home/salzman/devel/ExternalLib/build/PETSC/petsc --doCleanup=1 >> --with-scalar-type=real --known-level1-dcach >> e-linesize=64 --with-cc=gcc --CFLAGS="-fPIC " --CC_LINKER_FLAGS=-fopenmp >> --with-cxx=g++ --with-cxx-dialect=c++20 --CXXFLAGS="-fPIC " >> --CXX_LINKER_FLAGS=-fopenmp --with-fc=gfortran --FFLAGS= >> "-fPIC " --FC_LINKER_FLAGS=-fopenmp --with-debugging=0 >> --with-fortran-bindings=0 --with-fortran-kernels=1 --with-mpi-compilers=0 >> --with-mpi-include=/usr/include/openmpi-x86_64 --with-mpi-li >> b="[/usr/lib64/openmpi/lib/libmpi.so,/usr/lib64/openmpi/lib/libmpi.so,/usr/lib64/openmpi/lib/libmpi_mpifh.so]" >> --with-blas-lib="[/opt/intel/oneapi/mkl/latest/lib/libmkl_intel_lp64.so,/opt/i >> ntel/oneapi/mkl/latest/lib/libmkl_gnu_thread.so,/opt/intel/oneapi/mkl/latest/lib/libmkl_core.so]" >> --with-lapack-lib="[/opt/intel/oneapi/mkl/latest/lib/libmkl_intel_lp64.so,/opt/intel/oneapi >> /mkl/latest/lib/libmkl_gnu_thread.so,/opt/intel/oneapi/mkl/latest/lib/libmkl_core.so]" >> --with-mumps=1 --with-mumps-include=/home/salzman/local/i4_gcc/include >> --with-mumps-lib="[/home/salzma >> n/local/i4_gcc/lib/libdmumps.so,/home/salzman/local/i4_gcc/lib/libmumps_common.so,/home/salzman/local/i4_gcc/lib/libpord.so]" >> --with-scalapack-lib="[/opt/intel/oneapi/mkl/latest/lib/libmkl_ >> scalapack_lp64.so,/opt/intel/oneapi/mkl/latest/lib/libmkl_blacs_openmpi_lp64.so]" >> --with-mkl_pardiso=1 >> --with-mkl_pardiso-include=/opt/intel/oneapi/mkl/latest/include >> --with-mkl_pardiso-lib >> ="[/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_intel_lp64.so]" >> --with-hdf5=1 --with-hdf5-include=/usr/include/openmpi-x86_64 >> --with-hdf5-lib="[/usr/lib64/openmpi/lib/libhdf5.so]" --with >> -pastix=0 --download-pastix=no --with-hwloc=1 >> --with-hwloc-dir=/home/salzman/local/i4_gcc --download-hwloc=no >> --with-ptscotch-include=/home/salzman/local/i4_gcc/include >> --with-ptscotch-lib= >> >> "[/home/salzman/local/i4_gcc/lib/libptscotch.a,/home/salzman/local/i4_gcc/lib/libptscotcherr.a,/home/salzman/local/i4_gcc/lib/libptscotcherrexit.a,/home/salzman/local/i4_gcc/lib/libscotch.a >> ,/home/salzman/local/i4_gcc/lib/libscotcherr.a,/home/salzman/local/i4_gcc/lib/libscotcherrexit.a]" >> --with-hypre=1 --download-hypre=yes --with-suitesparse=1 >> --with-suitesparse-include=/home/ >> salzman/local/i4_gcc/include >> --with-suitesparse-lib="[/home/salzman/local/i4_gcc/lib/libsuitesparseconfig.so,/home/salzman/local/i4_gcc/lib/libumfpack.so,/home/salzman/local/i4_gcc/lib/libk >> >> lu.so,/home/salzman/local/i4_gcc/lib/libcholmod.so,/home/salzman/local/i4_gcc/lib/libspqr.so,/home/salzman/local/i4_gcc/lib/libcolamd.so,/home/salzman/local/i4_gcc/lib/libccolamd.so,/home/s >> alzman/local/i4_gcc/lib/libcamd.so,/home/salzman/local/i4_gcc/lib/libamd.so,/home/salzman/local/i4_gcc/lib/libmetis.so]" >> --download-suitesparse=no --with-python-exec=python3.12 --have-numpy >> =1 ---with-petsc4py=1 ---with-petsc4py-test-np=4 ---with-mpi4py=1 >> --prefix=/home/salzman/local/i4_gcc/real_arithmetic COPTFLAGS="-O3 -g " >> CXXOPTFLAGS="-O3 -g " FOPTFLAGS="-O3 -g " >> [0]PETSC ERROR: #1 MatSetValues_SeqAIJ() at >> /home/salzman/devel/PETSc/petsc/src/mat/impls/aij/seq/aij.c:426 >> [0]PETSC ERROR: #2 MatSetValues() at >> /home/salzman/devel/PETSc/petsc/src/mat/interface/matrix.c:1543 >> [0]PETSC ERROR: #3 MatSetSeqMats_MPIAIJ() at >> /home/salzman/devel/PETSc/petsc/src/mat/impls/aij/mpi/mpiov.c:2965 >> [0]PETSC ERROR: #4 MatCreateSubMatricesMPI_MPIXAIJ() at >> /home/salzman/devel/PETSc/petsc/src/mat/impls/aij/mpi/mpiov.c:3163 >> [0]PETSC ERROR: #5 MatCreateSubMatricesMPI_MPIAIJ() at >> /home/salzman/devel/PETSc/petsc/src/mat/impls/aij/mpi/mpiov.c:3196 >> [0]PETSC ERROR: #6 MatCreateSubMatricesMPI() at >> /home/salzman/devel/PETSc/petsc/src/mat/interface/matrix.c:7293 >> [0]PETSC ERROR: #7 main() at subnb.c:181 >> [0]PETSC ERROR: No PETSc Option Table entries >> [0]PETSC ERROR: ----------------End of Error Message -------send entire >> error message to petsc-maint at mcs.anl.gov---------- >> -------------------------------------------------------------------------- >> >> This message comes from executing the attached test (I simplified the >> test by removing the block size from the matrix used for extraction, >> compared to the July test). In proc_xx_output.txt, you will find the output >> from the code execution with the -ok option (i.e. irow/idxr and icol/idxc >> are the same, i.e. a square sub-block for colour 0 distributed across the >> first two processes). >> >> Has expected in this case we obtain the 0,3,6,9 sub-block terms, which >> are distributed across processes 0 and 1 (two rows per proc). >> >> When asking for rectangular sub-block (i.e. with no option) it crash with >> column to large on process 0: 4 col max 3 ??? I ask for 4 rows and 2 >> columns in this process ??? >> >> Otherwise, I mention the dense aspect of the matrix in ex183.c, because, >> in this case, no matter what selection is requested, all terms are >> non-null. If there is an issue with the way the selection is coded in the >> user program, I think it will be masked thanks to the full graph >> representation. However, this may not be the case ? I should test it. >> >> I'll take a look at ex23.c. >> >> Thanks, >> >> A.S. >> >> >> >> Le 25/08/2025 ? 17:55, Mark Adams a ?crit : >> >> Ah, OK, never say never. >> >> MatCreateSubMatrices seems to support creating a new matrix with the >> communicator of the IS. >> It just needs to read from the input matrix and does not use it for >> communication, so it can do that. >> >> As far as rectangular matrices, there is no reason not to support that >> (the row IS and column IS can be distinct). >> Can you send the whole error message? >> There may not be a test that does this, but src/mat/tests/ex23.c looks >> like it may be a rectangular matrix output. >> >> And, it should not matter if the input matrix has a 100% full sparse >> matrix. It is still MatAIJ. >> The semantics and API is the same for sparse or dense matrices. >> >> Thanks, >> Mark >> >> On Mon, Aug 25, 2025 at 7:31?AM Alexis SALZMAN < >> alexis.salzman at ec-nantes.fr> wrote: >> >>> Hi, >>> >>> Thanks for your answer, Mark. Perhaps MatCreateSubMatricesMPI is the >>> only PETSc function that acts on a sub-communicator ? I'm not sure ? but >>> it's clear that there's no ambiguity on that point. The first line of the >>> documentation for that function states that it 'may live on subcomms'. This >>> is confirmed by the 'src/mat/tests/ex183.c' test case. I used this test >>> case to understand the function, which helped me with my code and the >>> example I provided in my initial post. Unfortunately, in this example, the >>> matrix from which the sub-matrices are extracted is dense, even though it >>> uses a sparse structure. This does not clarify how to define sub-matrices >>> when extracting from a sparse distributed matrix. Since my initial post, I >>> have discovered that having more columns than rows can also result in the >>> same error message. >>> >>> So, my questions boil down to: >>> >>> Can MatCreateSubMatricesMPI extract rectangular matrices from a square >>> distributed sparse matrix? >>> >>> If not, the fact that only square matrices can be extracted in this >>> context should perhaps be mentioned in the documentation. >>> >>> If so, I would be very grateful for any assistance in defining an IS >>> pair in this context. >>> >>> Regards >>> >>> A.S. >>> Le 27/07/2025 ? 00:15, Mark Adams a ?crit : >>> >>> First, you can not mix communicators in PETSc calls in general (ever?), >>> but this error looks like you might be asking for a row from the matrix >>> that does not exist. >>> You should start with a PETSc example code. Test it and modify it to >>> suit your needs. >>> >>> Good luck, >>> Mark >>> >>> On Fri, Jul 25, 2025 at 9:31?AM Alexis SALZMAN < >>> alexis.salzman at ec-nantes.fr> wrote: >>> >>>> Hi, >>>> >>>> As I am relatively new to Petsc, I may have misunderstood how to use >>>> the >>>> MatCreateSubMatricesMPI function. The attached code is tuned for three >>>> processes and extracts one matrix for each colour of a subcommunicator >>>> that has been created using the MPI_Comm_split function from an MPIAij >>>> matrix. The following error message appears when the code is set to its >>>> default configuration (i.e. when a rectangular matrix is extracted with >>>> more rows than columns for colour 0): >>>> >>>> [0]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> [0]PETSC ERROR: Argument out of range >>>> [0]PETSC ERROR: Column too large: col 4 max 3 >>>> [0]PETSC ERROR: See >>>> https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!ZqH097BZ0G0O3WI7RWrwIKFNpyk0czSWEqfusAeTlgEygAffwpgBUzsLw1TIoGkjZ3mYG-NRQxxFoxU4y8EyY0ofiz9I43Qwe0w$ >>>> for trouble shooting. >>>> [0]PETSC ERROR: Petsc Release Version 3.22.2, unknown >>>> >>>> ... petsc git hash 2a89477b25f compiled on a dell i9 computer with Gcc >>>> 14.3, mkl 2025.2, ..... >>>> [0]PETSC ERROR: #1 MatSetValues_SeqAIJ() at >>>> ...petsc/src/mat/impls/aij/seq/aij.c:426 >>>> [0]PETSC ERROR: #2 MatSetValues() at >>>> ...petsc/src/mat/interface/matrix.c:1543 >>>> [0]PETSC ERROR: #3 MatSetSeqMats_MPIAIJ() at >>>> .../petsc/src/mat/impls/aij/mpi/mpiov.c:2965 >>>> [0]PETSC ERROR: #4 MatCreateSubMatricesMPI_MPIXAIJ() at >>>> .../petsc/src/mat/impls/aij/mpi/mpiov.c:3163 >>>> [0]PETSC ERROR: #5 MatCreateSubMatricesMPI_MPIAIJ() at >>>> .../petsc/src/mat/impls/aij/mpi/mpiov.c:3196 >>>> [0]PETSC ERROR: #6 MatCreateSubMatricesMPI() at >>>> .../petsc/src/mat/interface/matrix.c:7293 >>>> [0]PETSC ERROR: #7 main() at sub.c:169 >>>> >>>> When the '-ok' option is selected, the code extracts a square matrix >>>> for >>>> colour 0, which runs smoothly in this case. Selecting the '-trans' >>>> option swaps the row and column selection indices, providing a >>>> transposed submatrix smoothly. For colour 1, which uses only one >>>> process >>>> and is therefore sequential, rectangular extraction is OK regardless of >>>> the shape. >>>> >>>> Is this dependency on the shape expected? Have I missed an important >>>> tuning step somewhere? >>>> >>>> Thank you in advance for any clarification. >>>> >>>> Regards >>>> >>>> A.S. >>>> >>>> P.S.: I'm sorry, but as I'm leaving my office for the following weeks >>>> this evening, I won't be very responsive during this period. >>>> >>>> >>>> >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!eHMhc5L1tNjyBrEpH0recWwhp5pSwBnZuh09zH8p-ZQyM8m3kxz8ryLVwi7nXr0NNKedHWcHfY5etJ5Zn86b$ > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!eHMhc5L1tNjyBrEpH0recWwhp5pSwBnZuh09zH8p-ZQyM8m3kxz8ryLVwi7nXr0NNKedHWcHfY5etJ5Zn86b$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Aug 27 07:05:38 2025 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 27 Aug 2025 08:05:38 -0400 Subject: [petsc-users] subdomain solver options In-Reply-To: <39a89c20-0927-48f3-9ba1-caee3ba3e31b@uni-mainz.de> References: <210a91d0-9d0e-44a5-8827-09f74861450d@uni-mainz.de> <39a89c20-0927-48f3-9ba1-caee3ba3e31b@uni-mainz.de> Message-ID: See you next week! Matt On Wed, Aug 27, 2025 at 3:39?AM Anton Popov wrote: > Thanks Matt, it works! > > Anton > On 26.08.25 22:34, Matthew Knepley wrote: > > On Tue, Aug 26, 2025 at 12:45?PM Anton Popov wrote: > >> There is an important addition, I use asm and bjacobi as smoothers on >> multigrid levels. >> >> So I probably need to set the options with -mg_levels_sub_, or >> -mg_levels_3_sub_(if they are different on different levels), instead of >> just -sub_. >> >> Correct? >> > > Yes, and as Mark says, you can see the prefix if you View the solver. > > Thanks, > > Matt > > >> Anton >> >> >> >> On 26.08.25 18:20, Anton Popov wrote: >> > Hi all, >> > >> > I have a strange behavior setting the solver options for asm and >> > bjacobi blocks. >> > >> > Irrespective of what I set with -sub_ksp_ -sub_pc_ it just selects the >> > default preonly and ilu options. >> > >> > This behavior does not change no matter how many local blocks per cpu >> > I try to use, for example with the default one block/cpu it does not >> > work as well. >> > >> > Is there anything wrong? >> > >> > Best, >> > >> > Anton >> > >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!ae4Aps9OCcaKciUHsxH-E9BnL2D7koMv1G07Exb_JY9ucOmJAyxzUmcN2-yQA2HhQwqM60PF6N6iaxjalQE9$ > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!ae4Aps9OCcaKciUHsxH-E9BnL2D7koMv1G07Exb_JY9ucOmJAyxzUmcN2-yQA2HhQwqM60PF6N6iaxjalQE9$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From barryfsmith at icloud.com Fri Aug 29 13:52:39 2025 From: barryfsmith at icloud.com (Barry Smith) Date: Fri, 29 Aug 2025 14:52:39 -0400 Subject: [petsc-users] Question about Pseudo Time Stepping In-Reply-To: <85e1d3ad-6a32-444e-94dd-b5ebb6cba819@unibas.it> References: <85e1d3ad-6a32-444e-94dd-b5ebb6cba819@unibas.it> Message-ID: Aldo, I understand the situation you are facing. With pseudo-continuation, when you provide an RHS function, SNES is computing udot - F(u). Inside the SNESSolve(), specifically in TSComputeIFunction(), with the lines if (!ts->ops->ifunction) { ierr = TSComputeRHSFunction(ts,t,X,Y);CHKERRQ(ierr); ierr = VecAYPX(Y,-1,Xdot);CHKERRQ(ierr); But for the original system, we care about the norm for F(u). So in TSStep_Pseudo(), after the nonlinear solve there is the line pseudo->fnorm = -1; /* The current norm is no longer valid, monitor must recompute it. */ and further down if (pseudo->fnorm < 0) { PetscCall(VecZeroEntries(pseudo->xdot)); PetscCall(TSComputeIFunction(ts, ts->ptime, ts->vec_sol, pseudo->xdot, pseudo->func, PETSC_FALSE)); PetscCall(VecNorm(pseudo->func, NORM_2, &pseudo->fnorm)); this norm is needed to determine of the nonlinear system has converged satisfactory (or if more pseudo time steps are needed). Hence, your FormRHSFunction gets called twice with identical input. To make matters even worse your FormRHSFunction is actually being called THREE times with identical input. Because at the start of TSStep_Pseudo() it calls TSPseudoComputeTimeStep(ts, &next_time_step) which by default uses TSPseudoTimeStepDefault() which has the same chunk of code PetscCall(VecZeroEntries(pseudo->xdot)); PetscCall(TSComputeIFunction(ts, ts->ptime, ts->vec_sol, pseudo->xdot, pseudo->func, PETSC_FALSE)); PetscCall(VecNorm(pseudo->func, NORM_2, &pseudo->fnorm)); And the same norm is also recomputed. Now the diff diff --git a/src/ts/impls/pseudo/posindep.c b/src/ts/impls/pseudo/posindep.c index 7c013435f98..813a28311e7 100644 --- a/src/ts/impls/pseudo/posindep.c +++ b/src/ts/impls/pseudo/posindep.c @@ -660,9 +660,11 @@ PetscErrorCode TSPseudoTimeStepDefault(TS ts, PetscReal *newdt, void *dtctx) PetscReal inc = pseudo->dt_increment; PetscFunctionBegin; - PetscCall(VecZeroEntries(pseudo->xdot)); - PetscCall(TSComputeIFunction(ts, ts->ptime, ts->vec_sol, pseudo->xdot, pseudo->func, PETSC_FALSE)); - PetscCall(VecNorm(pseudo->func, NORM_2, &pseudo->fnorm)); + if (pseudo->fnorm < 0.0) { + PetscCall(VecZeroEntries(pseudo->xdot)); + PetscCall(TSComputeIFunction(ts, ts->ptime, ts->vec_sol, pseudo->xdot, pseudo->func, PETSC_FALSE)); + PetscCall(VecNorm(pseudo->func, NORM_2, &pseudo->fnorm)); + } if (pseudo->fnorm_initial < 0) { /* first time through so compute initial function norm */ pseudo->fnorm_initial = pseudo->fnorm will prevent the THIRD computation of the same function evaluation. So this is a good optimization But eliminating the second computation is not so trivial. To review, deep inside the TS code there is TSComputeRHSFunction(ts,t,X,Y); VecAYPX(Y,-1,Xdot); This code knows nothing about TSPPseudo etc. Then later in the Pseudo code PetscCall(VecZeroEntries(pseudo->xdot)); PetscCall(TSComputeIFunction(ts, ts->ptime, ts->vec_sol, pseudo->xdot, pseudo->func, PETSC_FALSE)); PetscCall(VecNorm(pseudo->func, NORM_2, &pseudo->fnorm)); again calls TSComputeRHSFunction(ts,t,X,Y); with the same X. (since Xdot is not used directly by TSComputeRHSFunction()). Now if all the code was custom for Pseudo we could simply put in a line TSComputeRHSFunction(ts,t,X,Y); VecNorm(Y,NORM_2, &pseudo->fnorm) VecAYPX(Y,-1,Xdot); into TSComputeIFunction() and have the norm available later for pseudo. It is not clear that there is any definitive benefit for pseudo to utilize the TS framework. It possibly gets some code reuse from TS but at the cost of extra code that won't be needed if written directly. Plus this performance hit. Anyways I will make a merge request to remove the third redundant computation of FormRHSFunction but will await input from others on if anything else could be done to remove the inefficiency. Thanks for reporting this challenging problem. Barry Note: I don't think providing the Jacobian improves the efficiency, it is just better at hiding the redundant computation. > On Aug 26, 2025, at 4:21?AM, Aldo Bonfiglioli wrote: > > Hi there, > > I am using -ts_type pseudo to find steady solutions of the Euler/NS PDEs. > > It looks like FormRHSFunction is called twice with the exactly same value of the dependent variable (u) > > before invoking TSRHSJacobianFn and then solving the linear system with KSP (u is then updated) > > >> 1 TS dt 0.141197 time 0.141197 >> Calling FormRHSFunction; ||u|| = 60.489405479528003 463.15635675129079 62.813532026019146 5.9789502719351155 >> Calling FormRHSFunction; ||u|| = 60.489405479528003 463.15635675129079 62.813532026019146 5.9789502719351155 >> Calling TSRHSJacobianFn with ||u|| = 60.489405479528003 463.15635675129079 62.813532026019146 5.9789502719351155 >> Linear solve converged due to CONVERGED_RTOL iterations 9 >> Calling FormRHSFunction; ||u|| = 60.489405476878737 463.15635675602186 62.813532031616965 5.9789502710519811 >> 2 TS dt 0.155917 time 0.297115 >> Calling FormRHSFunction; ||u|| = 60.489405476878737 463.15635675602186 62.813532031616965 5.9789502710519811 >> Calling FormRHSFunction; ||u|| = 60.489405476878737 463.15635675602186 62.813532031616965 5.9789502710519811 >> Calling TSRHSJacobianFn with ||u|| = 60.489405476878737 463.15635675602186 62.813532031616965 5.9789502710519811 >> > Why is it so? Am I possibly misusing (or missing) some options? > > My .petscrc file is attached. > > Thank you, > > Aldo > > -- > Dr. Aldo Bonfiglioli > Associate professor of Fluid Mechanics > Dipartimento di Ingegneria > Universita' della Basilicata > V.le dell'Ateneo Lucano, 10 85100 Potenza ITALY > tel:+39.0971.205203 fax:+39.0971.205215 > web: https://urldefense.us/v3/__http://docenti.unibas.it/site/home/docente.html?m=002423__;!!G_uCfscf7eWS!acYbzJhys-2ITKblPtrsEnhe2r7CNdFSDaXF2nMHOEw5-ee_VvldwIA-zUDXIwE1-ygMTV054Xm0g3bENdHOkmXWePU$ ? -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: dotpetscrc.txt URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: From Pierre.LEDAC at cea.fr Sat Aug 30 12:16:08 2025 From: Pierre.LEDAC at cea.fr (LEDAC Pierre) Date: Sat, 30 Aug 2025 17:16:08 +0000 Subject: [petsc-users] [MPI][GPU] Message-ID: Hello, My code is built with PETSc 3.23+OpenMPI 4.1.6 (Cuda support enabled) and profling indicates that MPI communications are done between GPUs in all the code except PETSc part where D2H transfers occur. I reproduced the PETSc issue with the example under src/ksp/ksp/tutorials/ex10 on 2 MPI ranks. See output in ex10.log Also below the Nsys system profiling on ex10 with D2H and H2D copies before/after MPI calls. Thanks for your help, [cid:fe096b00-e3b7-4aa3-9d78-d8a44adbe145] Pierre LEDAC Commissariat ? l??nergie atomique et aux ?nergies alternatives Centre de SACLAY DES/ISAS/DM2S/SGLS/LCAN B?timent 451 ? point courrier n?41 F-91191 Gif-sur-Yvette +33 1 69 08 04 03 +33 6 83 42 05 79 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pastedImage.png Type: image/png Size: 123750 bytes Desc: pastedImage.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ex10.log Type: text/x-log Size: 31221 bytes Desc: ex10.log URL: From bsmith at petsc.dev Sat Aug 30 14:47:07 2025 From: bsmith at petsc.dev (Barry Smith) Date: Sat, 30 Aug 2025 15:47:07 -0400 Subject: [petsc-users] [MPI][GPU] In-Reply-To: References: Message-ID: Did you try the additional option -vec_type cuda with ex10.c ? > On Aug 30, 2025, at 1:16?PM, LEDAC Pierre wrote: > > Hello, > > My code is built with PETSc 3.23+OpenMPI 4.1.6 (Cuda support enabled) and profling indicates that MPI communications are done between GPUs in all the code except PETSc part where D2H transfers occur. > > I reproduced the PETSc issue with the example under src/ksp/ksp/tutorials/ex10 on 2 MPI ranks. See output in ex10.log > > Also below the Nsys system profiling on ex10 with D2H and H2D copies before/after MPI calls. > > Thanks for your help, > > > > > Pierre LEDAC > Commissariat ? l??nergie atomique et aux ?nergies alternatives > Centre de SACLAY > DES/ISAS/DM2S/SGLS/LCAN > B?timent 451 ? point courrier n?41 > F-91191 Gif-sur-Yvette > +33 1 69 08 04 03 > +33 6 83 42 05 79 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aldo.bonfiglioli at unibas.it Sat Aug 30 16:42:19 2025 From: aldo.bonfiglioli at unibas.it (Aldo Bonfiglioli) Date: Sat, 30 Aug 2025 23:42:19 +0200 Subject: [petsc-users] Question about Pseudo Time Stepping In-Reply-To: References: <85e1d3ad-6a32-444e-94dd-b5ebb6cba819@unibas.it> Message-ID: Barry, thank you for the update. Aldo *Aldo Bonfiglioli* Professore associato di Fluidodinamica settore scientifico disciplinare IIND-01/F Dipartimento di Ingegneria Viale dell'Ateneo Lucano, 10 85100 Potenza Tel: 0971.205203 Il ven 29 ago 2025, 20:52 Barry Smith ha scritto: > > Aldo, > > I understand the situation you are facing. > > With pseudo-continuation, when you provide an RHS function, SNES is > computing udot - > F(u). Inside the SNESSolve(), specifically in TSComputeIFunction(), with > the lines > > if (!ts->ops->ifunction) { > ierr = TSComputeRHSFunction(ts,t,X,Y);CHKERRQ(ierr); > ierr = VecAYPX(Y,-1,Xdot);CHKERRQ(ierr); > > But for the original system, we care about the norm for F(u). So in > TSStep_Pseudo(), after the nonlinear solve > there is the line > > pseudo->fnorm = -1; /* The current norm is no longer valid, monitor > must recompute it. */ > > and further down > > if (pseudo->fnorm < 0) { > PetscCall(VecZeroEntries(pseudo->xdot)); > PetscCall(TSComputeIFunction(ts, ts->ptime, ts->vec_sol, pseudo->xdot, > pseudo->func, PETSC_FALSE)); > PetscCall(VecNorm(pseudo->func, NORM_2, &pseudo->fnorm)); > > this norm is needed to determine of the nonlinear system has converged > satisfactory (or if more pseudo time steps are needed). > > Hence, your FormRHSFunction gets called twice with identical input. > > To make matters even worse your FormRHSFunction is actually being called > THREE times with identical input. Because at the start of TSStep_Pseudo() > it calls TSPseudoComputeTimeStep(ts, &next_time_step) which by default > uses TSPseudoTimeStepDefault() which has the same chunk of code > > PetscCall(VecZeroEntries(pseudo->xdot)); > PetscCall(TSComputeIFunction(ts, ts->ptime, ts->vec_sol, pseudo->xdot, > pseudo->func, PETSC_FALSE)); > PetscCall(VecNorm(pseudo->func, NORM_2, &pseudo->fnorm)); > > And the same norm is also recomputed. > > Now the diff > > diff --git a/src/ts/impls/pseudo/posindep.c > b/src/ts/impls/pseudo/posindep.c > index 7c013435f98..813a28311e7 100644 > --- a/src/ts/impls/pseudo/posindep.c > +++ b/src/ts/impls/pseudo/posindep.c > @@ -660,9 +660,11 @@ PetscErrorCode TSPseudoTimeStepDefault(TS ts, > PetscReal *newdt, void *dtctx) > PetscReal inc = pseudo->dt_increment; > > PetscFunctionBegin; > - PetscCall(VecZeroEntries(pseudo->xdot)); > - PetscCall(TSComputeIFunction(ts, ts->ptime, ts->vec_sol, pseudo->xdot, > pseudo->func, PETSC_FALSE)); > - PetscCall(VecNorm(pseudo->func, NORM_2, &pseudo->fnorm)); > + if (pseudo->fnorm < 0.0) { > + PetscCall(VecZeroEntries(pseudo->xdot)); > + PetscCall(TSComputeIFunction(ts, ts->ptime, ts->vec_sol, > pseudo->xdot, pseudo->func, PETSC_FALSE)); > + PetscCall(VecNorm(pseudo->func, NORM_2, &pseudo->fnorm)); > + } > if (pseudo->fnorm_initial < 0) { > /* first time through so compute initial function norm */ > pseudo->fnorm_initial = pseudo->fnorm > > will prevent the THIRD computation of the same function evaluation. So > this is a good optimization > > But eliminating the second computation is not so trivial. To review, deep > inside the TS code there is > > TSComputeRHSFunction(ts,t,X,Y); > VecAYPX(Y,-1,Xdot); > > This code knows nothing about TSPPseudo etc. > > Then later in the Pseudo code > > PetscCall(VecZeroEntries(pseudo->xdot)); > PetscCall(TSComputeIFunction(ts, ts->ptime, ts->vec_sol, pseudo->xdot, > pseudo->func, PETSC_FALSE)); > PetscCall(VecNorm(pseudo->func, NORM_2, &pseudo->fnorm)); > > again calls TSComputeRHSFunction(ts,t,X,Y); with the same X. (since Xdot > is not used directly by TSComputeRHSFunction()). > > Now if all the code was custom for Pseudo we could simply put in a line > > TSComputeRHSFunction(ts,t,X,Y); > VecNorm(Y,NORM_2, &pseudo->fnorm) > VecAYPX(Y,-1,Xdot); > > into TSComputeIFunction() and have the norm available later for pseudo. > > It is not clear that there is any definitive benefit for pseudo to utilize > the TS framework. It possibly gets some code reuse from TS but at the cost > of extra code that won't be needed if written directly. Plus this > performance hit. > > Anyways I will make a merge request to remove the third redundant > computation of FormRHSFunction but will await input from others on if > anything else could be done to remove the inefficiency. > > Thanks for reporting this challenging problem. > > Barry > > Note: I don't think providing the Jacobian improves the efficiency, it is > just better at hiding the redundant computation. > > > > > > > > > On Aug 26, 2025, at 4:21?AM, Aldo Bonfiglioli > wrote: > > Hi there, > > I am using -ts_type pseudo to find steady solutions of the Euler/NS PDEs. > > It looks like FormRHSFunction is called twice with the exactly same value > of the dependent variable (u) > > before invoking TSRHSJacobianFn and then solving the linear system with > KSP (u is then updated) > > 1 TS dt 0.141197 time 0.141197 > Calling FormRHSFunction; ||u|| = 60.489405479528003 > 463.15635675129079 62.813532026019146 > 5.9789502719351155 > Calling FormRHSFunction; ||u|| = 60.489405479528003 > 463.15635675129079 62.813532026019146 > 5.9789502719351155 > Calling TSRHSJacobianFn with ||u|| = 60.489405479528003 > 463.15635675129079 62.813532026019146 > 5.9789502719351155 > Linear solve converged due to CONVERGED_RTOL iterations 9 > Calling FormRHSFunction; ||u|| = 60.489405476878737 > 463.15635675602186 62.813532031616965 5.9789502710519811 > 2 TS dt 0.155917 time 0.297115 > Calling FormRHSFunction; ||u|| = 60.489405476878737 > 463.15635675602186 62.813532031616965 5.9789502710519811 > Calling FormRHSFunction; ||u|| = 60.489405476878737 > 463.15635675602186 62.813532031616965 5.9789502710519811 > Calling TSRHSJacobianFn with ||u|| = 60.489405476878737 > 463.15635675602186 62.813532031616965 5.9789502710519811 > > Why is it so? Am I possibly misusing (or missing) some options? > > My .petscrc file is attached. > > Thank you, > > Aldo > > -- > Dr. Aldo Bonfiglioli > Associate professor of Fluid Mechanics > Dipartimento di Ingegneria > Universita' della Basilicata > V.le dell'Ateneo Lucano, 10 85100 Potenza ITALY > tel:+39.0971.205203 fax:+39.0971.205215 > web: https://urldefense.us/v3/__http://docenti.unibas.it/site/home/docente.html?m=002423__;!!G_uCfscf7eWS!cZ77DpN-ZOmpsgsHdMBQK2uUmuwgxPNkFjRagyCeSOxwLdNzcF4P6MSOrFAI0DkC-mAR9S9bUYDAYLbaid8o_doR6zJYMQyUmBE$ > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From barryfsmith at icloud.com Sat Aug 30 22:18:24 2025 From: barryfsmith at icloud.com (Barry Smith) Date: Sat, 30 Aug 2025 23:18:24 -0400 Subject: [petsc-users] Question about Pseudo Time Stepping In-Reply-To: References: <85e1d3ad-6a32-444e-94dd-b5ebb6cba819@unibas.it> Message-ID: <4DD4EF03-30EF-48A7-87DC-70E35E97FCC8@icloud.com> I always like a challenge. The MR https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/merge_requests/8675__;!!G_uCfscf7eWS!eEJyFKOE1fIr1zG138qFEBHnEAPDDHz6-nhpA3l8TnLtt8VK3i98Cc13eRy7zJGgmWBtGQ8IB0yIDNjR7M6rtACjJrE$ should eliminate both redundant TSComputeRHSFunction() calls in your code and thus decrease its run time. Thanks for bringing the performance bug to our attention Barry > On Aug 30, 2025, at 5:42?PM, Aldo Bonfiglioli wrote: > > Barry, > thank you for the update. > Aldo > > > Aldo Bonfiglioli > Professore associato di Fluidodinamica settore scientifico disciplinare IIND-01/F > Dipartimento di Ingegneria > Viale dell'Ateneo Lucano, 10 85100 Potenza > Tel: 0971.205203 > > > Il ven 29 ago 2025, 20:52 Barry Smith > ha scritto: >> >> Aldo, >> >> I understand the situation you are facing. >> >> With pseudo-continuation, when you provide an RHS function, SNES is computing udot - >> F(u). Inside the SNESSolve(), specifically in TSComputeIFunction(), with the lines >> >> if (!ts->ops->ifunction) { >> ierr = TSComputeRHSFunction(ts,t,X,Y);CHKERRQ(ierr); >> ierr = VecAYPX(Y,-1,Xdot);CHKERRQ(ierr); >> >> But for the original system, we care about the norm for F(u). So in TSStep_Pseudo(), after the nonlinear solve >> there is the line >> >> pseudo->fnorm = -1; /* The current norm is no longer valid, monitor must recompute it. */ >> >> and further down >> >> if (pseudo->fnorm < 0) { >> PetscCall(VecZeroEntries(pseudo->xdot)); >> PetscCall(TSComputeIFunction(ts, ts->ptime, ts->vec_sol, pseudo->xdot, pseudo->func, PETSC_FALSE)); >> PetscCall(VecNorm(pseudo->func, NORM_2, &pseudo->fnorm)); >> >> this norm is needed to determine of the nonlinear system has converged satisfactory (or if more pseudo time steps are needed). >> >> Hence, your FormRHSFunction gets called twice with identical input. >> >> To make matters even worse your FormRHSFunction is actually being called THREE times with identical input. Because at the start of TSStep_Pseudo() it calls TSPseudoComputeTimeStep(ts, &next_time_step) which by default uses TSPseudoTimeStepDefault() which has the same chunk of code >> >> PetscCall(VecZeroEntries(pseudo->xdot)); >> PetscCall(TSComputeIFunction(ts, ts->ptime, ts->vec_sol, pseudo->xdot, pseudo->func, PETSC_FALSE)); >> PetscCall(VecNorm(pseudo->func, NORM_2, &pseudo->fnorm)); >> >> And the same norm is also recomputed. >> >> Now the diff >> >> diff --git a/src/ts/impls/pseudo/posindep.c b/src/ts/impls/pseudo/posindep.c >> index 7c013435f98..813a28311e7 100644 >> --- a/src/ts/impls/pseudo/posindep.c >> +++ b/src/ts/impls/pseudo/posindep.c >> @@ -660,9 +660,11 @@ PetscErrorCode TSPseudoTimeStepDefault(TS ts, PetscReal *newdt, void *dtctx) >> PetscReal inc = pseudo->dt_increment; >> >> PetscFunctionBegin; >> - PetscCall(VecZeroEntries(pseudo->xdot)); >> - PetscCall(TSComputeIFunction(ts, ts->ptime, ts->vec_sol, pseudo->xdot, pseudo->func, PETSC_FALSE)); >> - PetscCall(VecNorm(pseudo->func, NORM_2, &pseudo->fnorm)); >> + if (pseudo->fnorm < 0.0) { >> + PetscCall(VecZeroEntries(pseudo->xdot)); >> + PetscCall(TSComputeIFunction(ts, ts->ptime, ts->vec_sol, pseudo->xdot, pseudo->func, PETSC_FALSE)); >> + PetscCall(VecNorm(pseudo->func, NORM_2, &pseudo->fnorm)); >> + } >> if (pseudo->fnorm_initial < 0) { >> /* first time through so compute initial function norm */ >> pseudo->fnorm_initial = pseudo->fnorm >> >> will prevent the THIRD computation of the same function evaluation. So this is a good optimization >> >> But eliminating the second computation is not so trivial. To review, deep inside the TS code there is >> >> TSComputeRHSFunction(ts,t,X,Y); >> VecAYPX(Y,-1,Xdot); >> >> This code knows nothing about TSPPseudo etc. >> >> Then later in the Pseudo code >> >> PetscCall(VecZeroEntries(pseudo->xdot)); >> PetscCall(TSComputeIFunction(ts, ts->ptime, ts->vec_sol, pseudo->xdot, pseudo->func, PETSC_FALSE)); >> PetscCall(VecNorm(pseudo->func, NORM_2, &pseudo->fnorm)); >> >> again calls TSComputeRHSFunction(ts,t,X,Y); with the same X. (since Xdot is not used directly by TSComputeRHSFunction()). >> >> Now if all the code was custom for Pseudo we could simply put in a line >> >> TSComputeRHSFunction(ts,t,X,Y); >> VecNorm(Y,NORM_2, &pseudo->fnorm) >> VecAYPX(Y,-1,Xdot); >> >> into TSComputeIFunction() and have the norm available later for pseudo. >> >> It is not clear that there is any definitive benefit for pseudo to utilize the TS framework. It possibly gets some code reuse from TS but at the cost of extra code that won't be needed if written directly. Plus this performance hit. >> >> Anyways I will make a merge request to remove the third redundant computation of FormRHSFunction but will await input from others on if anything else could be done to remove the inefficiency. >> >> Thanks for reporting this challenging problem. >> >> Barry >> >> Note: I don't think providing the Jacobian improves the efficiency, it is just better at hiding the redundant computation. >> >> >> >> >> >> >> >> >>> On Aug 26, 2025, at 4:21?AM, Aldo Bonfiglioli > wrote: >>> >>> Hi there, >>> >>> I am using -ts_type pseudo to find steady solutions of the Euler/NS PDEs. >>> >>> It looks like FormRHSFunction is called twice with the exactly same value of the dependent variable (u) >>> >>> before invoking TSRHSJacobianFn and then solving the linear system with KSP (u is then updated) >>> >>> >>>> 1 TS dt 0.141197 time 0.141197 >>>> Calling FormRHSFunction; ||u|| = 60.489405479528003 463.15635675129079 62.813532026019146 5.9789502719351155 >>>> Calling FormRHSFunction; ||u|| = 60.489405479528003 463.15635675129079 62.813532026019146 5.9789502719351155 >>>> Calling TSRHSJacobianFn with ||u|| = 60.489405479528003 463.15635675129079 62.813532026019146 5.9789502719351155 >>>> Linear solve converged due to CONVERGED_RTOL iterations 9 >>>> Calling FormRHSFunction; ||u|| = 60.489405476878737 463.15635675602186 62.813532031616965 5.9789502710519811 >>>> 2 TS dt 0.155917 time 0.297115 >>>> Calling FormRHSFunction; ||u|| = 60.489405476878737 463.15635675602186 62.813532031616965 5.9789502710519811 >>>> Calling FormRHSFunction; ||u|| = 60.489405476878737 463.15635675602186 62.813532031616965 5.9789502710519811 >>>> Calling TSRHSJacobianFn with ||u|| = 60.489405476878737 463.15635675602186 62.813532031616965 5.9789502710519811 >>>> >>> Why is it so? Am I possibly misusing (or missing) some options? >>> >>> My .petscrc file is attached. >>> >>> Thank you, >>> >>> Aldo >>> >>> -- >>> Dr. Aldo Bonfiglioli >>> Associate professor of Fluid Mechanics >>> Dipartimento di Ingegneria >>> Universita' della Basilicata >>> V.le dell'Ateneo Lucano, 10 85100 Potenza ITALY >>> tel:+39.0971.205203 fax:+39.0971.205215 >>> web: https://urldefense.us/v3/__http://docenti.unibas.it/site/home/docente.html?m=002423__;!!G_uCfscf7eWS!eEJyFKOE1fIr1zG138qFEBHnEAPDDHz6-nhpA3l8TnLtt8VK3i98Cc13eRy7zJGgmWBtGQ8IB0yIDNjR7M6rchJhC0A$ >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From Pierre.LEDAC at cea.fr Sun Aug 31 03:32:40 2025 From: Pierre.LEDAC at cea.fr (LEDAC Pierre) Date: Sun, 31 Aug 2025 08:32:40 +0000 Subject: [petsc-users] [MPI][GPU] In-Reply-To: References: , Message-ID: Yes, but was surprised it was not used, so I removed it (same for -vec_type mpicuda) mpirun -np 2 ./ex10 2 -f Matrix_3133717_rows_1_cpus.petsc -ksp_view -log_view -ksp_monitor -ksp_type cg -pc_type hypre -pc_hypre_type boomeramg -pc_hypre_boomeramg_strong_threshold 0.7 -mat_type aijcusparse -vec_type cuda ... WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! There is one unused database option. It is: Option left: name:-vec_type value: cuda source: command lin Pierre LEDAC Commissariat ? l??nergie atomique et aux ?nergies alternatives Centre de SACLAY DES/ISAS/DM2S/SGLS/LCAN B?timent 451 ? point courrier n?41 F-91191 Gif-sur-Yvette +33 1 69 08 04 03 +33 6 83 42 05 79 ________________________________ De : Barry Smith Envoy? : samedi 30 ao?t 2025 21:47:07 ? : LEDAC Pierre Cc : petsc-users at mcs.anl.gov Objet : Re: [petsc-users] [MPI][GPU] Did you try the additional option -vec_type cuda with ex10.c ? On Aug 30, 2025, at 1:16?PM, LEDAC Pierre wrote: Hello, My code is built with PETSc 3.23+OpenMPI 4.1.6 (Cuda support enabled) and profling indicates that MPI communications are done between GPUs in all the code except PETSc part where D2H transfers occur. I reproduced the PETSc issue with the example under src/ksp/ksp/tutorials/ex10 on 2 MPI ranks. See output in ex10.log Also below the Nsys system profiling on ex10 with D2H and H2D copies before/after MPI calls. Thanks for your help, Pierre LEDAC Commissariat ? l??nergie atomique et aux ?nergies alternatives Centre de SACLAY DES/ISAS/DM2S/SGLS/LCAN B?timent 451 ? point courrier n?41 F-91191 Gif-sur-Yvette +33 1 69 08 04 03 +33 6 83 42 05 79 -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Sun Aug 31 09:33:38 2025 From: bsmith at petsc.dev (Barry Smith) Date: Sun, 31 Aug 2025 10:33:38 -0400 Subject: [petsc-users] [MPI][GPU] In-Reply-To: References: Message-ID: Ahh, that ex10.c is missing a VecSetFromOptions() call before the VecLoad() and friends. In contrast, the matrix has a MatSetFromOptions(). Can you try adding it to ex10.c and see if that resolves the problem with ex10.c (and may be a path forward for your code)? Barry > On Aug 31, 2025, at 4:32?AM, LEDAC Pierre wrote: > > Yes, but was surprised it was not used, so I removed it (same for -vec_type mpicuda) > > mpirun -np 2 ./ex10 2 -f Matrix_3133717_rows_1_cpus.petsc -ksp_view -log_view -ksp_monitor -ksp_type cg -pc_type hypre -pc_hypre_type boomeramg -pc_hypre_boomeramg_strong_threshold 0.7 -mat_type aijcusparse -vec_type cuda > ... > WARNING! There are options you set that were not used! > WARNING! could be spelling mistake, etc! > There is one unused database option. It is: > Option left: name:-vec_type value: cuda source: command lin > > Pierre LEDAC > Commissariat ? l??nergie atomique et aux ?nergies alternatives > Centre de SACLAY > DES/ISAS/DM2S/SGLS/LCAN > B?timent 451 ? point courrier n?41 > F-91191 Gif-sur-Yvette > +33 1 69 08 04 03 > +33 6 83 42 05 79 > De : Barry Smith > > Envoy? : samedi 30 ao?t 2025 21:47:07 > ? : LEDAC Pierre > Cc : petsc-users at mcs.anl.gov > Objet : Re: [petsc-users] [MPI][GPU] > > > Did you try the additional option -vec_type cuda with ex10.c ? > > > >> On Aug 30, 2025, at 1:16?PM, LEDAC Pierre > wrote: >> >> Hello, >> >> My code is built with PETSc 3.23+OpenMPI 4.1.6 (Cuda support enabled) and profling indicates that MPI communications are done between GPUs in all the code except PETSc part where D2H transfers occur. >> >> I reproduced the PETSc issue with the example under src/ksp/ksp/tutorials/ex10 on 2 MPI ranks. See output in ex10.log >> >> Also below the Nsys system profiling on ex10 with D2H and H2D copies before/after MPI calls. >> >> Thanks for your help, >> >> >> >> >> Pierre LEDAC >> Commissariat ? l??nergie atomique et aux ?nergies alternatives >> Centre de SACLAY >> DES/ISAS/DM2S/SGLS/LCAN >> B?timent 451 ? point courrier n?41 >> F-91191 Gif-sur-Yvette >> +33 1 69 08 04 03 >> +33 6 83 42 05 79 >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From Pierre.LEDAC at cea.fr Sun Aug 31 12:13:36 2025 From: Pierre.LEDAC at cea.fr (LEDAC Pierre) Date: Sun, 31 Aug 2025 17:13:36 +0000 Subject: [petsc-users] [MPI][GPU] In-Reply-To: References: , Message-ID: Barry, It solved the unrecognized option but still exchanging MPI messages through the host. I switch to a simpler test case without reading a matrix (src/ksp/ksp/tutorials/ex46.c) but get the same behaviour. In the Nsys profile for ex46, the MPI synchronizations occurs during PCApply so now I am wondering if the issue is related to the fact than Hypre is not configured/enabled with MPI GPU-Aware in the PETSc build. I will give a try with --enable-gpu-aware-mpi passed to Hypre. Do you know an example in PETSc which specifically bench with/without Cuda-Aware enabled for MPI ? [cid:93d886d4-28ec-4b73-b602-3a559fffd435] Pierre LEDAC Commissariat ? l??nergie atomique et aux ?nergies alternatives Centre de SACLAY DES/ISAS/DM2S/SGLS/LCAN B?timent 451 ? point courrier n?41 F-91191 Gif-sur-Yvette +33 1 69 08 04 03 +33 6 83 42 05 79 ________________________________ De : Barry Smith Envoy? : dimanche 31 ao?t 2025 16:33:38 ? : LEDAC Pierre Cc : petsc-users at mcs.anl.gov Objet : Re: [petsc-users] [MPI][GPU] Ahh, that ex10.c is missing a VecSetFromOptions() call before the VecLoad() and friends. In contrast, the matrix has a MatSetFromOptions(). Can you try adding it to ex10.c and see if that resolves the problem with ex10.c (and may be a path forward for your code)? Barry On Aug 31, 2025, at 4:32?AM, LEDAC Pierre wrote: Yes, but was surprised it was not used, so I removed it (same for -vec_type mpicuda) mpirun -np 2 ./ex10 2 -f Matrix_3133717_rows_1_cpus.petsc -ksp_view -log_view -ksp_monitor -ksp_type cg -pc_type hypre -pc_hypre_type boomeramg -pc_hypre_boomeramg_strong_threshold 0.7 -mat_type aijcusparse -vec_type cuda ... WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! There is one unused database option. It is: Option left: name:-vec_type value: cuda source: command lin Pierre LEDAC Commissariat ? l??nergie atomique et aux ?nergies alternatives Centre de SACLAY DES/ISAS/DM2S/SGLS/LCAN B?timent 451 ? point courrier n?41 F-91191 Gif-sur-Yvette +33 1 69 08 04 03 +33 6 83 42 05 79 ________________________________ De : Barry Smith > Envoy? : samedi 30 ao?t 2025 21:47:07 ? : LEDAC Pierre Cc : petsc-users at mcs.anl.gov Objet : Re: [petsc-users] [MPI][GPU] Did you try the additional option -vec_type cuda with ex10.c ? On Aug 30, 2025, at 1:16?PM, LEDAC Pierre > wrote: Hello, My code is built with PETSc 3.23+OpenMPI 4.1.6 (Cuda support enabled) and profling indicates that MPI communications are done between GPUs in all the code except PETSc part where D2H transfers occur. I reproduced the PETSc issue with the example under src/ksp/ksp/tutorials/ex10 on 2 MPI ranks. See output in ex10.log Also below the Nsys system profiling on ex10 with D2H and H2D copies before/after MPI calls. Thanks for your help, Pierre LEDAC Commissariat ? l??nergie atomique et aux ?nergies alternatives Centre de SACLAY DES/ISAS/DM2S/SGLS/LCAN B?timent 451 ? point courrier n?41 F-91191 Gif-sur-Yvette +33 1 69 08 04 03 +33 6 83 42 05 79 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pastedImage.png Type: image/png Size: 65351 bytes Desc: pastedImage.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ex46.log Type: text/x-log Size: 77204 bytes Desc: ex46.log URL: From Pierre.LEDAC at cea.fr Sun Aug 31 12:30:56 2025 From: Pierre.LEDAC at cea.fr (LEDAC Pierre) Date: Sun, 31 Aug 2025 17:30:56 +0000 Subject: [petsc-users] [MPI][GPU] In-Reply-To: References: , , Message-ID: Ok, I just tried --enable-gpu-aware-mpi passed to Hypre, Hypre_config.h defines now HYPRE_USING_GPU_AWARE_MPI 1 But still no D2D copy near MPI calls in ex46.c example. Probably an obvious thing I forgot during PETSc configure, but I don't see... Pierre LEDAC Commissariat ? l??nergie atomique et aux ?nergies alternatives Centre de SACLAY DES/ISAS/DM2S/SGLS/LCAN B?timent 451 ? point courrier n?41 F-91191 Gif-sur-Yvette +33 1 69 08 04 03 +33 6 83 42 05 79 ________________________________ De : LEDAC Pierre Envoy? : dimanche 31 ao?t 2025 19:13:36 ? : Barry Smith Cc : petsc-users at mcs.anl.gov Objet : RE: [petsc-users] [MPI][GPU] Barry, It solved the unrecognized option but still exchanging MPI messages through the host. I switch to a simpler test case without reading a matrix (src/ksp/ksp/tutorials/ex46.c) but get the same behaviour. In the Nsys profile for ex46, the MPI synchronizations occurs during PCApply so now I am wondering if the issue is related to the fact than Hypre is not configured/enabled with MPI GPU-Aware in the PETSc build. I will give a try with --enable-gpu-aware-mpi passed to Hypre. Do you know an example in PETSc which specifically bench with/without Cuda-Aware enabled for MPI ? [cid:93d886d4-28ec-4b73-b602-3a559fffd435] Pierre LEDAC Commissariat ? l??nergie atomique et aux ?nergies alternatives Centre de SACLAY DES/ISAS/DM2S/SGLS/LCAN B?timent 451 ? point courrier n?41 F-91191 Gif-sur-Yvette +33 1 69 08 04 03 +33 6 83 42 05 79 ________________________________ De : Barry Smith Envoy? : dimanche 31 ao?t 2025 16:33:38 ? : LEDAC Pierre Cc : petsc-users at mcs.anl.gov Objet : Re: [petsc-users] [MPI][GPU] Ahh, that ex10.c is missing a VecSetFromOptions() call before the VecLoad() and friends. In contrast, the matrix has a MatSetFromOptions(). Can you try adding it to ex10.c and see if that resolves the problem with ex10.c (and may be a path forward for your code)? Barry On Aug 31, 2025, at 4:32?AM, LEDAC Pierre wrote: Yes, but was surprised it was not used, so I removed it (same for -vec_type mpicuda) mpirun -np 2 ./ex10 2 -f Matrix_3133717_rows_1_cpus.petsc -ksp_view -log_view -ksp_monitor -ksp_type cg -pc_type hypre -pc_hypre_type boomeramg -pc_hypre_boomeramg_strong_threshold 0.7 -mat_type aijcusparse -vec_type cuda ... WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! There is one unused database option. It is: Option left: name:-vec_type value: cuda source: command lin Pierre LEDAC Commissariat ? l??nergie atomique et aux ?nergies alternatives Centre de SACLAY DES/ISAS/DM2S/SGLS/LCAN B?timent 451 ? point courrier n?41 F-91191 Gif-sur-Yvette +33 1 69 08 04 03 +33 6 83 42 05 79 ________________________________ De : Barry Smith > Envoy? : samedi 30 ao?t 2025 21:47:07 ? : LEDAC Pierre Cc : petsc-users at mcs.anl.gov Objet : Re: [petsc-users] [MPI][GPU] Did you try the additional option -vec_type cuda with ex10.c ? On Aug 30, 2025, at 1:16?PM, LEDAC Pierre > wrote: Hello, My code is built with PETSc 3.23+OpenMPI 4.1.6 (Cuda support enabled) and profling indicates that MPI communications are done between GPUs in all the code except PETSc part where D2H transfers occur. I reproduced the PETSc issue with the example under src/ksp/ksp/tutorials/ex10 on 2 MPI ranks. See output in ex10.log Also below the Nsys system profiling on ex10 with D2H and H2D copies before/after MPI calls. Thanks for your help, Pierre LEDAC Commissariat ? l??nergie atomique et aux ?nergies alternatives Centre de SACLAY DES/ISAS/DM2S/SGLS/LCAN B?timent 451 ? point courrier n?41 F-91191 Gif-sur-Yvette +33 1 69 08 04 03 +33 6 83 42 05 79 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pastedImage.png Type: image/png Size: 65351 bytes Desc: pastedImage.png URL: From bsmith at petsc.dev Sun Aug 31 17:32:23 2025 From: bsmith at petsc.dev (Barry Smith) Date: Sun, 31 Aug 2025 18:32:23 -0400 Subject: [petsc-users] [MPI][GPU] In-Reply-To: References: Message-ID: <84040E99-73D3-4F3F-BDF3-C942CD51FD92@petsc.dev> Can you pinpoint the MPI calls (and the routines in PETSc or hypre that they are in) that are not using CUDA-aware message passing? That is, inside the MPI call, they are copying to host memory and doing the needed inter-process communication from there? I do not understand the graphics you have sent. Barry Or is it possible the buffers passed to MPI are not on the GPU and so naturally do the MPI from host memory? If so, where? > On Aug 31, 2025, at 1:30?PM, LEDAC Pierre wrote: > > Ok, I just tried --enable-gpu-aware-mpi passed to Hypre, > > Hypre_config.h defines now HYPRE_USING_GPU_AWARE_MPI 1 > > But still no D2D copy near MPI calls in ex46.c example. > > Probably an obvious thing I forgot during PETSc configure, but I don't see... > > Pierre LEDAC > Commissariat ? l??nergie atomique et aux ?nergies alternatives > Centre de SACLAY > DES/ISAS/DM2S/SGLS/LCAN > B?timent 451 ? point courrier n?41 > F-91191 Gif-sur-Yvette > +33 1 69 08 04 03 > +33 6 83 42 05 79 > De : LEDAC Pierre > Envoy? : dimanche 31 ao?t 2025 19:13:36 > ? : Barry Smith > Cc : petsc-users at mcs.anl.gov > Objet : RE: [petsc-users] [MPI][GPU] > > Barry, > > It solved the unrecognized option but still exchanging MPI messages through the host. > > I switch to a simpler test case without reading a matrix (src/ksp/ksp/tutorials/ex46.c) but get the same behaviour. > > In the Nsys profile for ex46, the MPI synchronizations occurs during PCApply so now I am wondering if the issue is related to the fact than Hypre is not configured/enabled with MPI GPU-Aware in the PETSc build. > I will give a try with --enable-gpu-aware-mpi passed to Hypre. > > Do you know an example in PETSc which specifically bench with/without Cuda-Aware enabled for MPI ? > > > > Pierre LEDAC > Commissariat ? l??nergie atomique et aux ?nergies alternatives > Centre de SACLAY > DES/ISAS/DM2S/SGLS/LCAN > B?timent 451 ? point courrier n?41 > F-91191 Gif-sur-Yvette > +33 1 69 08 04 03 > +33 6 83 42 05 79 > De : Barry Smith > > Envoy? : dimanche 31 ao?t 2025 16:33:38 > ? : LEDAC Pierre > Cc : petsc-users at mcs.anl.gov > Objet : Re: [petsc-users] [MPI][GPU] > > > Ahh, that ex10.c is missing a VecSetFromOptions() call before the VecLoad() and friends. In contrast, the matrix has a MatSetFromOptions(). Can you try adding it to ex10.c and see if that resolves the problem with ex10.c (and may be a path forward for your code)? > > Barry > > >> On Aug 31, 2025, at 4:32?AM, LEDAC Pierre > wrote: >> >> Yes, but was surprised it was not used, so I removed it (same for -vec_type mpicuda) >> >> mpirun -np 2 ./ex10 2 -f Matrix_3133717_rows_1_cpus.petsc -ksp_view -log_view -ksp_monitor -ksp_type cg -pc_type hypre -pc_hypre_type boomeramg -pc_hypre_boomeramg_strong_threshold 0.7 -mat_type aijcusparse -vec_type cuda >> ... >> WARNING! There are options you set that were not used! >> WARNING! could be spelling mistake, etc! >> There is one unused database option. It is: >> Option left: name:-vec_type value: cuda source: command lin >> >> Pierre LEDAC >> Commissariat ? l??nergie atomique et aux ?nergies alternatives >> Centre de SACLAY >> DES/ISAS/DM2S/SGLS/LCAN >> B?timent 451 ? point courrier n?41 >> F-91191 Gif-sur-Yvette >> +33 1 69 08 04 03 >> +33 6 83 42 05 79 >> De : Barry Smith > >> Envoy? : samedi 30 ao?t 2025 21:47:07 >> ? : LEDAC Pierre >> Cc : petsc-users at mcs.anl.gov >> Objet : Re: [petsc-users] [MPI][GPU] >> >> >> Did you try the additional option -vec_type cuda with ex10.c ? >> >> >> >>> On Aug 30, 2025, at 1:16?PM, LEDAC Pierre > wrote: >>> >>> Hello, >>> >>> My code is built with PETSc 3.23+OpenMPI 4.1.6 (Cuda support enabled) and profling indicates that MPI communications are done between GPUs in all the code except PETSc part where D2H transfers occur. >>> >>> I reproduced the PETSc issue with the example under src/ksp/ksp/tutorials/ex10 on 2 MPI ranks. See output in ex10.log >>> >>> Also below the Nsys system profiling on ex10 with D2H and H2D copies before/after MPI calls. >>> >>> Thanks for your help, >>> >>> >>> >>> >>> Pierre LEDAC >>> Commissariat ? l??nergie atomique et aux ?nergies alternatives >>> Centre de SACLAY >>> DES/ISAS/DM2S/SGLS/LCAN >>> B?timent 451 ? point courrier n?41 >>> F-91191 Gif-sur-Yvette >>> +33 1 69 08 04 03 >>> +33 6 83 42 05 79 >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: