From sayosale at hotmail.com Mon Sep 2 01:56:10 2024 From: sayosale at hotmail.com (dazza simplythebest) Date: Mon, 2 Sep 2024 06:56:10 +0000 Subject: [petsc-users] How to create an array of vectors of type Vec (for use with slepc EPSGetInvariantSubspace) Message-ID: Dear All, I am seeking to perform a large number of eigenvalue calculations (related to a fluid dynamics problem) using the slepc framework. I have written the program that calls slepc and validated its solutions against known results for special cases - the Jacobi-Davidson iteration seemed to perform best on this problem. Since in the 'production-level' calculation I will step through a parameter space which should mean only small changes to the eigenproblem between each calculation, it seems to make sense to use the previous solution as the initial guess for the next calculation. The sequence to execute this strategy would seem to be: To write out the current soln subspace: call EPSGetInvariantSubspace [ call VecView -> binary file (if a file storage is required for possible restart)] To then read in this data to use it as the initial subspace for the next run: call EPSSetInitialSpace My question is that EPSGetInvariantSubspace needs an 'array of nconv vectors', how does one create such an array ? My understanding is that the Vec structure is just a one-dimensional array (with contents possibly scattered in non-contiguous memory locations !?) , so do we just need to create a long Vec that stores multiple vectors ? How can we do this (the examples I have seen thus far use a MatCreateVecs to specify the size of the Vec)? If loading such a subspace from a binary file (created maybe by VecView), presumably VecLoad is the best way to load such an array of vectors from the binary file ? Best wishes and many thanks, Dan. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Mon Sep 2 02:06:06 2024 From: jroman at dsic.upv.es (Jose E. Roman) Date: Mon, 2 Sep 2024 07:06:06 +0000 Subject: [petsc-users] How to create an array of vectors of type Vec (for use with slepc EPSGetInvariantSubspace) In-Reply-To: References: Message-ID: <01AFD9AF-F905-435F-AEF0-CF1E6B793A5F@dsic.upv.es> You can use VecDuplicateVecs() to create an array of Vec. Yes, you can use VecLoad() to read the data from a binary file. Jose > El 2 sept 2024, a las 8:56, dazza simplythebest escribi?: > > Dear All, > I am seeking to perform a large number of eigenvalue calculations (related to a fluid dynamics > problem) using the slepc framework. I have written the program that calls slepc and validated its solutions against > known results for special cases - the Jacobi-Davidson iteration seemed to perform best on this problem. > Since in the 'production-level' calculation I will step through a parameter space which should > mean only small changes to the eigenproblem between each calculation, it seems to make sense > to use the previous solution as the initial guess for the next calculation. > > The sequence to execute this strategy would seem to be: > To write out the current soln subspace: > > call EPSGetInvariantSubspace > [ call VecView -> binary file (if a file storage is required for possible restart)] > > To then read in this data to use it as the initial subspace for the next run: > > call EPSSetInitialSpace > > My question is that EPSGetInvariantSubspace needs an 'array of nconv vectors', how does one create > such an array ? My understanding is that the Vec structure is just a one-dimensional array (with contents > possibly scattered in non-contiguous memory locations !?) , so do we just need to create a long Vec > that stores multiple vectors ? How can we do this (the examples I have seen thus far use a MatCreateVecs to specify the size > of the Vec)? > > If loading such a subspace from a binary file (created maybe by VecView), presumably VecLoad is the > best way to load such an array of vectors from the binary file ? > > Best wishes and many thanks, > Dan. From sayosale at hotmail.com Mon Sep 2 02:33:20 2024 From: sayosale at hotmail.com (dazza simplythebest) Date: Mon, 2 Sep 2024 07:33:20 +0000 Subject: [petsc-users] How to create an array of vectors of type Vec (for use with slepc EPSGetInvariantSubspace) In-Reply-To: <01AFD9AF-F905-435F-AEF0-CF1E6B793A5F@dsic.upv.es> References: <01AFD9AF-F905-435F-AEF0-CF1E6B793A5F@dsic.upv.es> Message-ID: Dear Jose, Many thanks for your speedy response ... and of course for creating this great software. The VecDuplicateVecs does seem to be the routine I was looking for! Best wishes, Dan. ________________________________ From: Jose E. Roman Sent: Monday, September 2, 2024 7:06 AM To: dazza simplythebest Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] How to create an array of vectors of type Vec (for use with slepc EPSGetInvariantSubspace) You can use VecDuplicateVecs() to create an array of Vec. Yes, you can use VecLoad() to read the data from a binary file. Jose > El 2 sept 2024, a las 8:56, dazza simplythebest escribi?: > > Dear All, > I am seeking to perform a large number of eigenvalue calculations (related to a fluid dynamics > problem) using the slepc framework. I have written the program that calls slepc and validated its solutions against > known results for special cases - the Jacobi-Davidson iteration seemed to perform best on this problem. > Since in the 'production-level' calculation I will step through a parameter space which should > mean only small changes to the eigenproblem between each calculation, it seems to make sense > to use the previous solution as the initial guess for the next calculation. > > The sequence to execute this strategy would seem to be: > To write out the current soln subspace: > > call EPSGetInvariantSubspace > [ call VecView -> binary file (if a file storage is required for possible restart)] > > To then read in this data to use it as the initial subspace for the next run: > > call EPSSetInitialSpace > > My question is that EPSGetInvariantSubspace needs an 'array of nconv vectors', how does one create > such an array ? My understanding is that the Vec structure is just a one-dimensional array (with contents > possibly scattered in non-contiguous memory locations !?) , so do we just need to create a long Vec > that stores multiple vectors ? How can we do this (the examples I have seen thus far use a MatCreateVecs to specify the size > of the Vec)? > > If loading such a subspace from a binary file (created maybe by VecView), presumably VecLoad is the > best way to load such an array of vectors from the binary file ? > > Best wishes and many thanks, > Dan. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bodhinandach at gmail.com Tue Sep 3 20:43:21 2024 From: bodhinandach at gmail.com (Bodhinanda Chandra) Date: Tue, 3 Sep 2024 18:43:21 -0700 Subject: [petsc-users] Installing PETSc in Ubuntu 24.04 - check test error Message-ID: Dear PETSc community, I recently tried to install PETSc in Ubuntu 24.04. All seems to work fine, but one of the check tests ex5f thow the following errors: ---------------------------------------------------------------------------------------------------- Running PETSc check examples to verify correct installation Using PETSC_DIR=/home/bchandra/Program/petsc and PETSC_ARCH=arch-linux-c-opt C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes C/C++ example src/snes/tutorials/ex19 run successfully with HYPRE *******************Error detected during compile or link!******************* See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!bNBtvML4qAKu9WGzBH2kRjtiP8AwN3ijY4qZngu7nvl_TUwz8pQ6WGyCJh0AtIEOqLHU5NHUJRL1oWg2FeTR0Oz6868$ /home/bchandra/Program/petsc/src/snes/tutorials ex5f ********************************************************* /usr/local/openmpi-4.1.5/bin/mpif90 -fPIC -Wall -ffree-line-length-none -ffree-line-length-0 -Wno-lto-type-mismatch -Wno-unused-dummy-argument -Wall -ffree-line-length-none -ffree-line-length-0 -Wno-lto-type-mismatch -Wno-unused-dummy-argument -O3 -march=native -mtune=native -I/home/bchandra/Program/petsc/include -I/home/bchandra/Program/petsc/arch-linux-c-opt/include -I/usr/local/openmpi-4.1.5/include ex5f.F90 -Wl,-rpath,/home/bchandra/Program/petsc/arch-linux-c-opt/lib -L/home/bchandra/Program/petsc/arch-linux-c-opt/lib -Wl,-rpath,/usr/local/openmpi-4.1.5/lib -L/usr/local/openmpi-4.1.5/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/13 -L/usr/lib/gcc/x86_64-linux-gnu/13 -lpetsc -lHYPRE -lflapack -lfblas -lm -lX11 -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lstdc++ -lquadmath -o ex5f ex5f.F90:439:52: 439 | PetscInt row,col(5),i,j,i1,i5 | ^ Warning: ?col[1]? may be used uninitialized [-Wmaybe-uninitialized] ex5f.F90:439:52: Warning: ?col[2]? may be used uninitialized [-Wmaybe-uninitialized] ex5f.F90:439:52: Warning: ?col[3]? may be used uninitialized [-Wmaybe-uninitialized] ex5f.F90:476:83: 476 | call MatSetValuesLocal(jac,i1,[row],i1,[col],[v],INSERT_VALUES,ierr) | ^ Warning: ?col[4]? may be used uninitialized [-Wmaybe-uninitialized] ex5f.F90:439:52: 439 | PetscInt row,col(5),i,j,i1,i5 | ^ note: ?col[4]? was declared here ex5f.F90:476:83: 476 | call MatSetValuesLocal(jac,i1,[row],i1,[col],[v],INSERT_VALUES,ierr) | ^ Warning: ?v[1]? may be used uninitialized [-Wmaybe-uninitialized] ex5f.F90:440:58: 440 | PetscScalar two,one,hx,hy,v(5) | ^ note: ?v[1]? was declared here ex5f.F90:476:83: 476 | call MatSetValuesLocal(jac,i1,[row],i1,[col],[v],INSERT_VALUES,ierr) | ^ Warning: ?v[2]? may be used uninitialized [-Wmaybe-uninitialized] ex5f.F90:440:58: 440 | PetscScalar two,one,hx,hy,v(5) | ^ note: ?v[2]? was declared here ex5f.F90:476:83: 476 | call MatSetValuesLocal(jac,i1,[row],i1,[col],[v],INSERT_VALUES,ierr) | ^ Warning: ?v[3]? may be used uninitialized [-Wmaybe-uninitialized] ex5f.F90:440:58: 440 | PetscScalar two,one,hx,hy,v(5) | ^ note: ?v[3]? was declared here ex5f.F90:476:83: 476 | call MatSetValuesLocal(jac,i1,[row],i1,[col],[v],INSERT_VALUES,ierr) | ^ Warning: ?v[4]? may be used uninitialized [-Wmaybe-uninitialized] ex5f.F90:440:58: 440 | PetscScalar two,one,hx,hy,v(5) | ^ note: ?v[4]? was declared here Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process Completed PETSc check examples ---------------------------------------------------------------------------------------------------- Any ideas what happened and how to fix it? Many thanks in advance. Kind regards, Nanda -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Tue Sep 3 21:25:23 2024 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 3 Sep 2024 22:25:23 -0400 Subject: [petsc-users] Installing PETSc in Ubuntu 24.04 - check test error In-Reply-To: References: Message-ID: <8D7FB301-29C2-4BEB-A59A-0C367C72EC68@petsc.dev> This is fine. Just an over-enthusiastic compiler Barry > On Sep 3, 2024, at 9:43?PM, Bodhinanda Chandra wrote: > > Dear PETSc community, > > I recently tried to install PETSc in Ubuntu 24.04. All seems to work fine, but one of the check tests ex5f thow the following errors: > > ---------------------------------------------------------------------------------------------------- > > Running PETSc check examples to verify correct installation > Using PETSC_DIR=/home/bchandra/Program/petsc and PETSC_ARCH=arch-linux-c-opt > C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process > C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes > C/C++ example src/snes/tutorials/ex19 run successfully with HYPRE > *******************Error detected during compile or link!******************* > See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!aWfx77pFkinEWnfZuOJV9iSUGz1PtCHOLEjTdL_APZHj6LB9f3z5sn2XDepCVrIL3IqhVVJH51ih6RUxgdeFsJ0$ > /home/bchandra/Program/petsc/src/snes/tutorials ex5f > ********************************************************* > /usr/local/openmpi-4.1.5/bin/mpif90 -fPIC -Wall -ffree-line-length-none -ffree-line-length-0 -Wno-lto-type-mismatch -Wno-unused-dummy-argument -Wall -ffree-line-length-none -ffree-line-length-0 -Wno-lto-type-mismatch -Wno-unused-dummy-argument -O3 -march=native -mtune=native -I/home/bchandra/Program/petsc/include -I/home/bchandra/Program/petsc/arch-linux-c-opt/include -I/usr/local/openmpi-4.1.5/include ex5f.F90 -Wl,-rpath,/home/bchandra/Program/petsc/arch-linux-c-opt/lib -L/home/bchandra/Program/petsc/arch-linux-c-opt/lib -Wl,-rpath,/usr/local/openmpi-4.1.5/lib -L/usr/local/openmpi-4.1.5/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/13 -L/usr/lib/gcc/x86_64-linux-gnu/13 -lpetsc -lHYPRE -lflapack -lfblas -lm -lX11 -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lstdc++ -lquadmath -o ex5f > ex5f.F90:439:52: > > 439 | PetscInt row,col(5),i,j,i1,i5 > | ^ > Warning: ?col[1]? may be used uninitialized [-Wmaybe-uninitialized] > ex5f.F90:439:52: Warning: ?col[2]? may be used uninitialized [-Wmaybe-uninitialized] > ex5f.F90:439:52: Warning: ?col[3]? may be used uninitialized [-Wmaybe-uninitialized] > ex5f.F90:476:83: > > 476 | call MatSetValuesLocal(jac,i1,[row],i1,[col],[v],INSERT_VALUES,ierr) > | ^ > Warning: ?col[4]? may be used uninitialized [-Wmaybe-uninitialized] > ex5f.F90:439:52: > > 439 | PetscInt row,col(5),i,j,i1,i5 > | ^ > note: ?col[4]? was declared here > ex5f.F90:476:83: > > 476 | call MatSetValuesLocal(jac,i1,[row],i1,[col],[v],INSERT_VALUES,ierr) > | ^ > Warning: ?v[1]? may be used uninitialized [-Wmaybe-uninitialized] > ex5f.F90:440:58: > > 440 | PetscScalar two,one,hx,hy,v(5) > | ^ > note: ?v[1]? was declared here > ex5f.F90:476:83: > > 476 | call MatSetValuesLocal(jac,i1,[row],i1,[col],[v],INSERT_VALUES,ierr) > | ^ > Warning: ?v[2]? may be used uninitialized [-Wmaybe-uninitialized] > ex5f.F90:440:58: > > 440 | PetscScalar two,one,hx,hy,v(5) > | ^ > note: ?v[2]? was declared here > ex5f.F90:476:83: > > 476 | call MatSetValuesLocal(jac,i1,[row],i1,[col],[v],INSERT_VALUES,ierr) > | ^ > Warning: ?v[3]? may be used uninitialized [-Wmaybe-uninitialized] > ex5f.F90:440:58: > > 440 | PetscScalar two,one,hx,hy,v(5) > | ^ > note: ?v[3]? was declared here > ex5f.F90:476:83: > > 476 | call MatSetValuesLocal(jac,i1,[row],i1,[col],[v],INSERT_VALUES,ierr) > | ^ > Warning: ?v[4]? may be used uninitialized [-Wmaybe-uninitialized] > ex5f.F90:440:58: > > 440 | PetscScalar two,one,hx,hy,v(5) > | ^ > note: ?v[4]? was declared here > Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process > Completed PETSc check examples > > ---------------------------------------------------------------------------------------------------- > > Any ideas what happened and how to fix it? Many thanks in advance. > > Kind regards, > Nanda -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay.anl at fastmail.org Tue Sep 3 21:57:33 2024 From: balay.anl at fastmail.org (Satish Balay) Date: Tue, 3 Sep 2024 21:57:33 -0500 (CDT) Subject: [petsc-users] Installing PETSc in Ubuntu 24.04 - check test error In-Reply-To: <8D7FB301-29C2-4BEB-A59A-0C367C72EC68@petsc.dev> References: <8D7FB301-29C2-4BEB-A59A-0C367C72EC68@petsc.dev> Message-ID: <6288277f-dfbb-e86b-2eed-5321e760ef2e@fastmail.org> what do you have for: /usr/local/openmpi-4.1.5/bin/mpif90 -show Also, can you try: make check FFLAGS=-Wno-maybe-uninitialized Satish On Tue, 3 Sep 2024, Barry Smith wrote: > > This is fine. Just an over-enthusiastic compiler > > Barry > > > > On Sep 3, 2024, at 9:43?PM, Bodhinanda Chandra wrote: > > > > Dear PETSc community, > > > > I recently tried to install PETSc in Ubuntu 24.04. All seems to work fine, but one of the check tests ex5f thow the following errors: > > > > ---------------------------------------------------------------------------------------------------- > > > > Running PETSc check examples to verify correct installation > > Using PETSC_DIR=/home/bchandra/Program/petsc and PETSC_ARCH=arch-linux-c-opt > > C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process > > C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes > > C/C++ example src/snes/tutorials/ex19 run successfully with HYPRE > > *******************Error detected during compile or link!******************* > > See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!aWfx77pFkinEWnfZuOJV9iSUGz1PtCHOLEjTdL_APZHj6LB9f3z5sn2XDepCVrIL3IqhVVJH51ih6RUxgdeFsJ0$ > > /home/bchandra/Program/petsc/src/snes/tutorials ex5f > > ********************************************************* > > /usr/local/openmpi-4.1.5/bin/mpif90 -fPIC -Wall -ffree-line-length-none -ffree-line-length-0 -Wno-lto-type-mismatch -Wno-unused-dummy-argument -Wall -ffree-line-length-none -ffree-line-length-0 -Wno-lto-type-mismatch -Wno-unused-dummy-argument -O3 -march=native -mtune=native -I/home/bchandra/Program/petsc/include -I/home/bchandra/Program/petsc/arch-linux-c-opt/include -I/usr/local/openmpi-4.1.5/include ex5f.F90 -Wl,-rpath,/home/bchandra/Program/petsc/arch-linux-c-opt/lib -L/home/bchandra/Program/petsc/arch-linux-c-opt/lib -Wl,-rpath,/usr/local/openmpi-4.1.5/lib -L/usr/local/openmpi-4.1.5/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/13 -L/usr/lib/gcc/x86_64-linux-gnu/13 -lpetsc -lHYPRE -lflapack -lfblas -lm -lX11 -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lstdc++ -lquadmath -o ex5f > > ex5f.F90:439:52: > > > > 439 | PetscInt row,col(5),i,j,i1,i5 > > | ^ > > Warning: ?col[1]? may be used uninitialized [-Wmaybe-uninitialized] > > ex5f.F90:439:52: Warning: ?col[2]? may be used uninitialized [-Wmaybe-uninitialized] > > ex5f.F90:439:52: Warning: ?col[3]? may be used uninitialized [-Wmaybe-uninitialized] > > ex5f.F90:476:83: > > > > 476 | call MatSetValuesLocal(jac,i1,[row],i1,[col],[v],INSERT_VALUES,ierr) > > | ^ > > Warning: ?col[4]? may be used uninitialized [-Wmaybe-uninitialized] > > ex5f.F90:439:52: > > > > 439 | PetscInt row,col(5),i,j,i1,i5 > > | ^ > > note: ?col[4]? was declared here > > ex5f.F90:476:83: > > > > 476 | call MatSetValuesLocal(jac,i1,[row],i1,[col],[v],INSERT_VALUES,ierr) > > | ^ > > Warning: ?v[1]? may be used uninitialized [-Wmaybe-uninitialized] > > ex5f.F90:440:58: > > > > 440 | PetscScalar two,one,hx,hy,v(5) > > | ^ > > note: ?v[1]? was declared here > > ex5f.F90:476:83: > > > > 476 | call MatSetValuesLocal(jac,i1,[row],i1,[col],[v],INSERT_VALUES,ierr) > > | ^ > > Warning: ?v[2]? may be used uninitialized [-Wmaybe-uninitialized] > > ex5f.F90:440:58: > > > > 440 | PetscScalar two,one,hx,hy,v(5) > > | ^ > > note: ?v[2]? was declared here > > ex5f.F90:476:83: > > > > 476 | call MatSetValuesLocal(jac,i1,[row],i1,[col],[v],INSERT_VALUES,ierr) > > | ^ > > Warning: ?v[3]? may be used uninitialized [-Wmaybe-uninitialized] > > ex5f.F90:440:58: > > > > 440 | PetscScalar two,one,hx,hy,v(5) > > | ^ > > note: ?v[3]? was declared here > > ex5f.F90:476:83: > > > > 476 | call MatSetValuesLocal(jac,i1,[row],i1,[col],[v],INSERT_VALUES,ierr) > > | ^ > > Warning: ?v[4]? may be used uninitialized [-Wmaybe-uninitialized] > > ex5f.F90:440:58: > > > > 440 | PetscScalar two,one,hx,hy,v(5) > > | ^ > > note: ?v[4]? was declared here > > Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process > > Completed PETSc check examples > > > > ---------------------------------------------------------------------------------------------------- > > > > Any ideas what happened and how to fix it? Many thanks in advance. > > > > Kind regards, > > Nanda > > From bodhinandach at gmail.com Wed Sep 4 11:49:02 2024 From: bodhinandach at gmail.com (Bodhinanda Chandra) Date: Wed, 4 Sep 2024 09:49:02 -0700 Subject: [petsc-users] Installing PETSc in Ubuntu 24.04 - check test error In-Reply-To: <6288277f-dfbb-e86b-2eed-5321e760ef2e@fastmail.org> References: <8D7FB301-29C2-4BEB-A59A-0C367C72EC68@petsc.dev> <6288277f-dfbb-e86b-2eed-5321e760ef2e@fastmail.org> Message-ID: Dear Satish and Barry, thanks for answering. This command: /usr/local/openmpi-4.1.5/bin/mpif90 -show outputs: gfortran -I/usr/local/openmpi-4.1.5/include -I/usr/local/openmpi-4.1.5/lib -L/usr/local/openmpi-4.1.5/lib -Wl,-rpath -Wl,/usr/local/openmpi-4.1.5/lib -Wl,--enable-new-dtags -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi While this command: make PETSC_DIR=/home/bchandra/Program/petsc PETSC_ARCH=arch-linux-c-opt check FFLAGS=-Wno-maybe-uninitialized seems to hide the errors: Running PETSc check examples to verify correct installation Using PETSC_DIR=/home/bchandra/Program/petsc and PETSC_ARCH=arch-linux-c-opt C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes C/C++ example src/snes/tutorials/ex19 run successfully with HYPRE Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process Completed PETSc check examples Kind regards, Nanda On Tue, Sep 3, 2024 at 7:57?PM Satish Balay wrote: > what do you have for: > > /usr/local/openmpi-4.1.5/bin/mpif90 -show > > Also, can you try: > > make check FFLAGS=-Wno-maybe-uninitialized > > Satish > > On Tue, 3 Sep 2024, Barry Smith wrote: > > > > > This is fine. Just an over-enthusiastic compiler > > > > Barry > > > > > > > On Sep 3, 2024, at 9:43?PM, Bodhinanda Chandra > wrote: > > > > > > Dear PETSc community, > > > > > > I recently tried to install PETSc in Ubuntu 24.04. All seems to work > fine, but one of the check tests ex5f thow the following errors: > > > > > > > ---------------------------------------------------------------------------------------------------- > > > > > > Running PETSc check examples to verify correct installation > > > Using PETSC_DIR=/home/bchandra/Program/petsc and > PETSC_ARCH=arch-linux-c-opt > > > C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI > process > > > C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI > processes > > > C/C++ example src/snes/tutorials/ex19 run successfully with HYPRE > > > *******************Error detected during compile or > link!******************* > > > See > https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!aWfx77pFkinEWnfZuOJV9iSUGz1PtCHOLEjTdL_APZHj6LB9f3z5sn2XDepCVrIL3IqhVVJH51ih6RUxgdeFsJ0$ > < > https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!bNBtvML4qAKu9WGzBH2kRjtiP8AwN3ijY4qZngu7nvl_TUwz8pQ6WGyCJh0AtIEOqLHU5NHUJRL1oWg2FeTR0Oz6868$ > > > > > /home/bchandra/Program/petsc/src/snes/tutorials ex5f > > > ********************************************************* > > > /usr/local/openmpi-4.1.5/bin/mpif90 -fPIC -Wall > -ffree-line-length-none -ffree-line-length-0 -Wno-lto-type-mismatch > -Wno-unused-dummy-argument -Wall -ffree-line-length-none > -ffree-line-length-0 -Wno-lto-type-mismatch -Wno-unused-dummy-argument -O3 > -march=native -mtune=native -I/home/bchandra/Program/petsc/include > -I/home/bchandra/Program/petsc/arch-linux-c-opt/include > -I/usr/local/openmpi-4.1.5/include ex5f.F90 > -Wl,-rpath,/home/bchandra/Program/petsc/arch-linux-c-opt/lib > -L/home/bchandra/Program/petsc/arch-linux-c-opt/lib > -Wl,-rpath,/usr/local/openmpi-4.1.5/lib -L/usr/local/openmpi-4.1.5/lib > -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/13 > -L/usr/lib/gcc/x86_64-linux-gnu/13 -lpetsc -lHYPRE -lflapack -lfblas -lm > -lX11 -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran > -lm -lgfortran -lm -lgcc_s -lquadmath -lstdc++ -lquadmath -o ex5f > > > ex5f.F90:439:52: > > > > > > 439 | PetscInt row,col(5),i,j,i1,i5 > > > | ^ > > > Warning: ?col[1]? may be used uninitialized [-Wmaybe-uninitialized] > > > ex5f.F90:439:52: Warning: ?col[2]? may be used uninitialized > [-Wmaybe-uninitialized] > > > ex5f.F90:439:52: Warning: ?col[3]? may be used uninitialized > [-Wmaybe-uninitialized] > > > ex5f.F90:476:83: > > > > > > 476 | call > MatSetValuesLocal(jac,i1,[row],i1,[col],[v],INSERT_VALUES,ierr) > > > | > ^ > > > Warning: ?col[4]? may be used uninitialized [-Wmaybe-uninitialized] > > > ex5f.F90:439:52: > > > > > > 439 | PetscInt row,col(5),i,j,i1,i5 > > > | ^ > > > note: ?col[4]? was declared here > > > ex5f.F90:476:83: > > > > > > 476 | call > MatSetValuesLocal(jac,i1,[row],i1,[col],[v],INSERT_VALUES,ierr) > > > | > ^ > > > Warning: ?v[1]? may be used uninitialized [-Wmaybe-uninitialized] > > > ex5f.F90:440:58: > > > > > > 440 | PetscScalar two,one,hx,hy,v(5) > > > | ^ > > > note: ?v[1]? was declared here > > > ex5f.F90:476:83: > > > > > > 476 | call > MatSetValuesLocal(jac,i1,[row],i1,[col],[v],INSERT_VALUES,ierr) > > > | > ^ > > > Warning: ?v[2]? may be used uninitialized [-Wmaybe-uninitialized] > > > ex5f.F90:440:58: > > > > > > 440 | PetscScalar two,one,hx,hy,v(5) > > > | ^ > > > note: ?v[2]? was declared here > > > ex5f.F90:476:83: > > > > > > 476 | call > MatSetValuesLocal(jac,i1,[row],i1,[col],[v],INSERT_VALUES,ierr) > > > | > ^ > > > Warning: ?v[3]? may be used uninitialized [-Wmaybe-uninitialized] > > > ex5f.F90:440:58: > > > > > > 440 | PetscScalar two,one,hx,hy,v(5) > > > | ^ > > > note: ?v[3]? was declared here > > > ex5f.F90:476:83: > > > > > > 476 | call > MatSetValuesLocal(jac,i1,[row],i1,[col],[v],INSERT_VALUES,ierr) > > > | > ^ > > > Warning: ?v[4]? may be used uninitialized [-Wmaybe-uninitialized] > > > ex5f.F90:440:58: > > > > > > 440 | PetscScalar two,one,hx,hy,v(5) > > > | ^ > > > note: ?v[4]? was declared here > > > Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI > process > > > Completed PETSc check examples > > > > > > > ---------------------------------------------------------------------------------------------------- > > > > > > Any ideas what happened and how to fix it? Many thanks in advance. > > > > > > Kind regards, > > > Nanda > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From linyx199071 at gmail.com Wed Sep 4 14:52:59 2024 From: linyx199071 at gmail.com (Lin_Yuxiang) Date: Wed, 4 Sep 2024 14:52:59 -0500 Subject: [petsc-users] Bug or mis-use for 64 indices PETSc mpi linear solver server with more than 8 cores Message-ID: To whom it may concern: I recently tried to use the 64 indices PETSc to replace the legacy code's solver using MPI linear solver server. However, it gives me error when I use more than 8 cores, saying Get NNZ MatsetPreallocation MatsetValue MatSetValue Time per kernel: 43.1147 s Matassembly VecsetValue pestc_solve Read -1, expected 1951397280, errno = 14 When I tried the -start_in_debugger, the error seems from MPI_Scatter: Rank0: #3 0x00001555512e4de5 in mca_pml_ob1_recv () from /usr/lib/x86_64-linux-gnu/openmpi/lib/openmpi3/mca_pml_ob1.so #4 0x0000155553e01e60 in PMPI_Scatterv () from /lib/x86_64-linux-gnu/libmpi.so.40 #5 0x0000155554b13eab in PCMPISetMat (pc=pc at entry=0x0) at /auto/research/rdfs/home/lyuxiang/petsc-3.20.4/src/ksp/pc/impls/mpi/pcmpi.c:230 #6 0x0000155554b17403 in PCMPIServerBegin () at /auto/research/rdfs/home/lyuxiang/petsc-3.20.4/src/ksp/pc/impls/mpi/pcmpi.c:464 #7 0x00001555540b9aa4 in PetscInitialize_Common (prog=0x7fffffffe27b "geosimtrs_mpiserver", file=file at entry=0x0, help=help at entry=0x55555555a1e0 "Solves a linear system in parallel with KSP.\nInput parameters include:\n -view_exact_sol : write exact solution vector to stdout\n -m : number of mesh points in x-direction\n -n "Solves a linear system in parallel with KSP.\nInput parameters include:\n -view_exact_sol : write exact solution vector to stdout\n -m : number of mesh points in x-direction\n -n , args=) at geosimtrs_mpiserver.c:29 Rank1-10 0x0000155553e1f030 in ompi_coll_base_allgather_intra_bruck () from /lib/x86_64-linux-gnu/libmpi.so.40 #4 0x0000155550f62aaa in ompi_coll_tuned_allgather_intra_dec_fixed () from /usr/lib/x86_64-linux-gnu/openmpi/lib/openmpi3/mca_coll_tuned.so #5 0x0000155553ddb431 in PMPI_Allgather () from /lib/x86_64-linux-gnu/libmpi.so.40 #6 0x00001555541a2289 in PetscLayoutSetUp (map=0x555555721ed0) at /auto/research/rdfs/home/lyuxiang/petsc-3.20.4/src/vec/is/utils/pmap.c:248 #7 0x000015555442e06a in MatMPIAIJSetPreallocationCSR_MPIAIJ (B=0x55555572d850, Ii=0x15545a778010, J=0x15545beacb60, v=0x1554cff55e60) at /auto/research/rdfs/home/lyuxiang/petsc-3.20.4/src/mat/impls/aij/mpi/mpiaij.c:3885 #8 0x00001555544284e3 in MatMPIAIJSetPreallocationCSR (B=0x55555572d850, i=0x15545a778010, j=0x15545beacb60, v=0x1554cff55e60) at /auto/research/rdfs/home/lyuxiang/petsc-3.20.4/src/mat/impls/aij/mpi/mpiaij.c:3998 #9 0x0000155554b1412f in PCMPISetMat (pc=pc at entry=0x0) at /auto/research/rdfs/home/lyuxiang/petsc-3.20.4/src/ksp/pc/impls/mpi/pcmpi.c:250 #10 0x0000155554b17403 in PCMPIServerBegin () at /auto/research/rdfs/home/lyuxiang/petsc-3.20.4/src/ksp/pc/impls/mpi/pcmpi.c:464 #11 0x00001555540b9aa4 in PetscInitialize_Common (prog=0x7fffffffe27b "geosimtrs_mpiserver", file=file at entry=0x0, help=help at entry=0x55555555a1e0 "Solves a linear system in parallel with KSP.\nInput parameters include:\n -view_exact_sol : write exact solution vector to stdout\n -m : number of mesh points in x-direction\n -n "Solves a linear system in parallel with KSP.\nInput parameters include:\n -view_exact_sol : write exact solution vector to stdout\n -m : number of mesh points in x-direction\n -n , args=) at geosimtrs_mpiserver.c:29 This did not happen in 32bit indiced PETSc, running with more than 8 cores runs smoothly using MPI linear solver server, nor did it happen on 64 bit indiced MPI version (not with mpi_linear_solver_server), only happens on 64 bit PETSc mpi linear solver server, I think it maybe a potential bug? Any advice would be greatly appreciated, the matrix and ia, ja is too big to upload, so if anything you need to debug pls let me know - Machine type: HPC - OS version and type: Linux houamd009 6.1.55-cggdb11-1 #1 SMP Fri Sep 29 10:09:13 UTC 2023 x86_64 GNU/Linux - PETSc version: #define PETSC_VERSION_RELEASE 1 #define PETSC_VERSION_MAJOR 3 #define PETSC_VERSION_MINOR 20 #define PETSC_VERSION_SUBMINOR 4 #define PETSC_RELEASE_DATE "Sep 28, 2023" #define PETSC_VERSION_DATE "Jan 29, 2024" - MPI implementation: OpenMPI - Compiler and version: GNU Yuxiang Lin -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Thu Sep 5 06:56:27 2024 From: mfadams at lbl.gov (Mark Adams) Date: Thu, 5 Sep 2024 07:56:27 -0400 Subject: [petsc-users] Bug or mis-use for 64 indices PETSc mpi linear solver server with more than 8 cores In-Reply-To: References: Message-ID: Barry's suggestion for testing got garbled in the gitlab issue posting. Here it is, I think: 07:53 main *= ~/Codes/petsc$ make test s=ksp_ksp_tutorials-ex1_mpi_linear_solver_server_1 /usr/local/bin/gmake --no-print-directory -f /Users/markadams/Codes/petsc/gmakefile.test PETSC_ARCH=arch-macosx-gnu-O PETSC_DIR=/Users/markadams/Codes/petsc test Using MAKEFLAGS: --no-print-directory -- PETSC_DIR=/Users/markadams/Codes/petsc PETSC_ARCH=arch-macosx-gnu-O s=ksp_ksp_tutorials-ex1_mpi_linear_solver_server_1 Application at path ( /Users/markadams/Codes/petsc/arch-macosx-gnu-O/bin/mpiexec.hydra ) removed from firewall Application at path ( /Users/markadams/Codes/petsc/arch-macosx-gnu-O/bin/mpiexec.hydra ) added to firewall Incoming connection to the application is blocked TEST arch-macosx-gnu-O/tests/counts/ksp_ksp_tutorials-ex1_mpi_linear_solver_server_1.counts ok ksp_ksp_tutorials-ex1_mpi_linear_solver_server_1 ok diff-ksp_ksp_tutorials-ex1_mpi_linear_solver_server_1 On Wed, Sep 4, 2024 at 3:59?PM Lin_Yuxiang wrote: > To whom it may concern: > > > > I recently tried to use the 64 indices PETSc to replace the legacy code's > solver using MPI linear solver server. However, it gives me error when I > use more than 8 cores, saying > > > > Get NNZ > > MatsetPreallocation > > MatsetValue > > MatSetValue Time per kernel: 43.1147 s > > Matassembly > > VecsetValue > > pestc_solve > > Read -1, expected 1951397280, errno = 14 > > > > When I tried the -start_in_debugger, the error seems from MPI_Scatter: > > > > Rank0: > > #3 0x00001555512e4de5 in mca_pml_ob1_recv () from > /usr/lib/x86_64-linux-gnu/openmpi/lib/openmpi3/mca_pml_ob1.so > > #4 0x0000155553e01e60 in PMPI_Scatterv () from > /lib/x86_64-linux-gnu/libmpi.so.40 > > #5 0x0000155554b13eab in PCMPISetMat (pc=pc at entry=0x0) at > /auto/research/rdfs/home/lyuxiang/petsc-3.20.4/src/ksp/pc/impls/mpi/pcmpi.c:230 > > #6 0x0000155554b17403 in PCMPIServerBegin () at > /auto/research/rdfs/home/lyuxiang/petsc-3.20.4/src/ksp/pc/impls/mpi/pcmpi.c:464 > > #7 0x00001555540b9aa4 in PetscInitialize_Common (prog=0x7fffffffe27b > "geosimtrs_mpiserver", file=file at entry=0x0, > > help=help at entry=0x55555555a1e0 "Solves a linear system in > parallel with KSP.\nInput parameters include:\n -view_exact_sol : write > exact solution vector to stdout\n -m : number of mesh > points in x-direction\n -n readarguments=readarguments at entry=PETSC_FALSE, len=len at entry=0) > > at > /auto/research/rdfs/home/lyuxiang/petsc-3.20.4/src/sys/objects/pinit.c:1109 > > #8 0x00001555540bba82 in PetscInitialize (argc=argc at entry=0x7fffffffda8c, > args=args at entry=0x7fffffffda80, file=file at entry=0x0, > > help=help at entry=0x55555555a1e0 "Solves a linear system in > parallel with KSP.\nInput parameters include:\n -view_exact_sol : write > exact solution vector to stdout\n -m : number of mesh > points in x-direction\n -n /auto/research/rdfs/home/lyuxiang/petsc-3.20.4/src/sys/objects/pinit.c:1274 > > #9 0x0000555555557673 in main (argc=, args= out>) at geosimtrs_mpiserver.c:29 > > > > Rank1-10 > > 0x0000155553e1f030 in ompi_coll_base_allgather_intra_bruck () from > /lib/x86_64-linux-gnu/libmpi.so.40 > > #4 0x0000155550f62aaa in ompi_coll_tuned_allgather_intra_dec_fixed () > from /usr/lib/x86_64-linux-gnu/openmpi/lib/openmpi3/mca_coll_tuned.so > > #5 0x0000155553ddb431 in PMPI_Allgather () from > /lib/x86_64-linux-gnu/libmpi.so.40 > > #6 0x00001555541a2289 in PetscLayoutSetUp (map=0x555555721ed0) at > /auto/research/rdfs/home/lyuxiang/petsc-3.20.4/src/vec/is/utils/pmap.c:248 > > #7 0x000015555442e06a in MatMPIAIJSetPreallocationCSR_MPIAIJ > (B=0x55555572d850, Ii=0x15545a778010, J=0x15545beacb60, v=0x1554cff55e60) > > at > /auto/research/rdfs/home/lyuxiang/petsc-3.20.4/src/mat/impls/aij/mpi/mpiaij.c:3885 > > #8 0x00001555544284e3 in MatMPIAIJSetPreallocationCSR (B=0x55555572d850, > i=0x15545a778010, j=0x15545beacb60, v=0x1554cff55e60) at > /auto/research/rdfs/home/lyuxiang/petsc-3.20.4/src/mat/impls/aij/mpi/mpiaij.c:3998 > > #9 0x0000155554b1412f in PCMPISetMat (pc=pc at entry=0x0) at > /auto/research/rdfs/home/lyuxiang/petsc-3.20.4/src/ksp/pc/impls/mpi/pcmpi.c:250 > > #10 0x0000155554b17403 in PCMPIServerBegin () at > /auto/research/rdfs/home/lyuxiang/petsc-3.20.4/src/ksp/pc/impls/mpi/pcmpi.c:464 > > #11 0x00001555540b9aa4 in PetscInitialize_Common (prog=0x7fffffffe27b > "geosimtrs_mpiserver", file=file at entry=0x0, > > help=help at entry=0x55555555a1e0 "Solves a linear system in > parallel with KSP.\nInput parameters include:\n -view_exact_sol : write > exact solution vector to stdout\n -m : number of mesh > points in x-direction\n -n readarguments=readarguments at entry=PETSC_FALSE, len=len at entry=0) at > /auto/research/rdfs/home/lyuxiang/petsc-3.20.4/src/sys/objects/pinit.c:1109 > > #12 0x00001555540bba82 in PetscInitialize (argc=argc at entry=0x7fffffffda8c, > args=args at entry=0x7fffffffda80, file=file at entry=0x0, > > help=help at entry=0x55555555a1e0 "Solves a linear system in > parallel with KSP.\nInput parameters include:\n -view_exact_sol : write > exact solution vector to stdout\n -m : number of mesh > points in x-direction\n -n /auto/research/rdfs/home/lyuxiang/petsc-3.20.4/src/sys/objects/pinit.c:1274 > > #13 0x0000555555557673 in main (argc=, args= out>) at geosimtrs_mpiserver.c:29 > > > > This did not happen in 32bit indiced PETSc, running with more than 8 cores > runs smoothly using MPI linear solver server, nor did it happen on 64 bit > indiced MPI version (not with mpi_linear_solver_server), only happens on 64 > bit PETSc mpi linear solver server, I think it maybe a potential bug? > > > > Any advice would be greatly appreciated, the matrix and ia, ja is too big > to upload, so if anything you need to debug pls let me know > > > > - > > Machine type: HPC > - > > OS version and type: Linux houamd009 6.1.55-cggdb11-1 #1 SMP Fri Sep > 29 10:09:13 UTC 2023 x86_64 GNU/Linux > - > > PETSc version: #define PETSC_VERSION_RELEASE 1 > #define PETSC_VERSION_MAJOR 3 > #define PETSC_VERSION_MINOR 20 > #define PETSC_VERSION_SUBMINOR 4 > #define PETSC_RELEASE_DATE "Sep 28, 2023" > #define PETSC_VERSION_DATE "Jan 29, 2024" > - > > MPI implementation: OpenMPI > - > > Compiler and version: GNU > > > > Yuxiang Lin > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay.anl at fastmail.org Thu Sep 5 08:48:08 2024 From: balay.anl at fastmail.org (Satish Balay) Date: Thu, 5 Sep 2024 08:48:08 -0500 (CDT) Subject: [petsc-users] Bug or mis-use for 64 indices PETSc mpi linear solver server with more than 8 cores In-Reply-To: References: Message-ID: I fixed the "garbled" text at https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/issues/1643__;!!G_uCfscf7eWS!YEs2AyYzwZQphspkgEhgojcPm1ypNLOD9_BZwl9NzvkOhN7BdIXzBHuN2n_6s2yV2e8VMCn58IgSGrN6NakWS5sRZnw$ - and best if additional followup is on the issue tracker [so that the replies to this issue don't get scattered] Satish On Thu, 5 Sep 2024, Mark Adams wrote: > Barry's suggestion for testing got garbled in the gitlab issue posting. > Here it is, I think: > > 07:53 main *= ~/Codes/petsc$ make test > s=ksp_ksp_tutorials-ex1_mpi_linear_solver_server_1 > /usr/local/bin/gmake --no-print-directory -f > /Users/markadams/Codes/petsc/gmakefile.test PETSC_ARCH=arch-macosx-gnu-O > PETSC_DIR=/Users/markadams/Codes/petsc test > Using MAKEFLAGS: --no-print-directory -- > PETSC_DIR=/Users/markadams/Codes/petsc PETSC_ARCH=arch-macosx-gnu-O > s=ksp_ksp_tutorials-ex1_mpi_linear_solver_server_1 > Application at path ( > /Users/markadams/Codes/petsc/arch-macosx-gnu-O/bin/mpiexec.hydra ) removed > from firewall > Application at path ( > /Users/markadams/Codes/petsc/arch-macosx-gnu-O/bin/mpiexec.hydra ) added to > firewall > Incoming connection to the application is blocked > TEST > arch-macosx-gnu-O/tests/counts/ksp_ksp_tutorials-ex1_mpi_linear_solver_server_1.counts > ok ksp_ksp_tutorials-ex1_mpi_linear_solver_server_1 > ok diff-ksp_ksp_tutorials-ex1_mpi_linear_solver_server_1 > > On Wed, Sep 4, 2024 at 3:59?PM Lin_Yuxiang wrote: > > > To whom it may concern: > > > > > > > > I recently tried to use the 64 indices PETSc to replace the legacy code's > > solver using MPI linear solver server. However, it gives me error when I > > use more than 8 cores, saying > > > > > > > > Get NNZ > > > > MatsetPreallocation > > > > MatsetValue > > > > MatSetValue Time per kernel: 43.1147 s > > > > Matassembly > > > > VecsetValue > > > > pestc_solve > > > > Read -1, expected 1951397280, errno = 14 > > > > > > > > When I tried the -start_in_debugger, the error seems from MPI_Scatter: > > > > > > > > Rank0: > > > > #3 0x00001555512e4de5 in mca_pml_ob1_recv () from > > /usr/lib/x86_64-linux-gnu/openmpi/lib/openmpi3/mca_pml_ob1.so > > > > #4 0x0000155553e01e60 in PMPI_Scatterv () from > > /lib/x86_64-linux-gnu/libmpi.so.40 > > > > #5 0x0000155554b13eab in PCMPISetMat (pc=pc at entry=0x0) at > > /auto/research/rdfs/home/lyuxiang/petsc-3.20.4/src/ksp/pc/impls/mpi/pcmpi.c:230 > > > > #6 0x0000155554b17403 in PCMPIServerBegin () at > > /auto/research/rdfs/home/lyuxiang/petsc-3.20.4/src/ksp/pc/impls/mpi/pcmpi.c:464 > > > > #7 0x00001555540b9aa4 in PetscInitialize_Common (prog=0x7fffffffe27b > > "geosimtrs_mpiserver", file=file at entry=0x0, > > > > help=help at entry=0x55555555a1e0 "Solves a linear system in > > parallel with KSP.\nInput parameters include:\n -view_exact_sol : write > > exact solution vector to stdout\n -m : number of mesh > > points in x-direction\n -n > readarguments=readarguments at entry=PETSC_FALSE, len=len at entry=0) > > > > at > > /auto/research/rdfs/home/lyuxiang/petsc-3.20.4/src/sys/objects/pinit.c:1109 > > > > #8 0x00001555540bba82 in PetscInitialize (argc=argc at entry=0x7fffffffda8c, > > args=args at entry=0x7fffffffda80, file=file at entry=0x0, > > > > help=help at entry=0x55555555a1e0 "Solves a linear system in > > parallel with KSP.\nInput parameters include:\n -view_exact_sol : write > > exact solution vector to stdout\n -m : number of mesh > > points in x-direction\n -n > /auto/research/rdfs/home/lyuxiang/petsc-3.20.4/src/sys/objects/pinit.c:1274 > > > > #9 0x0000555555557673 in main (argc=, args= > out>) at geosimtrs_mpiserver.c:29 > > > > > > > > Rank1-10 > > > > 0x0000155553e1f030 in ompi_coll_base_allgather_intra_bruck () from > > /lib/x86_64-linux-gnu/libmpi.so.40 > > > > #4 0x0000155550f62aaa in ompi_coll_tuned_allgather_intra_dec_fixed () > > from /usr/lib/x86_64-linux-gnu/openmpi/lib/openmpi3/mca_coll_tuned.so > > > > #5 0x0000155553ddb431 in PMPI_Allgather () from > > /lib/x86_64-linux-gnu/libmpi.so.40 > > > > #6 0x00001555541a2289 in PetscLayoutSetUp (map=0x555555721ed0) at > > /auto/research/rdfs/home/lyuxiang/petsc-3.20.4/src/vec/is/utils/pmap.c:248 > > > > #7 0x000015555442e06a in MatMPIAIJSetPreallocationCSR_MPIAIJ > > (B=0x55555572d850, Ii=0x15545a778010, J=0x15545beacb60, v=0x1554cff55e60) > > > > at > > /auto/research/rdfs/home/lyuxiang/petsc-3.20.4/src/mat/impls/aij/mpi/mpiaij.c:3885 > > > > #8 0x00001555544284e3 in MatMPIAIJSetPreallocationCSR (B=0x55555572d850, > > i=0x15545a778010, j=0x15545beacb60, v=0x1554cff55e60) at > > /auto/research/rdfs/home/lyuxiang/petsc-3.20.4/src/mat/impls/aij/mpi/mpiaij.c:3998 > > > > #9 0x0000155554b1412f in PCMPISetMat (pc=pc at entry=0x0) at > > /auto/research/rdfs/home/lyuxiang/petsc-3.20.4/src/ksp/pc/impls/mpi/pcmpi.c:250 > > > > #10 0x0000155554b17403 in PCMPIServerBegin () at > > /auto/research/rdfs/home/lyuxiang/petsc-3.20.4/src/ksp/pc/impls/mpi/pcmpi.c:464 > > > > #11 0x00001555540b9aa4 in PetscInitialize_Common (prog=0x7fffffffe27b > > "geosimtrs_mpiserver", file=file at entry=0x0, > > > > help=help at entry=0x55555555a1e0 "Solves a linear system in > > parallel with KSP.\nInput parameters include:\n -view_exact_sol : write > > exact solution vector to stdout\n -m : number of mesh > > points in x-direction\n -n > readarguments=readarguments at entry=PETSC_FALSE, len=len at entry=0) at > > /auto/research/rdfs/home/lyuxiang/petsc-3.20.4/src/sys/objects/pinit.c:1109 > > > > #12 0x00001555540bba82 in PetscInitialize (argc=argc at entry=0x7fffffffda8c, > > args=args at entry=0x7fffffffda80, file=file at entry=0x0, > > > > help=help at entry=0x55555555a1e0 "Solves a linear system in > > parallel with KSP.\nInput parameters include:\n -view_exact_sol : write > > exact solution vector to stdout\n -m : number of mesh > > points in x-direction\n -n > /auto/research/rdfs/home/lyuxiang/petsc-3.20.4/src/sys/objects/pinit.c:1274 > > > > #13 0x0000555555557673 in main (argc=, args= > out>) at geosimtrs_mpiserver.c:29 > > > > > > > > This did not happen in 32bit indiced PETSc, running with more than 8 cores > > runs smoothly using MPI linear solver server, nor did it happen on 64 bit > > indiced MPI version (not with mpi_linear_solver_server), only happens on 64 > > bit PETSc mpi linear solver server, I think it maybe a potential bug? > > > > > > > > Any advice would be greatly appreciated, the matrix and ia, ja is too big > > to upload, so if anything you need to debug pls let me know > > > > > > > > - > > > > Machine type: HPC > > - > > > > OS version and type: Linux houamd009 6.1.55-cggdb11-1 #1 SMP Fri Sep > > 29 10:09:13 UTC 2023 x86_64 GNU/Linux > > - > > > > PETSc version: #define PETSC_VERSION_RELEASE 1 > > #define PETSC_VERSION_MAJOR 3 > > #define PETSC_VERSION_MINOR 20 > > #define PETSC_VERSION_SUBMINOR 4 > > #define PETSC_RELEASE_DATE "Sep 28, 2023" > > #define PETSC_VERSION_DATE "Jan 29, 2024" > > - > > > > MPI implementation: OpenMPI > > - > > > > Compiler and version: GNU > > > > > > > > Yuxiang Lin > > > From l.j.corbijnvanwillenswaard at utwente.nl Thu Sep 5 12:33:07 2024 From: l.j.corbijnvanwillenswaard at utwente.nl (Corbijn van Willenswaard, Lars (UT)) Date: Thu, 5 Sep 2024 17:33:07 +0000 Subject: [petsc-users] KSPSolve + MUMPS memory growth issues Message-ID: <68AAED91-ED33-4E9F-BCE1-55C88020236C@utwente.nl> Dear PETSc, For the last months I?ve struggled with a solver that I wrote for a FEM eigenvalue problem running out of memory. I?ve traced it to KSPSolve + MUMPS being the issue, but I'm getting stuck on digging deeper. The reason I suspect the KSPSolve/MUMPS is that when commenting out the KSPSolve the memory stays constant while running the rest of the algorithm. Of course, the algorithm also converges to a different result in this setup. When changing the KSP statement to for(int i = 0; i < 100000000; i++) KSPSolve(A_, vec1_, vec2_); the memory grows faster than when running the algorithm. Logging shows that the program never the terminating i=100M. Measuring the memory growth using ps (started debugging before I knew of PETSc's features) I see a growth in the RSS on a single compute node of up to 300MB/min for this artificial case. Real cases grow more like 60MB/min/node, which causes a kill due to memory exhaustion after about 2-3 days. Locally (Mac) I've been able to reproduce this both with 6 MPI processes and with a single one. Instrumenting the code to show differences in PetscMemoryGetCurrentUsage (full code below), shows that the memory increases every step at the start, but also does at later iterations (small excerpt from the output): rank step memory (increase since prev step) 0 6544 current 39469056( 8192) 0 7086 current 39477248( 8192) 0 7735 current 39497728( 20480) 0 9029 current 39501824( 4096) A similar output is visible in a run with 6 ranks, where there does not seem to be a pattern as to which of the ranks increases at which step. (Note I've checked PetscMallocGetCurrentUsage, but that is constant) Switching the solver to petsc's own solver on a single rank does not show a memory increase after the first solve. Changing the solve to overwrite the vector will result in a few increases after the first solve, but these do not seem to repeat. So, changes like VecCopy(vec2, vec1_); KSPSolve(A_, vec1_, vec1_);. Does anyone have an idea on how to further dig into this problem? Kind regards, Lars Corbijn Instrumentation: PetscLogDouble lastCurrent, current; int rank; MPI_Comm_rank(PETSC_COMM_WORLD, &rank); for(int i = 0; i < 100000000; ++i) { PetscMemoryGetCurrentUsage(&lastCurrent); KSPSolve(A_, vec1_, vec2_); PetscMemoryGetCurrentUsage(¤t); if(current != lastCurrent) { std::cout << std::setw(2) << rank << " " << std::setw(6) << i << " current " << std::setw(8) << (int) current << std::right << "(" << std::setw(6) << (int)(current - lastCurrent) << ")" << std::endl; } lastCurrent = current; } Matrix details The matrix A in question is created from a complex valued matrix C_ (type mataij) using the following code (modulo renames). Theoretically it should be a Laplacian with phase-shift periodic boundary conditions MatHermitianTranspose(C_, MAT_INITIAL_MATRIX, &Y_); MatProductCreate(C_, Y_, NULL, & A_); MatProductSetType(A_, MATPRODUCT_AB); MatProductSetFromOptions(A_); MatProductSymbolic(A_); MatProductNumeric(A_); Petsc arguments: -log_view_memory -log_view :petsc.log -ksp_type preonly -pc_type lu -pc_factor_mat_solver_type mumps -bv_matmult vecs -memory_view From bsmith at petsc.dev Thu Sep 5 12:44:45 2024 From: bsmith at petsc.dev (Barry Smith) Date: Thu, 5 Sep 2024 13:44:45 -0400 Subject: [petsc-users] KSPSolve + MUMPS memory growth issues In-Reply-To: <68AAED91-ED33-4E9F-BCE1-55C88020236C@utwente.nl> References: <68AAED91-ED33-4E9F-BCE1-55C88020236C@utwente.nl> Message-ID: <5EB092FC-9307-46B3-AB46-BB36240D50F0@petsc.dev> Use Valgrind. It will show the exact lines that memory is allocated that does not get freed later. I am guess some memory allocation within MUMPS is not properly freed. Barry > On Sep 5, 2024, at 1:33?PM, Corbijn van Willenswaard, Lars (UT) via petsc-users wrote: > > Dear PETSc, > > For the last months I?ve struggled with a solver that I wrote for a FEM eigenvalue problem running out of memory. I?ve traced it to KSPSolve + MUMPS being the issue, but I'm getting stuck on digging deeper. > > The reason I suspect the KSPSolve/MUMPS is that when commenting out the KSPSolve the memory stays constant while running the rest of the algorithm. Of course, the algorithm also converges to a different result in this setup. When changing the KSP statement to > for(int i = 0; i < 100000000; i++) KSPSolve(A_, vec1_, vec2_); > the memory grows faster than when running the algorithm. Logging shows that the program never the terminating i=100M. Measuring the memory growth using ps (started debugging before I knew of PETSc's features) I see a growth in the RSS on a single compute node of up to 300MB/min for this artificial case. Real cases grow more like 60MB/min/node, which causes a kill due to memory exhaustion after about 2-3 days. > > Locally (Mac) I've been able to reproduce this both with 6 MPI processes and with a single one. Instrumenting the code to show differences in PetscMemoryGetCurrentUsage (full code below), shows that the memory increases every step at the start, but also does at later iterations (small excerpt from the output): > rank step memory (increase since prev step) > 0 6544 current 39469056( 8192) > 0 7086 current 39477248( 8192) > 0 7735 current 39497728( 20480) > 0 9029 current 39501824( 4096) > A similar output is visible in a run with 6 ranks, where there does not seem to be a pattern as to which of the ranks increases at which step. (Note I've checked PetscMallocGetCurrentUsage, but that is constant) > > Switching the solver to petsc's own solver on a single rank does not show a memory increase after the first solve. Changing the solve to overwrite the vector will result in a few increases after the first solve, but these do not seem to repeat. So, changes like VecCopy(vec2, vec1_); KSPSolve(A_, vec1_, vec1_);. > > Does anyone have an idea on how to further dig into this problem? > > Kind regards, > Lars Corbijn > > > Instrumentation: > > PetscLogDouble lastCurrent, current; > int rank; > MPI_Comm_rank(PETSC_COMM_WORLD, &rank); > for(int i = 0; i < 100000000; ++i) { > PetscMemoryGetCurrentUsage(&lastCurrent); > KSPSolve(A_, vec1_, vec2_); > PetscMemoryGetCurrentUsage(¤t); > if(current != lastCurrent) { > std::cout << std::setw(2) << rank << " " << std::setw(6) << i > << " current " << std::setw(8) << (int) current << std::right > << "(" << std::setw(6) << (int)(current - lastCurrent) << ")" > << std::endl; > } > lastCurrent = current; > } > > > Matrix details > The matrix A in question is created from a complex valued matrix C_ (type mataij) using the following code (modulo renames). Theoretically it should be a Laplacian with phase-shift periodic boundary conditions > MatHermitianTranspose(C_, MAT_INITIAL_MATRIX, &Y_); > MatProductCreate(C_, Y_, NULL, & A_); > MatProductSetType(A_, MATPRODUCT_AB); > MatProductSetFromOptions(A_); > MatProductSymbolic(A_); > MatProductNumeric(A_); > > Petsc arguments: -log_view_memory -log_view :petsc.log -ksp_type preonly -pc_type lu -pc_factor_mat_solver_type mumps -bv_matmult vecs -memory_view > From knepley at gmail.com Thu Sep 5 12:55:47 2024 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 5 Sep 2024 13:55:47 -0400 Subject: [petsc-users] KSPSolve + MUMPS memory growth issues In-Reply-To: <68AAED91-ED33-4E9F-BCE1-55C88020236C@utwente.nl> References: <68AAED91-ED33-4E9F-BCE1-55C88020236C@utwente.nl> Message-ID: On Thu, Sep 5, 2024 at 1:40?PM Corbijn van Willenswaard, Lars (UT) via petsc-users wrote: > Dear PETSc, > > For the last months I?ve struggled with a solver that I wrote for a FEM > eigenvalue problem running out of memory. I?ve traced it to KSPSolve + > MUMPS being the issue, but I'm getting stuck on digging deeper. > > The reason I suspect the KSPSolve/MUMPS is that when commenting out the > KSPSolve the memory stays constant while running the rest of the algorithm. > Of course, the algorithm also converges to a different result in this > setup. When changing the KSP statement to > for(int i = 0; i < 100000000; i++) KSPSolve(A_, vec1_, vec2_); > the memory grows faster than when running the algorithm. Logging shows > that the program never the terminating i=100M. Measuring the memory growth > using ps (started debugging before I knew of PETSc's features) I see a > growth in the RSS on a single compute node of up to 300MB/min for this > artificial case. Real cases grow more like 60MB/min/node, which causes a > kill due to memory exhaustion after about 2-3 days. > > Locally (Mac) I've been able to reproduce this both with 6 MPI processes > and with a single one. Instrumenting the code to show differences in > PetscMemoryGetCurrentUsage (full code below), shows that the memory > increases every step at the start, but also does at later iterations (small > excerpt from the output): > rank step memory (increase since prev step) > 0 6544 current 39469056( 8192) > 0 7086 current 39477248( 8192) > 0 7735 current 39497728( 20480) > 0 9029 current 39501824( 4096) > A similar output is visible in a run with 6 ranks, where there does not > seem to be a pattern as to which of the ranks increases at which step. > (Note I've checked PetscMallocGetCurrentUsage, but that is constant) > > Switching the solver to petsc's own solver on a single rank does not show > a memory increase after the first solve. Changing the solve to overwrite > the vector will result in a few increases after the first solve, but these > do not seem to repeat. So, changes like VecCopy(vec2, vec1_); KSPSolve(A_, > vec1_, vec1_);. > > Does anyone have an idea on how to further dig into this problem? > I think the best way is to construct the simplest code that reproduces your problem. For example, we could save your matrix in a binary file -ksp_view_mat binary:mat.bin and then use a very simple code: #include int main(int argc, char **argv) { PetscViewer viewer; Mat A; Vec b, x; PetscCall(PetscInitialize(&argc, &argv, NULL, NULL)); PetscCall(PetscViewerBinaryOpen(PETSC_COMM_WORLD, "mat.bin", PETSC_MODE_READ, &viewer)); PetscCall(MatLoad(A, viewer)); PetscCall(PetscViewerDestroy(&viewer)); PetscCall(MatCreateVecs(A, &x, &b)); PetscCall(VecSet(b, 1.)); PetscCall(KSPCreate(PETSC_COMM_WORLD, &ksp)); PetscCall(KSPSetOperators(ksp, A, A)); PetscCall(KSPSetFromOptions(ksp)); for (PetscInt i = 0; i < 100000; ++i) PetscCall(KSPSolve(ksp, b, x)); PetscCall(KSPDestroy(&ksp)); PetscCall(MatDestroy(&A)); PetscCall(VecDestroy(&b)); PetscCall(VecDestroy(&x)); PetscCall(PetscFinalize()); return(0); } and see if you get memory increase. Thanks, Matt > Kind regards, > Lars Corbijn > > > Instrumentation: > > PetscLogDouble lastCurrent, current; > int rank; > MPI_Comm_rank(PETSC_COMM_WORLD, &rank); > for(int i = 0; i < 100000000; ++i) { > PetscMemoryGetCurrentUsage(&lastCurrent); > KSPSolve(A_, vec1_, vec2_); > PetscMemoryGetCurrentUsage(¤t); > if(current != lastCurrent) { > std::cout << std::setw(2) << rank << " " << std::setw(6) << i > << " current " << std::setw(8) << (int) current << > std::right > << "(" << std::setw(6) << (int)(current - lastCurrent) > << ")" > << std::endl; > } > lastCurrent = current; > } > > > Matrix details > The matrix A in question is created from a complex valued matrix C_ (type > mataij) using the following code (modulo renames). Theoretically it should > be a Laplacian with phase-shift periodic boundary conditions > MatHermitianTranspose(C_, MAT_INITIAL_MATRIX, &Y_); > MatProductCreate(C_, Y_, NULL, & A_); > MatProductSetType(A_, MATPRODUCT_AB); > MatProductSetFromOptions(A_); > MatProductSymbolic(A_); > MatProductNumeric(A_); > > Petsc arguments: -log_view_memory -log_view :petsc.log -ksp_type preonly > -pc_type lu -pc_factor_mat_solver_type mumps -bv_matmult vecs -memory_view > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aOL-SsOXXPOKEgMX-Lf2zBM59ObfrSKQMpa_bh8gN0Lwam9yu6vKE1ZBQW3nM26gVAYbVnMsu_zXglv_gOeR$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From l.j.corbijnvanwillenswaard at utwente.nl Thu Sep 5 13:46:48 2024 From: l.j.corbijnvanwillenswaard at utwente.nl (Corbijn van Willenswaard, Lars (UT)) Date: Thu, 5 Sep 2024 18:46:48 +0000 Subject: [petsc-users] KSPSolve + MUMPS memory growth issues In-Reply-To: References: <68AAED91-ED33-4E9F-BCE1-55C88020236C@utwente.nl> Message-ID: Thank you, that makes testing so much easier. So far, I?ve been able to shrink the matrix (now only 64x64) and see that it still has growing memory usage over time. Unfortunately, I?ve no access to a linux machine right now, so running through valgrind like Barry suggested has to wait. Lars From: Matthew Knepley Date: Thursday, 5 September 2024 at 19:56 To: "Corbijn van Willenswaard, Lars (UT)" Cc: "petsc-users at mcs.anl.gov" Subject: Re: [petsc-users] KSPSolve + MUMPS memory growth issues On Thu, Sep 5, 2024 at 1:40?PM Corbijn van Willenswaard, Lars (UT) via petsc-users > wrote: Dear PETSc, For the last months I?ve struggled with a solver that I wrote for a FEM eigenvalue problem running out of memory. I?ve traced it to KSPSolve + MUMPS being the issue, but I'm getting stuck on digging deeper. The reason I suspect the KSPSolve/MUMPS is that when commenting out the KSPSolve the memory stays constant while running the rest of the algorithm. Of course, the algorithm also converges to a different result in this setup. When changing the KSP statement to for(int i = 0; i < 100000000; i++) KSPSolve(A_, vec1_, vec2_); the memory grows faster than when running the algorithm. Logging shows that the program never the terminating i=100M. Measuring the memory growth using ps (started debugging before I knew of PETSc's features) I see a growth in the RSS on a single compute node of up to 300MB/min for this artificial case. Real cases grow more like 60MB/min/node, which causes a kill due to memory exhaustion after about 2-3 days. Locally (Mac) I've been able to reproduce this both with 6 MPI processes and with a single one. Instrumenting the code to show differences in PetscMemoryGetCurrentUsage (full code below), shows that the memory increases every step at the start, but also does at later iterations (small excerpt from the output): rank step memory (increase since prev step) 0 6544 current 39469056( 8192) 0 7086 current 39477248( 8192) 0 7735 current 39497728( 20480) 0 9029 current 39501824( 4096) A similar output is visible in a run with 6 ranks, where there does not seem to be a pattern as to which of the ranks increases at which step. (Note I've checked PetscMallocGetCurrentUsage, but that is constant) Switching the solver to petsc's own solver on a single rank does not show a memory increase after the first solve. Changing the solve to overwrite the vector will result in a few increases after the first solve, but these do not seem to repeat. So, changes like VecCopy(vec2, vec1_); KSPSolve(A_, vec1_, vec1_);. Does anyone have an idea on how to further dig into this problem? I think the best way is to construct the simplest code that reproduces your problem. For example, we could save your matrix in a binary file -ksp_view_mat binary:mat.bin and then use a very simple code: #include int main(int argc, char **argv) { PetscViewer viewer; Mat A; Vec b, x; PetscCall(PetscInitialize(&argc, &argv, NULL, NULL)); PetscCall(PetscViewerBinaryOpen(PETSC_COMM_WORLD, "mat.bin", PETSC_MODE_READ, &viewer)); PetscCall(MatLoad(A, viewer)); PetscCall(PetscViewerDestroy(&viewer)); PetscCall(MatCreateVecs(A, &x, &b)); PetscCall(VecSet(b, 1.)); PetscCall(KSPCreate(PETSC_COMM_WORLD, &ksp)); PetscCall(KSPSetOperators(ksp, A, A)); PetscCall(KSPSetFromOptions(ksp)); for (PetscInt i = 0; i < 100000; ++i) PetscCall(KSPSolve(ksp, b, x)); PetscCall(KSPDestroy(&ksp)); PetscCall(MatDestroy(&A)); PetscCall(VecDestroy(&b)); PetscCall(VecDestroy(&x)); PetscCall(PetscFinalize()); return(0); } and see if you get memory increase. Thanks, Matt Kind regards, Lars Corbijn Instrumentation: PetscLogDouble lastCurrent, current; int rank; MPI_Comm_rank(PETSC_COMM_WORLD, &rank); for(int i = 0; i < 100000000; ++i) { PetscMemoryGetCurrentUsage(&lastCurrent); KSPSolve(A_, vec1_, vec2_); PetscMemoryGetCurrentUsage(¤t); if(current != lastCurrent) { std::cout << std::setw(2) << rank << " " << std::setw(6) << i << " current " << std::setw(8) << (int) current << std::right << "(" << std::setw(6) << (int)(current - lastCurrent) << ")" << std::endl; } lastCurrent = current; } Matrix details The matrix A in question is created from a complex valued matrix C_ (type mataij) using the following code (modulo renames). Theoretically it should be a Laplacian with phase-shift periodic boundary conditions MatHermitianTranspose(C_, MAT_INITIAL_MATRIX, &Y_); MatProductCreate(C_, Y_, NULL, & A_); MatProductSetType(A_, MATPRODUCT_AB); MatProductSetFromOptions(A_); MatProductSymbolic(A_); MatProductNumeric(A_); Petsc arguments: -log_view_memory -log_view :petsc.log -ksp_type preonly -pc_type lu -pc_factor_mat_solver_type mumps -bv_matmult vecs -memory_view -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bKTdR_AbB6A1b2l-FZyzbS1zaRY0f2Hc1eIWUu_y4SfPUGOHmJ-ik6LjzXWVOBLBSWuHNjKRZnpPG5HOaak5fTmPe4-kJx8gGlMh$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Sep 5 13:47:51 2024 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 5 Sep 2024 14:47:51 -0400 Subject: [petsc-users] KSPSolve + MUMPS memory growth issues In-Reply-To: References: <68AAED91-ED33-4E9F-BCE1-55C88020236C@utwente.nl> Message-ID: On Thu, Sep 5, 2024 at 2:46?PM Corbijn van Willenswaard, Lars (UT) < l.j.corbijnvanwillenswaard at utwente.nl> wrote: > Thank you, that makes testing so much easier. So far, I?ve been able to > shrink the matrix (now only 64x64) and see that it still has growing memory > usage over time. Unfortunately, I?ve no access to a linux machine right > now, so running through valgrind like Barry suggested has to wait. > Just to check. Your matrix, with that simple code, producing growing memory? If you send the matrix, I will find the problem. Thanks, Matt > Lars > > > > *From: *Matthew Knepley > *Date: *Thursday, 5 September 2024 at 19:56 > *To: *"Corbijn van Willenswaard, Lars (UT)" < > l.j.corbijnvanwillenswaard at utwente.nl> > *Cc: *"petsc-users at mcs.anl.gov" > *Subject: *Re: [petsc-users] KSPSolve + MUMPS memory growth issues > > > > On Thu, Sep 5, 2024 at 1:40?PM Corbijn van Willenswaard, Lars (UT) via > petsc-users wrote: > > Dear PETSc, > > For the last months I?ve struggled with a solver that I wrote for a FEM > eigenvalue problem running out of memory. I?ve traced it to KSPSolve + > MUMPS being the issue, but I'm getting stuck on digging deeper. > > The reason I suspect the KSPSolve/MUMPS is that when commenting out the > KSPSolve the memory stays constant while running the rest of the algorithm. > Of course, the algorithm also converges to a different result in this > setup. When changing the KSP statement to > for(int i = 0; i < 100000000; i++) KSPSolve(A_, vec1_, vec2_); > the memory grows faster than when running the algorithm. Logging shows > that the program never the terminating i=100M. Measuring the memory growth > using ps (started debugging before I knew of PETSc's features) I see a > growth in the RSS on a single compute node of up to 300MB/min for this > artificial case. Real cases grow more like 60MB/min/node, which causes a > kill due to memory exhaustion after about 2-3 days. > > Locally (Mac) I've been able to reproduce this both with 6 MPI processes > and with a single one. Instrumenting the code to show differences in > PetscMemoryGetCurrentUsage (full code below), shows that the memory > increases every step at the start, but also does at later iterations (small > excerpt from the output): > rank step memory (increase since prev step) > 0 6544 current 39469056( 8192) > 0 7086 current 39477248( 8192) > 0 7735 current 39497728( 20480) > 0 9029 current 39501824( 4096) > A similar output is visible in a run with 6 ranks, where there does not > seem to be a pattern as to which of the ranks increases at which step. > (Note I've checked PetscMallocGetCurrentUsage, but that is constant) > > Switching the solver to petsc's own solver on a single rank does not show > a memory increase after the first solve. Changing the solve to overwrite > the vector will result in a few increases after the first solve, but these > do not seem to repeat. So, changes like VecCopy(vec2, vec1_); KSPSolve(A_, > vec1_, vec1_);. > > Does anyone have an idea on how to further dig into this problem? > > > > I think the best way is to construct the simplest code that reproduces > your problem. For example, we could save your matrix in a binary file > > > > -ksp_view_mat binary:mat.bin > > > > and then use a very simple code: > > > > #include > > int main(int argc, char **argv) > { > PetscViewer viewer; > Mat A; > Vec b, x; > > PetscCall(PetscInitialize(&argc, &argv, NULL, NULL)); > PetscCall(PetscViewerBinaryOpen(PETSC_COMM_WORLD, "mat.bin", > PETSC_MODE_READ, &viewer)); > PetscCall(MatLoad(A, viewer)); > PetscCall(PetscViewerDestroy(&viewer)); > PetscCall(MatCreateVecs(A, &x, &b)); > PetscCall(VecSet(b, 1.)); > > PetscCall(KSPCreate(PETSC_COMM_WORLD, &ksp)); > PetscCall(KSPSetOperators(ksp, A, A)); > PetscCall(KSPSetFromOptions(ksp)); > for (PetscInt i = 0; i < 100000; ++i) PetscCall(KSPSolve(ksp, b, x)); > PetscCall(KSPDestroy(&ksp)); > > PetscCall(MatDestroy(&A)); > PetscCall(VecDestroy(&b)); > PetscCall(VecDestroy(&x)); > PetscCall(PetscFinalize()); > return(0); > } > > > > and see if you get memory increase. > > > > Thanks, > > > > Matt > > > > Kind regards, > Lars Corbijn > > > Instrumentation: > > PetscLogDouble lastCurrent, current; > int rank; > MPI_Comm_rank(PETSC_COMM_WORLD, &rank); > for(int i = 0; i < 100000000; ++i) { > PetscMemoryGetCurrentUsage(&lastCurrent); > KSPSolve(A_, vec1_, vec2_); > PetscMemoryGetCurrentUsage(¤t); > if(current != lastCurrent) { > std::cout << std::setw(2) << rank << " " << std::setw(6) << i > << " current " << std::setw(8) << (int) current << > std::right > << "(" << std::setw(6) << (int)(current - lastCurrent) > << ")" > << std::endl; > } > lastCurrent = current; > } > > > Matrix details > The matrix A in question is created from a complex valued matrix C_ (type > mataij) using the following code (modulo renames). Theoretically it should > be a Laplacian with phase-shift periodic boundary conditions > MatHermitianTranspose(C_, MAT_INITIAL_MATRIX, &Y_); > MatProductCreate(C_, Y_, NULL, & A_); > MatProductSetType(A_, MATPRODUCT_AB); > MatProductSetFromOptions(A_); > MatProductSymbolic(A_); > MatProductNumeric(A_); > > Petsc arguments: -log_view_memory -log_view :petsc.log -ksp_type preonly > -pc_type lu -pc_factor_mat_solver_type mumps -bv_matmult vecs -memory_view > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!a3fMSsz8iwy1xT-RP5SLCxWCyf4c8fdJbl-sBGzc7G9RNSjBFaOZMQIIs26Kkmx86CA_koT3TvTMMeiNQTeQ$ > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!a3fMSsz8iwy1xT-RP5SLCxWCyf4c8fdJbl-sBGzc7G9RNSjBFaOZMQIIs26Kkmx86CA_koT3TvTMMeiNQTeQ$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From karthikeyan.chockalingam at stfc.ac.uk Thu Sep 5 16:20:19 2024 From: karthikeyan.chockalingam at stfc.ac.uk (Karthikeyan Chockalingam - STFC UKRI) Date: Thu, 5 Sep 2024 21:20:19 +0000 Subject: [petsc-users] MPIAIJ MatMult and non conforming object sizes Message-ID: Hello, I am unsure why the program crashes even while running the code serially petscErr = MatCreate(mesh.comm().get(), &par_G); petscErr = MatSetType(par_G, MATMPIAIJ); petscErr = MatSetSizes(par_G, local_num_rows, local_num_cols, total_num_rows, total_num_cols); PetscInt d_nz = 2; PetscInt o_nz = 2; petscErr = MatMPIAIJSetPreallocation(par_G, d_nz, NULL, o_nz, NULL); petscErr = VecCreateMPI(mesh.comm().get(),local_num_cols,total_num_cols,&par_xcoord); petscErr = VecDuplicate(par_xcoord,&par_xvec); petscErr = MatMult(par_G, par_xcoord, par_xvec); [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Nonconforming object sizes [0]PETSC ERROR: Mat mat,Vec y: global dim 40 25 [0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc! [0]PETSC ERROR: Option left: name:--keep-cout (no value) source: command line [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!eNf_b1At1lAvdYeAlplIXxQN9WwVYKe1gkVUZRvNkdF0V6JCbVOG_mYTuCQrp2rs8FMod_waFjX0LLNuAvMcHmdf7Hrzm4s3JaQT$ for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.20.3, unknown [0]PETSC ERROR: ./example-dbg on a arch-moose named HC20210312 by karthikeyan.chockalingam Thu Sep 5 22:16:54 2024 [0]PETSC ERROR: Configure options --with-64-bit-indices --with-cxx-dialect=C++17 --with-debugging=no --with-fortran-bindings=0 --with-mpi=1 --with-openmp=1 --with-shared-libraries=1 --with-sowing=0 --download-fblaslapack=1 --download-hypre=1 --download-metis=1 --download-mumps=1 --download-ptscotch=1 --download-parmetis=1 --download-scalapack=1 --download-slepc=1 --download-strumpack=1 --download-superlu_dist=1 --download-bison=1 --download-hdf5=1 --with-hdf5-fortran-bindings=0 --download-hdf5-configure-arguments="--with-zlib" --with-make-np=4 [0]PETSC ERROR: #1 MatMult() at /Users/karthikeyan.chockalingam/moose/petsc/src/mat/interface/matrix.c:2587 Vec Object: 1 MPI process Many thanks for the help! Karthik. -- Karthik Chockalingam, Ph.D. Senior Research Software Engineer High Performance Systems Engineering Group Hartree Centre | Science and Technology Facilities Council karthikeyan.chockalingam at stfc.ac.uk [signature_3970890138] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 18270 bytes Desc: image001.png URL: From junchao.zhang at gmail.com Thu Sep 5 17:03:57 2024 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Thu, 5 Sep 2024 17:03:57 -0500 Subject: [petsc-users] MPIAIJ MatMult and non conforming object sizes In-Reply-To: References: Message-ID: It is triggered by PetscCheck(mat->rmap->N == y->map->N, PetscObjectComm((PetscObject)mat), PETSC_ERR_ARG_SIZ, "Mat mat,Vec y: global dim %" PetscInt_FMT " %" PetscInt_FMT, mat->rmap->N, y->map->N); The error says in your y = Ax, A has 40 rows but y has 25. Generally, VecDuplicate(par_xcoord,&par_xvec) is wrong since you assume x, y have the same size. --Junchao Zhang On Thu, Sep 5, 2024 at 4:20?PM Karthikeyan Chockalingam - STFC UKRI via petsc-users wrote: > Hello, > > > > I am unsure why the program crashes even while running the code serially > > > > > > petscErr = MatCreate(mesh.comm().get(), &par_G); > > petscErr = MatSetType(par_G, MATMPIAIJ); > > petscErr = MatSetSizes(par_G, local_num_rows, local_num_cols, > total_num_rows, total_num_cols); > > PetscInt d_nz = 2; > > PetscInt o_nz = 2; > > petscErr = MatMPIAIJSetPreallocation(par_G, d_nz, NULL, o_nz, NULL); > > > > petscErr = > VecCreateMPI(mesh.comm().get(),local_num_cols,total_num_cols,&par_xcoord); > > petscErr = VecDuplicate(par_xcoord,&par_xvec); > > > > petscErr = MatMult(par_G, par_xcoord, par_xvec); > > > > > > > > > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [0]PETSC ERROR: Nonconforming object sizes > > [0]PETSC ERROR: Mat mat,Vec y: global dim 40 25 > > [0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the > program crashed before usage or a spelling mistake, etc! > > [0]PETSC ERROR: Option left: name:--keep-cout (no value) source: command > line > > [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!auIH0jsOSEODe54KPaiJXMFaybeyFhBQSkvRAJLVAxRJbngqy9YCy0HUTTTAeLyEKq2W392V9Qix5MS7MaYhlkgHlnvb$ > > for trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.20.3, unknown > > [0]PETSC ERROR: ./example-dbg on a arch-moose named HC20210312 by > karthikeyan.chockalingam Thu Sep 5 22:16:54 2024 > > [0]PETSC ERROR: Configure options --with-64-bit-indices > --with-cxx-dialect=C++17 --with-debugging=no --with-fortran-bindings=0 > --with-mpi=1 --with-openmp=1 --with-shared-libraries=1 --with-sowing=0 > --download-fblaslapack=1 --download-hypre=1 --download-metis=1 > --download-mumps=1 --download-ptscotch=1 --download-parmetis=1 > --download-scalapack=1 --download-slepc=1 --download-strumpack=1 > --download-superlu_dist=1 --download-bison=1 --download-hdf5=1 > --with-hdf5-fortran-bindings=0 > --download-hdf5-configure-arguments="--with-zlib" --with-make-np=4 > > [0]PETSC ERROR: #1 MatMult() at > /Users/karthikeyan.chockalingam/moose/petsc/src/mat/interface/matrix.c:2587 > > Vec Object: 1 MPI process > > > > > > Many thanks for the help! > > > > Karthik. > > > > > > > > -- > > *Karthik Chockalingam, Ph.D.* > > Senior Research Software Engineer > > High Performance Systems Engineering Group > > Hartree Centre | Science and Technology Facilities Council > > karthikeyan.chockalingam at stfc.ac.uk > > > > [image: signature_3970890138] > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 18270 bytes Desc: not available URL: From bsmith at petsc.dev Thu Sep 5 17:19:30 2024 From: bsmith at petsc.dev (Barry Smith) Date: Thu, 5 Sep 2024 18:19:30 -0400 Subject: [petsc-users] MPIAIJ MatMult and non conforming object sizes In-Reply-To: References: Message-ID: <5182A32E-FB56-4B55-9947-A8C387355843@petsc.dev> You can use MatCreateVecs() to create vectors sized properly for both x and y in y = Ax when A is not square. > On Sep 5, 2024, at 5:20?PM, Karthikeyan Chockalingam - STFC UKRI via petsc-users wrote: > > Hello, > > I am unsure why the program crashes even while running the code serially > > > petscErr = MatCreate(mesh.comm().get(), &par_G); > petscErr = MatSetType(par_G, MATMPIAIJ); > petscErr = MatSetSizes(par_G, local_num_rows, local_num_cols, total_num_rows, total_num_cols); > PetscInt d_nz = 2; > PetscInt o_nz = 2; > petscErr = MatMPIAIJSetPreallocation(par_G, d_nz, NULL, o_nz, NULL); > > petscErr = VecCreateMPI(mesh.comm().get(),local_num_cols,total_num_cols,&par_xcoord); > petscErr = VecDuplicate(par_xcoord,&par_xvec); > > petscErr = MatMult(par_G, par_xcoord, par_xvec); > > > > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Nonconforming object sizes > [0]PETSC ERROR: Mat mat,Vec y: global dim 40 25 > [0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc! > [0]PETSC ERROR: Option left: name:--keep-cout (no value) source: command line > [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!bSaK_Wa6QhLPg-VTwXikWgAOcFo9HA3dOF3pqzXGgsJFZH1LHVzj4YWXyrj_lxhs5Thig6VFoXMS4dnkpcyJGYw$ for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.20.3, unknown > [0]PETSC ERROR: ./example-dbg on a arch-moose named HC20210312 by karthikeyan.chockalingam Thu Sep 5 22:16:54 2024 > [0]PETSC ERROR: Configure options --with-64-bit-indices --with-cxx-dialect=C++17 --with-debugging=no --with-fortran-bindings=0 --with-mpi=1 --with-openmp=1 --with-shared-libraries=1 --with-sowing=0 --download-fblaslapack=1 --download-hypre=1 --download-metis=1 --download-mumps=1 --download-ptscotch=1 --download-parmetis=1 --download-scalapack=1 --download-slepc=1 --download-strumpack=1 --download-superlu_dist=1 --download-bison=1 --download-hdf5=1 --with-hdf5-fortran-bindings=0 --download-hdf5-configure-arguments="--with-zlib" --with-make-np=4 > [0]PETSC ERROR: #1 MatMult() at /Users/karthikeyan.chockalingam/moose/petsc/src/mat/interface/matrix.c:2587 > Vec Object: 1 MPI process > > > Many thanks for the help! > > Karthik. > > > > -- > Karthik Chockalingam, Ph.D. > Senior Research Software Engineer > High Performance Systems Engineering Group > Hartree Centre | Science and Technology Facilities Council > karthikeyan.chockalingam at stfc.ac.uk > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From karthikeyan.chockalingam at stfc.ac.uk Thu Sep 5 17:47:57 2024 From: karthikeyan.chockalingam at stfc.ac.uk (Karthikeyan Chockalingam - STFC UKRI) Date: Thu, 5 Sep 2024 22:47:57 +0000 Subject: [petsc-users] MPIAIJ MatMult and non conforming object sizes In-Reply-To: <5182A32E-FB56-4B55-9947-A8C387355843@petsc.dev> References: <5182A32E-FB56-4B55-9947-A8C387355843@petsc.dev> Message-ID: Thank you both for your response. From: Barry Smith Date: Thursday, 5 September 2024 at 23:19 To: Chockalingam, Karthikeyan (STFC,DL,HC) Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] MPIAIJ MatMult and non conforming object sizes You can use MatCreateVecs() to create vectors sized properly for both x and y in y = Ax when A is not square. On Sep 5, 2024, at 5:20?PM, Karthikeyan Chockalingam - STFC UKRI via petsc-users wrote: Hello, I am unsure why the program crashes even while running the code serially petscErr = MatCreate(mesh.comm().get(), &par_G); petscErr = MatSetType(par_G, MATMPIAIJ); petscErr = MatSetSizes(par_G, local_num_rows, local_num_cols, total_num_rows, total_num_cols); PetscInt d_nz = 2; PetscInt o_nz = 2; petscErr = MatMPIAIJSetPreallocation(par_G, d_nz, NULL, o_nz, NULL); petscErr = VecCreateMPI(mesh.comm().get(),local_num_cols,total_num_cols,&par_xcoord); petscErr = VecDuplicate(par_xcoord,&par_xvec); petscErr = MatMult(par_G, par_xcoord, par_xvec); [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Nonconforming object sizes [0]PETSC ERROR: Mat mat,Vec y: global dim 40 25 [0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc! [0]PETSC ERROR: Option left: name:--keep-cout (no value) source: command line [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!dkFKYMJODzVEEvczpl2dC8oOLyzskTQ_Q3FJm8ewb0TiuxAwrMXwH1eMCstakS6pUT5C04m47ZecPpQFRWqJ1jRXwEU1LXKGK3E8$ for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.20.3, unknown [0]PETSC ERROR: ./example-dbg on a arch-moose named HC20210312 by karthikeyan.chockalingam Thu Sep 5 22:16:54 2024 [0]PETSC ERROR: Configure options --with-64-bit-indices --with-cxx-dialect=C++17 --with-debugging=no --with-fortran-bindings=0 --with-mpi=1 --with-openmp=1 --with-shared-libraries=1 --with-sowing=0 --download-fblaslapack=1 --download-hypre=1 --download-metis=1 --download-mumps=1 --download-ptscotch=1 --download-parmetis=1 --download-scalapack=1 --download-slepc=1 --download-strumpack=1 --download-superlu_dist=1 --download-bison=1 --download-hdf5=1 --with-hdf5-fortran-bindings=0 --download-hdf5-configure-arguments="--with-zlib" --with-make-np=4 [0]PETSC ERROR: #1 MatMult() at /Users/karthikeyan.chockalingam/moose/petsc/src/mat/interface/matrix.c:2587 Vec Object: 1 MPI process Many thanks for the help! Karthik. -- Karthik Chockalingam, Ph.D. Senior Research Software Engineer High Performance Systems Engineering Group Hartree Centre | Science and Technology Facilities Council karthikeyan.chockalingam at stfc.ac.uk -------------- next part -------------- An HTML attachment was scrubbed... URL: From karthikeyan.chockalingam at stfc.ac.uk Fri Sep 6 10:37:28 2024 From: karthikeyan.chockalingam at stfc.ac.uk (Karthikeyan Chockalingam - STFC UKRI) Date: Fri, 6 Sep 2024 15:37:28 +0000 Subject: [petsc-users] Hypre AMS usage Message-ID: Hello, I am trying to use the Hypre AMS preconditioner for the first time. I am following the example problem from Hypre https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L954__;Iw!!G_uCfscf7eWS!biuCOXkV-qD-e8iTXiwaD9XJ0EWuqr2PidZJdPqbnFzCshQFo-mclGSvGBOPuGSTkjPvMKnIfJzWos7QShcSitpwyrhL5ySgA0XM$ I have so far successfully set the discrete gradient operator and vertex co-ordinates, // Set discrete gradient petscErr = PCHYPRESetDiscreteGradient(pc, par_G); // Set vertex coordinates petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, par_zvec); Do I need to set the following smoothing options? https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L965__;Iw!!G_uCfscf7eWS!biuCOXkV-qD-e8iTXiwaD9XJ0EWuqr2PidZJdPqbnFzCshQFo-mclGSvGBOPuGSTkjPvMKnIfJzWos7QShcSitpwyrhL55yOMB-v$ Also, do I need to convert from MATMPIAIJ to CSR? https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L984__;Iw!!G_uCfscf7eWS!biuCOXkV-qD-e8iTXiwaD9XJ0EWuqr2PidZJdPqbnFzCshQFo-mclGSvGBOPuGSTkjPvMKnIfJzWos7QShcSitpwyrhL53Trv1si$ What are the other PETSc calls to invoke AMS? Is there an example problem I can look at? Thank you. Karthik. -- Karthik Chockalingam, Ph.D. Senior Research Software Engineer High Performance Systems Engineering Group Hartree Centre | Science and Technology Facilities Council karthikeyan.chockalingam at stfc.ac.uk [signature_3970890138] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 18270 bytes Desc: image001.png URL: From knepley at gmail.com Fri Sep 6 11:56:52 2024 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 6 Sep 2024 12:56:52 -0400 Subject: [petsc-users] Hypre AMS usage In-Reply-To: References: Message-ID: On Fri, Sep 6, 2024 at 11:37?AM Karthikeyan Chockalingam - STFC UKRI via petsc-users wrote: > Hello, > > > > I am trying to use the Hypre AMS preconditioner for the first time. > > > > I am following the example problem from Hypre > > > https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L954__;Iw!!G_uCfscf7eWS!Z1YtcjIl3JraPfN9GbBHhno-LcLOXJurvRZ3MKzzbqSMDLtq5x7_DICKdBj5idJnKPOPq7kmcpCvAzu6MVKD$ > > > > > I have so far successfully set the discrete gradient operator and vertex > co-ordinates, > > > > // Set discrete gradient > > petscErr = PCHYPRESetDiscreteGradient(pc, par_G); > > > > // Set vertex coordinates > > petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, > par_zvec); > > > > Do I need to set the following smoothing options? > > > https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L965__;Iw!!G_uCfscf7eWS!Z1YtcjIl3JraPfN9GbBHhno-LcLOXJurvRZ3MKzzbqSMDLtq5x7_DICKdBj5idJnKPOPq7kmcpCvA-1nohaB$ > > It cannot hurt. I would set them to begin with. > Also, do I need to convert from MATMPIAIJ to CSR? > > > https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L984__;Iw!!G_uCfscf7eWS!Z1YtcjIl3JraPfN9GbBHhno-LcLOXJurvRZ3MKzzbqSMDLtq5x7_DICKdBj5idJnKPOPq7kmcpCvA8aGosCn$ > > No. > What are the other PETSc calls to invoke AMS? Is there an example problem > I can look at? > I do not know. I don't think we have an example. Thanks, Matt > Thank you. > > > > Karthik. > > > > -- > > *Karthik Chockalingam, Ph.D.* > > Senior Research Software Engineer > > High Performance Systems Engineering Group > > Hartree Centre | Science and Technology Facilities Council > > karthikeyan.chockalingam at stfc.ac.uk > > > > [image: signature_3970890138] > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!Z1YtcjIl3JraPfN9GbBHhno-LcLOXJurvRZ3MKzzbqSMDLtq5x7_DICKdBj5idJnKPOPq7kmcpCvAyAOkwy1$ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 18270 bytes Desc: not available URL: From pierre at joliv.et Fri Sep 6 12:37:21 2024 From: pierre at joliv.et (Pierre Jolivet) Date: Fri, 6 Sep 2024 19:37:21 +0200 Subject: [petsc-users] Hypre AMS usage In-Reply-To: References: Message-ID: <088DF0FE-6F5A-40FE-92FC-D6AE7C7AC2E5@joliv.et> > On 6 Sep 2024, at 6:56?PM, Matthew Knepley wrote: > > On Fri, Sep 6, 2024 at 11:37?AM Karthikeyan Chockalingam - STFC UKRI via petsc-users > wrote: >> Hello, >> >> >> >> I am trying to use the Hypre AMS preconditioner for the first time. >> >> >> >> I am following the example problem from Hypre >> >> https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L954__;Iw!!G_uCfscf7eWS!Z9zog8mGTxJtHDwPnjywf0ppfrJPq23lCLWeE3gO-FYd_QgXq8S0g41HEHnsEbQ0YOFVE8vmL-OlCwyXA0U3cw$ >> >> >> I have so far successfully set the discrete gradient operator and vertex co-ordinates, >> >> >> >> // Set discrete gradient >> >> petscErr = PCHYPRESetDiscreteGradient(pc, par_G); >> >> >> >> // Set vertex coordinates >> >> petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, par_zvec); >> >> >> >> Do I need to set the following smoothing options? >> >> https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L965__;Iw!!G_uCfscf7eWS!Z9zog8mGTxJtHDwPnjywf0ppfrJPq23lCLWeE3gO-FYd_QgXq8S0g41HEHnsEbQ0YOFVE8vmL-OlCwwsyo4fRw$ > It cannot hurt. I would set them to begin with. > >> Also, do I need to convert from MATMPIAIJ to CSR? >> >> https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L984__;Iw!!G_uCfscf7eWS!Z9zog8mGTxJtHDwPnjywf0ppfrJPq23lCLWeE3gO-FYd_QgXq8S0g41HEHnsEbQ0YOFVE8vmL-OlCwzzrmDpiA$ > No. > >> What are the other PETSc calls to invoke AMS? Is there an example problem I can look at? >> -pc_hypre_type ams or PetscCall(PCHYPRESetType(pc, "ams")) Thanks, Pierre > I do not know. I don't think we have an example. > > Thanks, > > Matt > >> Thank you. >> >> >> >> Karthik. >> >> >> >> -- >> >> Karthik Chockalingam, Ph.D. >> >> Senior Research Software Engineer >> >> High Performance Systems Engineering Group >> >> Hartree Centre | Science and Technology Facilities Council >> >> karthikeyan.chockalingam at stfc.ac.uk >> >> >> >> >> >> > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!Z9zog8mGTxJtHDwPnjywf0ppfrJPq23lCLWeE3gO-FYd_QgXq8S0g41HEHnsEbQ0YOFVE8vmL-OlCwxghWiarQ$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From liufield at gmail.com Sat Sep 7 21:28:17 2024 From: liufield at gmail.com (neil liu) Date: Sat, 7 Sep 2024 22:28:17 -0400 Subject: [petsc-users] Questitons about viewing the mesh from "Meshing for Subsurface Flows in PETSc" Message-ID: Dear Petsc developers, I am trying to learn about the mesh refinement part. I tried to follow the instructions from the website, "Meshing for Subsurface Flows in PETSc, make -f ./gmakefile test globsearch="dm_impls_plex_tutorials-ex10_0", but it will run many tests and will not deliver " DM Object: Mesh 1 MPI process type: plexMesh in 3 dimensions: Number of 0-cells per rank: 8 Number of 1-cells per rank: 14 (4) Number of 2-cells per rank: 9 (5) Number of 3-cells per rank: 2Labels: celltype: 6 strata with value/size (3 (4), 9 (2), 0 (8), 2 (4), 1 (10), 5 (5)) depth: 4 strata with value/size (0 (8), 1 (14), 2 (9), 3 (2)) marker: 1 strata with value/size (1 (24)) Face Sets: 1 strata with value/size (1 (12)) " While it is successful if I go to the directory for ex10 and make ex10 there and run with options there. DId I miss something? In addition, in order t view the mesh, I added the following lines to ex10.c, // Create HDF5 viewer PetscViewer viewer; const char *filename = "mesh.h5"; // Output file PetscViewerCreate(PETSC_COMM_WORLD, &viewer); PetscViewerSetType(viewer, PETSCVIEWERHDF5); PetscViewerFileSetMode(viewer, FILE_MODE_WRITE); PetscViewerFileSetName(viewer, filename); // Optionally set the name of the DM object if you are saving multiple meshes PetscObjectSetName((PetscObject)dm, "MyMeshName"); // Save DM to HDF5 file DMView(dm, viewer); This wil genearte a .h5 file. Then I run petsc-with-docs-3.21.1/petsc-3.21.1/lib/petsc/bin/petsc_gen_xdmf.py mesh.h5 This gave a .xml file. I tried to load this xml file into paraview using xdmf reader. But it was not successful. Did I miss something ? Thanks a lot, Xiaodong -------------- next part -------------- An HTML attachment was scrubbed... URL: From liufield at gmail.com Sun Sep 8 03:37:27 2024 From: liufield at gmail.com (neil liu) Date: Sun, 8 Sep 2024 04:37:27 -0400 Subject: [petsc-users] Questitons about viewing the mesh from "Meshing for Subsurface Flows in PETSc" In-Reply-To: References: Message-ID: I tried vtk format by modifying the line as follows. PetscViewerSetType(viewer, PETSCVIEWERVTK); The resulting vtk file can be loaded into Paraview. But the mesh is different from the website. With ./ex10 -init_dm_plex_dim 2 -init_dm_plex_box_faces 5,5 -dm_extrude 10, the mesh is , [image: Screenshot from 2024-09-08 04-34-35.png] WIth adapt, ./ex10 -init_dm_plex_dim 2 -init_dm_plex_box_faces 5,5 -adapt -volume_constraint_1 0.01 -volume_constraint_2 0.000625 -dm_extrude 10 [image: Screenshot from 2024-09-08 04-35-45.png] The refined mesh is quite different from the website, [image: Screenshot from 2024-09-08 04-36-46.png] Did I miss something? Thanks a lot, Xiaodong On Sat, Sep 7, 2024 at 10:28?PM neil liu wrote: > Dear Petsc developers, > > I am trying to learn about the mesh refinement part. > I tried to follow the instructions from the website, "Meshing for > Subsurface Flows in PETSc, > make -f ./gmakefile test globsearch="dm_impls_plex_tutorials-ex10_0", > but it will run many tests and will not deliver " > > DM Object: Mesh 1 MPI process type: plexMesh in 3 dimensions: Number of 0-cells per rank: 8 Number of 1-cells per rank: 14 (4) Number of 2-cells per rank: 9 (5) Number of 3-cells per rank: 2Labels: celltype: 6 strata with value/size (3 (4), 9 (2), 0 (8), 2 (4), 1 (10), 5 (5)) depth: 4 strata with value/size (0 (8), 1 (14), 2 (9), 3 (2)) marker: 1 strata with value/size (1 (24)) Face Sets: 1 strata with value/size (1 (12)) > > " > While it is successful if I go to the directory for ex10 and make ex10 > there and run with options there. DId I miss something? > > In addition, in order t view the mesh, I added the following lines to > ex10.c, > > // Create HDF5 viewer > PetscViewer viewer; > const char *filename = "mesh.h5"; // Output file > PetscViewerCreate(PETSC_COMM_WORLD, &viewer); > PetscViewerSetType(viewer, PETSCVIEWERHDF5); > PetscViewerFileSetMode(viewer, FILE_MODE_WRITE); > PetscViewerFileSetName(viewer, filename); > > // Optionally set the name of the DM object if you are saving multiple > meshes > PetscObjectSetName((PetscObject)dm, "MyMeshName"); > > // Save DM to HDF5 file > DMView(dm, viewer); > > This wil genearte a .h5 file. Then I run > petsc-with-docs-3.21.1/petsc-3.21.1/lib/petsc/bin/petsc_gen_xdmf.py > mesh.h5 > > This gave a .xml file. I tried to load this xml file into paraview using > xdmf reader. But it was not successful. Did I miss something ? > > Thanks a lot, > > Xiaodong > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screenshot from 2024-09-08 04-34-35.png Type: image/png Size: 193783 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screenshot from 2024-09-08 04-35-45.png Type: image/png Size: 1000034 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screenshot from 2024-09-08 04-36-46.png Type: image/png Size: 926443 bytes Desc: not available URL: From karthikeyan.chockalingam at stfc.ac.uk Mon Sep 9 09:17:32 2024 From: karthikeyan.chockalingam at stfc.ac.uk (Karthikeyan Chockalingam - STFC UKRI) Date: Mon, 9 Sep 2024 14:17:32 +0000 Subject: [petsc-users] Hypre AMS usage In-Reply-To: References: Message-ID: Hi Matt, You mentioned it doesn?t hurt to set the smoothing flags https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L965__;Iw!!G_uCfscf7eWS!ab7xBNMnurfBQihWzSOZdJKuuhPObxZolYSuACYpW4-NN7L84moQXv7vO962vvaVT3soXUhcCaNH6_B2mLY_MzICr-c63C4mpYGo$ Do you know where I can look for the equivalent PETSc commands? Thank you. Kind regards, Karthik. From: Matthew Knepley Date: Friday, 6 September 2024 at 17:57 To: Chockalingam, Karthikeyan (STFC,DL,HC) Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Hypre AMS usage On Fri, Sep 6, 2024 at 11:37?AM Karthikeyan Chockalingam - STFC UKRI via petsc-users > wrote: Hello, I am trying to use the Hypre AMS preconditioner for the first time. I am following the example problem from Hypre https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L954__;Iw!!G_uCfscf7eWS!ab7xBNMnurfBQihWzSOZdJKuuhPObxZolYSuACYpW4-NN7L84moQXv7vO962vvaVT3soXUhcCaNH6_B2mLY_MzICr-c63EQeUx5h$ I have so far successfully set the discrete gradient operator and vertex co-ordinates, // Set discrete gradient petscErr = PCHYPRESetDiscreteGradient(pc, par_G); // Set vertex coordinates petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, par_zvec); Do I need to set the following smoothing options? https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L965__;Iw!!G_uCfscf7eWS!ab7xBNMnurfBQihWzSOZdJKuuhPObxZolYSuACYpW4-NN7L84moQXv7vO962vvaVT3soXUhcCaNH6_B2mLY_MzICr-c63C4mpYGo$ It cannot hurt. I would set them to begin with. Also, do I need to convert from MATMPIAIJ to CSR? https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L984__;Iw!!G_uCfscf7eWS!ab7xBNMnurfBQihWzSOZdJKuuhPObxZolYSuACYpW4-NN7L84moQXv7vO962vvaVT3soXUhcCaNH6_B2mLY_MzICr-c63J_yBxgt$ No. What are the other PETSc calls to invoke AMS? Is there an example problem I can look at? I do not know. I don't think we have an example. Thanks, Matt Thank you. Karthik. -- Karthik Chockalingam, Ph.D. Senior Research Software Engineer High Performance Systems Engineering Group Hartree Centre | Science and Technology Facilities Council karthikeyan.chockalingam at stfc.ac.uk [signature_3970890138] -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!ab7xBNMnurfBQihWzSOZdJKuuhPObxZolYSuACYpW4-NN7L84moQXv7vO962vvaVT3soXUhcCaNH6_B2mLY_MzICr-c63F-LoyUA$ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 18270 bytes Desc: image001.png URL: From knepley at gmail.com Mon Sep 9 10:49:51 2024 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 9 Sep 2024 11:49:51 -0400 Subject: [petsc-users] Hypre AMS usage In-Reply-To: References: Message-ID: On Mon, Sep 9, 2024 at 10:17?AM Karthikeyan Chockalingam - STFC UKRI < karthikeyan.chockalingam at stfc.ac.uk> wrote: > Hi Matt, > > > > You mentioned it doesn?t hurt to set the smoothing flags > > > > > https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L965__;Iw!!G_uCfscf7eWS!dM_jKT_vX0NCVNcY9XOTTUaTSW-TF2CgOkB2UDeZ2ztYXapdmLlSBT4XhV7PknN3ewygGWDbKFEP9UgAwQR8$ > > > Do you know where I can look for the equivalent PETSc commands? Thank you. > Yes, this is described here: https://urldefense.us/v3/__https://petsc.org/main/manualpages/PC/PCHYPRE/__;!!G_uCfscf7eWS!dM_jKT_vX0NCVNcY9XOTTUaTSW-TF2CgOkB2UDeZ2ztYXapdmLlSBT4XhV7PknN3ewygGWDbKFEP9e0RfePC$ The best way is to run with -help or look at the source. Thanks, Matt > > > Kind regards, > > Karthik. > > > > *From: *Matthew Knepley > *Date: *Friday, 6 September 2024 at 17:57 > *To: *Chockalingam, Karthikeyan (STFC,DL,HC) < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *petsc-users at mcs.anl.gov > *Subject: *Re: [petsc-users] Hypre AMS usage > > On Fri, Sep 6, 2024 at 11:37?AM Karthikeyan Chockalingam - STFC UKRI via > petsc-users wrote: > > Hello, > > > > I am trying to use the Hypre AMS preconditioner for the first time. > > > > I am following the example problem from Hypre > > > https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L954__;Iw!!G_uCfscf7eWS!dM_jKT_vX0NCVNcY9XOTTUaTSW-TF2CgOkB2UDeZ2ztYXapdmLlSBT4XhV7PknN3ewygGWDbKFEP9ceRKXFn$ > > > > > I have so far successfully set the discrete gradient operator and vertex > co-ordinates, > > > > // Set discrete gradient > > petscErr = PCHYPRESetDiscreteGradient(pc, par_G); > > > > // Set vertex coordinates > > petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, > par_zvec); > > > > Do I need to set the following smoothing options? > > > https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L965__;Iw!!G_uCfscf7eWS!dM_jKT_vX0NCVNcY9XOTTUaTSW-TF2CgOkB2UDeZ2ztYXapdmLlSBT4XhV7PknN3ewygGWDbKFEP9UgAwQR8$ > > > > > It cannot hurt. I would set them to begin with. > > > > Also, do I need to convert from MATMPIAIJ to CSR? > > > https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L984__;Iw!!G_uCfscf7eWS!dM_jKT_vX0NCVNcY9XOTTUaTSW-TF2CgOkB2UDeZ2ztYXapdmLlSBT4XhV7PknN3ewygGWDbKFEP9culzuUp$ > > > > > No. > > > > What are the other PETSc calls to invoke AMS? Is there an example problem > I can look at? > > > > I do not know. I don't think we have an example. > > > > Thanks, > > > > Matt > > > > Thank you. > > > > Karthik. > > > > -- > > *Karthik Chockalingam, Ph.D.* > > Senior Research Software Engineer > > High Performance Systems Engineering Group > > Hartree Centre | Science and Technology Facilities Council > > karthikeyan.chockalingam at stfc.ac.uk > > > > [image: signature_3970890138] > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!dM_jKT_vX0NCVNcY9XOTTUaTSW-TF2CgOkB2UDeZ2ztYXapdmLlSBT4XhV7PknN3ewygGWDbKFEP9RAVjWAj$ > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!dM_jKT_vX0NCVNcY9XOTTUaTSW-TF2CgOkB2UDeZ2ztYXapdmLlSBT4XhV7PknN3ewygGWDbKFEP9RAVjWAj$ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 18270 bytes Desc: not available URL: From stefano.zampini at gmail.com Mon Sep 9 11:02:24 2024 From: stefano.zampini at gmail.com (Stefano Zampini) Date: Mon, 9 Sep 2024 19:02:24 +0300 Subject: [petsc-users] Hypre AMS usage In-Reply-To: References: Message-ID: I would say the best way is to look at the source code https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/hypre/hypre.c?ref_type=heads*L2061__;Iw!!G_uCfscf7eWS!b8Tv_XEoRSSehhvrYTELfhg0W08IjaDPGx3gkioYVRlIia4rZz7XLH7GXmXmEbathoNXMMfFaU8hWftak1IUQLf7BSgg7vg$ https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/hypre/hypre.c?ref_type=heads*L1239__;Iw!!G_uCfscf7eWS!b8Tv_XEoRSSehhvrYTELfhg0W08IjaDPGx3gkioYVRlIia4rZz7XLH7GXmXmEbathoNXMMfFaU8hWftak1IUQLf7I5s85AM$ Il giorno lun 9 set 2024 alle ore 18:50 Matthew Knepley ha scritto: > On Mon, Sep 9, 2024 at 10:17?AM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > >> Hi Matt, >> >> >> >> You mentioned it doesn?t hurt to set the smoothing flags >> >> >> >> >> https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L965__;Iw!!G_uCfscf7eWS!b8Tv_XEoRSSehhvrYTELfhg0W08IjaDPGx3gkioYVRlIia4rZz7XLH7GXmXmEbathoNXMMfFaU8hWftak1IUQLf7ymdPBOQ$ >> >> >> Do you know where I can look for the equivalent PETSc commands? Thank you. >> > > Yes, this is described here: > https://urldefense.us/v3/__https://petsc.org/main/manualpages/PC/PCHYPRE/__;!!G_uCfscf7eWS!b8Tv_XEoRSSehhvrYTELfhg0W08IjaDPGx3gkioYVRlIia4rZz7XLH7GXmXmEbathoNXMMfFaU8hWftak1IUQLf7oW09bmM$ > > The best way is to run with -help or look at the source. > > Thanks, > > Matt > > >> >> >> Kind regards, >> >> Karthik. >> >> >> >> *From: *Matthew Knepley >> *Date: *Friday, 6 September 2024 at 17:57 >> *To: *Chockalingam, Karthikeyan (STFC,DL,HC) < >> karthikeyan.chockalingam at stfc.ac.uk> >> *Cc: *petsc-users at mcs.anl.gov >> *Subject: *Re: [petsc-users] Hypre AMS usage >> >> On Fri, Sep 6, 2024 at 11:37?AM Karthikeyan Chockalingam - STFC UKRI via >> petsc-users wrote: >> >> Hello, >> >> >> >> I am trying to use the Hypre AMS preconditioner for the first time. >> >> >> >> I am following the example problem from Hypre >> >> >> https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L954__;Iw!!G_uCfscf7eWS!b8Tv_XEoRSSehhvrYTELfhg0W08IjaDPGx3gkioYVRlIia4rZz7XLH7GXmXmEbathoNXMMfFaU8hWftak1IUQLf73VDWhpc$ >> >> >> >> >> I have so far successfully set the discrete gradient operator and vertex >> co-ordinates, >> >> >> >> // Set discrete gradient >> >> petscErr = PCHYPRESetDiscreteGradient(pc, par_G); >> >> >> >> // Set vertex coordinates >> >> petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, >> par_zvec); >> >> >> >> Do I need to set the following smoothing options? >> >> >> https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L965__;Iw!!G_uCfscf7eWS!b8Tv_XEoRSSehhvrYTELfhg0W08IjaDPGx3gkioYVRlIia4rZz7XLH7GXmXmEbathoNXMMfFaU8hWftak1IUQLf7ymdPBOQ$ >> >> >> >> >> It cannot hurt. I would set them to begin with. >> >> >> >> Also, do I need to convert from MATMPIAIJ to CSR? >> >> >> https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L984__;Iw!!G_uCfscf7eWS!b8Tv_XEoRSSehhvrYTELfhg0W08IjaDPGx3gkioYVRlIia4rZz7XLH7GXmXmEbathoNXMMfFaU8hWftak1IUQLf7wD78NAo$ >> >> >> >> >> No. >> >> >> >> What are the other PETSc calls to invoke AMS? Is there an example problem >> I can look at? >> >> >> >> I do not know. I don't think we have an example. >> >> >> >> Thanks, >> >> >> >> Matt >> >> >> >> Thank you. >> >> >> >> Karthik. >> >> >> >> -- >> >> *Karthik Chockalingam, Ph.D.* >> >> Senior Research Software Engineer >> >> High Performance Systems Engineering Group >> >> Hartree Centre | Science and Technology Facilities Council >> >> karthikeyan.chockalingam at stfc.ac.uk >> >> >> >> [image: signature_3970890138] >> >> >> >> >> >> >> -- >> >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> >> >> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!b8Tv_XEoRSSehhvrYTELfhg0W08IjaDPGx3gkioYVRlIia4rZz7XLH7GXmXmEbathoNXMMfFaU8hWftak1IUQLf7uDER6bU$ >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!b8Tv_XEoRSSehhvrYTELfhg0W08IjaDPGx3gkioYVRlIia4rZz7XLH7GXmXmEbathoNXMMfFaU8hWftak1IUQLf7uDER6bU$ > > -- Stefano -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 18270 bytes Desc: not available URL: From karthikeyan.chockalingam at stfc.ac.uk Mon Sep 9 12:21:03 2024 From: karthikeyan.chockalingam at stfc.ac.uk (Karthikeyan Chockalingam - STFC UKRI) Date: Mon, 9 Sep 2024 17:21:03 +0000 Subject: [petsc-users] Hypre AMS usage In-Reply-To: References: Message-ID: Hi Stefano, Thank you. That was helpful. I tried the following: petscErr = PetscOptionsSetValue(NULL,"-ksp_type", "gmres"); petscErr = PetscOptionsSetValue(NULL,"-pc_type", "hypre"); petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_type", "ams"); // Set discrete gradient petscErr = PCHYPRESetDiscreteGradient(pc, par_G); // Set vertex coordinates petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, par_zvec); petscErr = PetscOptionsSetValue(NULL, "-ksp_view", NULL); petscErr = PetscOptionsSetValue(NULL, "-ksp_monitor_true_residual", NULL); petscErr = PetscOptionsSetValue(NULL, "-ksp_converged_reason", NULL); petscErr = PetscOptionsSetValue(NULL, "-options_left", NULL); petscErr = KSPSetFromOptions(ksp); It errored though I have set PCHYPRESetEdgeConstantVectors. But my program works, without PetscOptionsSetValue and by passing everything on the command line. [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: HYPRE AMS preconditioner needs either the coordinate vectors via PCSetCoordinates() or the edge constant vectors via PCHYPRESetEdgeConstantVectors() or the interpolation matrix via PCHYPRESetInterpolations() [0]PETS C ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc! [0]PETSC ERROR: Option left: name:-ksp_converged_reason (no value) source: code [0]PETSC ERROR: Option left: name:-options_left (no value) source: code [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!bo8NogztIWNrMacTFrpY794WDAB_YywFIYjTEhfjw2AbsXby70K4dOS2r9FsXF_bZ8HoYvAVh93LRjf-_2KHd9PMnR3kMMB6siqa$ for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.20.3, unknown [0]PETSC ERROR: ./example-dbg on a arch-moose named HC20210312 by karthikeyan.chockalingam Mon Sep 9 18:10:13 2024 [0]PETSC ERROR: Configure options --with-64-bit-indices --with-cxx-dialect=C++17 --with-debugging=no --with-fortran-bindings=0 --with-mpi=1 --with-openmp=1 --with-shared-libraries=1 --with-sowing=0 --download-fblaslapack=1 --download-hypre=1 --download-metis=1 --download-mumps=1 --download-ptscotch=1 --download-parmetis=1 --download-scalapack=1 --download-slepc=1 --download-strumpack=1 --download-superlu_dist=1 --download-bison=1 --download-hdf5=1 --with-hdf5-fortran-bindings=0 --download-hdf5-configure-arguments="--with-zlib" --with-make-np=4 [0]PETSC ERROR: #1 PCSetUp_HYPRE() at /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/pc/impls/hypre/hypre.c:295 [0]PETSC ERROR: #2 PCSetUp() at /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/pc/interface/precon.c:1080 [0]PETSC ERROR: #3 KSPSetUp() at /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/ksp/interface/itfunc.c:415 [0]PETSC ERROR: #4 KSPSolve_Private() at /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/ksp/interface/itfunc.c:833 [0]PETSC ERROR: #5 KSPSolve() at /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/ksp/interface/itfunc.c:1080 libMesh terminating: HYPRE AMS preconditioner needs either the coordinate vectors via PCSetCoordinates() or the edge constant vectors via PCHYPRESetEdgeConstantVectors() or the interpolation matrix via PCHYPRESetInterpolations() From: Stefano Zampini Date: Monday, 9 September 2024 at 17:02 To: Matthew Knepley Cc: Chockalingam, Karthikeyan (STFC,DL,HC) , petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Hypre AMS usage I would say the best way is to look at the source code https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/hypre/hypre.c?ref_type=heads*L2061__;Iw!!G_uCfscf7eWS!bo8NogztIWNrMacTFrpY794WDAB_YywFIYjTEhfjw2AbsXby70K4dOS2r9FsXF_bZ8HoYvAVh93LRjf-_2KHd9PMnR3kMHJAFTNc$ https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/hypre/hypre.c?ref_type=heads*L1239__;Iw!!G_uCfscf7eWS!bo8NogztIWNrMacTFrpY794WDAB_YywFIYjTEhfjw2AbsXby70K4dOS2r9FsXF_bZ8HoYvAVh93LRjf-_2KHd9PMnR3kMEACXByg$ Il giorno lun 9 set 2024 alle ore 18:50 Matthew Knepley > ha scritto: On Mon, Sep 9, 2024 at 10:17?AM Karthikeyan Chockalingam - STFC UKRI > wrote: Hi Matt, You mentioned it doesn?t hurt to set the smoothing flags https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L965__;Iw!!G_uCfscf7eWS!bo8NogztIWNrMacTFrpY794WDAB_YywFIYjTEhfjw2AbsXby70K4dOS2r9FsXF_bZ8HoYvAVh93LRjf-_2KHd9PMnR3kMDXhR5ES$ Do you know where I can look for the equivalent PETSc commands? Thank you. Yes, this is described here: https://urldefense.us/v3/__https://petsc.org/main/manualpages/PC/PCHYPRE/__;!!G_uCfscf7eWS!bo8NogztIWNrMacTFrpY794WDAB_YywFIYjTEhfjw2AbsXby70K4dOS2r9FsXF_bZ8HoYvAVh93LRjf-_2KHd9PMnR3kMON8QZMk$ The best way is to run with -help or look at the source. Thanks, Matt Kind regards, Karthik. From: Matthew Knepley > Date: Friday, 6 September 2024 at 17:57 To: Chockalingam, Karthikeyan (STFC,DL,HC) > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Hypre AMS usage On Fri, Sep 6, 2024 at 11:37?AM Karthikeyan Chockalingam - STFC UKRI via petsc-users > wrote: Hello, I am trying to use the Hypre AMS preconditioner for the first time. I am following the example problem from Hypre https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L954__;Iw!!G_uCfscf7eWS!bo8NogztIWNrMacTFrpY794WDAB_YywFIYjTEhfjw2AbsXby70K4dOS2r9FsXF_bZ8HoYvAVh93LRjf-_2KHd9PMnR3kMFdo2F6F$ I have so far successfully set the discrete gradient operator and vertex co-ordinates, // Set discrete gradient petscErr = PCHYPRESetDiscreteGradient(pc, par_G); // Set vertex coordinates petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, par_zvec); Do I need to set the following smoothing options? https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L965__;Iw!!G_uCfscf7eWS!bo8NogztIWNrMacTFrpY794WDAB_YywFIYjTEhfjw2AbsXby70K4dOS2r9FsXF_bZ8HoYvAVh93LRjf-_2KHd9PMnR3kMDXhR5ES$ It cannot hurt. I would set them to begin with. Also, do I need to convert from MATMPIAIJ to CSR? https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L984__;Iw!!G_uCfscf7eWS!bo8NogztIWNrMacTFrpY794WDAB_YywFIYjTEhfjw2AbsXby70K4dOS2r9FsXF_bZ8HoYvAVh93LRjf-_2KHd9PMnR3kMJU8hG7u$ No. What are the other PETSc calls to invoke AMS? Is there an example problem I can look at? I do not know. I don't think we have an example. Thanks, Matt Thank you. Karthik. -- Karthik Chockalingam, Ph.D. Senior Research Software Engineer High Performance Systems Engineering Group Hartree Centre | Science and Technology Facilities Council karthikeyan.chockalingam at stfc.ac.uk [signature_3970890138] -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bo8NogztIWNrMacTFrpY794WDAB_YywFIYjTEhfjw2AbsXby70K4dOS2r9FsXF_bZ8HoYvAVh93LRjf-_2KHd9PMnR3kME7kopmI$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bo8NogztIWNrMacTFrpY794WDAB_YywFIYjTEhfjw2AbsXby70K4dOS2r9FsXF_bZ8HoYvAVh93LRjf-_2KHd9PMnR3kME7kopmI$ -- Stefano -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 18270 bytes Desc: image001.png URL: From knepley at gmail.com Mon Sep 9 13:09:20 2024 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 9 Sep 2024 14:09:20 -0400 Subject: [petsc-users] Hypre AMS usage In-Reply-To: References: Message-ID: On Mon, Sep 9, 2024 at 1:21?PM Karthikeyan Chockalingam - STFC UKRI < karthikeyan.chockalingam at stfc.ac.uk> wrote: > Hi Stefano, > > > > Thank you. That was helpful. I tried the following: > > > > petscErr = PetscOptionsSetValue(NULL,"-ksp_type", "gmres"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_type", "hypre"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_type", "ams"); > > > > // Set discrete gradient > > petscErr = PCHYPRESetDiscreteGradient(pc, par_G); > > > > // Set vertex coordinates > > petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, > par_zvec); > > > > petscErr = PetscOptionsSetValue(NULL, "-ksp_view", NULL); > > petscErr = PetscOptionsSetValue(NULL, "-ksp_monitor_true_residual", > NULL); > > petscErr = PetscOptionsSetValue(NULL, "-ksp_converged_reason", NULL); > > petscErr = PetscOptionsSetValue(NULL, "-options_left", NULL); > > petscErr = KSPSetFromOptions(ksp); > > > > > > It errored though I have set PCHYPRESetEdgeConstantVectors. > My guess is that the "pc" was not yet Hypre, since that type was not set until KSPSetFromOptions() was called. So you need to call those two functions after that call. Thanks, Matt > But my program works, without PetscOptionsSetValue and by passing > everything on the command line. > > > > *[0]PETSC ERROR: --------------------- Error Message > --------------------------------------------------------------* > > [0]PETSC ERROR: HYPRE AMS preconditioner needs either the coordinate > vectors via PCSetCoordinates() or the edge constant vectors via > PCHYPRESetEdgeConstantVectors() or the interpolation matrix via > PCHYPRESetInterpolations() > > [0]PETS C ERROR: WARNING! There are unused option(s) set! Could be the > program crashed before usage or a spelling mistake, etc! > > [0]PETSC ERROR: Option left: name:-ksp_converged_reason (no value) > source: code > > [0]PETSC ERROR: Option left: name:-options_left (no value) source: code > > [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!fe84MxfHZLtDHZMk46RBJyizvPamDZ_ysHWfHCyD26Z_SGQv2oMDYu7yeAV_WtK5hoqAqUuepnGqGC-mIHaN$ for trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.20.3, unknown > > [0]PETSC ERROR: ./example-dbg on a arch-moose named HC20210312 by > karthikeyan.chockalingam Mon Sep 9 18:10:13 2024 > > [0]PETSC ERROR: Configure options --with-64-bit-indices > --with-cxx-dialect=C++17 --with-debugging=no --with-fortran-bindings=0 > --with-mpi=1 --with-openmp=1 --with-shared-libraries=1 --with-sowing=0 > --download-fblaslapack=1 --download-hypre=1 --download-metis=1 > --download-mumps=1 --download-ptscotch=1 --download-parmetis=1 > --download-scalapack=1 --download-slepc=1 --download-strumpack=1 > --download-superlu_dist=1 --download-bison=1 --download-hdf5=1 > --with-hdf5-fortran-bindings=0 > --download-hdf5-configure-arguments="--with-zlib" --with-make-np=4 > > [0]PETSC ERROR: #1 PCSetUp_HYPRE() at > /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/pc/impls/hypre/hypre.c:295 > > [0]PETSC ERROR: #2 PCSetUp() at > /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/pc/interface/precon.c:1080 > > [0]PETSC ERROR: #3 KSPSetUp() at > /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/ksp/interface/itfunc.c:415 > > [0]PETSC ERROR: #4 KSPSolve_Private() at > /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/ksp/interface/itfunc.c:833 > > [0]PETSC ERROR: #5 KSPSolve() at > /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/ksp/interface/itfunc.c:1080 > > libMesh terminating: > > HYPRE AMS preconditioner needs either the coordinate vectors via > PCSetCoordinates() or the edge constant vectors via > PCHYPRESetEdgeConstantVectors() or the interpolation matrix via > PCHYPRESetInterpolations() > > > > > > *From: *Stefano Zampini > *Date: *Monday, 9 September 2024 at 17:02 > *To: *Matthew Knepley > *Cc: *Chockalingam, Karthikeyan (STFC,DL,HC) < > karthikeyan.chockalingam at stfc.ac.uk>, petsc-users at mcs.anl.gov < > petsc-users at mcs.anl.gov> > *Subject: *Re: [petsc-users] Hypre AMS usage > > I would say the best way is to look at the source code > > > > > https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/hypre/hypre.c?ref_type=heads*L2061__;Iw!!G_uCfscf7eWS!fe84MxfHZLtDHZMk46RBJyizvPamDZ_ysHWfHCyD26Z_SGQv2oMDYu7yeAV_WtK5hoqAqUuepnGqGGrHyHo_$ > > > https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/hypre/hypre.c?ref_type=heads*L1239__;Iw!!G_uCfscf7eWS!fe84MxfHZLtDHZMk46RBJyizvPamDZ_ysHWfHCyD26Z_SGQv2oMDYu7yeAV_WtK5hoqAqUuepnGqGJfzHU4z$ > > > > Il giorno lun 9 set 2024 alle ore 18:50 Matthew Knepley > ha scritto: > > On Mon, Sep 9, 2024 at 10:17?AM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > Hi Matt, > > > > You mentioned it doesn?t hurt to set the smoothing flags > > > > > https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L965__;Iw!!G_uCfscf7eWS!fe84MxfHZLtDHZMk46RBJyizvPamDZ_ysHWfHCyD26Z_SGQv2oMDYu7yeAV_WtK5hoqAqUuepnGqGNAEo0Pf$ > > > Do you know where I can look for the equivalent PETSc commands? Thank you. > > > > Yes, this is described here: > https://urldefense.us/v3/__https://petsc.org/main/manualpages/PC/PCHYPRE/__;!!G_uCfscf7eWS!fe84MxfHZLtDHZMk46RBJyizvPamDZ_ysHWfHCyD26Z_SGQv2oMDYu7yeAV_WtK5hoqAqUuepnGqGMxyA9Jy$ > > > The best way is to run with -help or look at the source. > > > > Thanks, > > > > Matt > > > > > > Kind regards, > > Karthik. > > > > *From: *Matthew Knepley > *Date: *Friday, 6 September 2024 at 17:57 > *To: *Chockalingam, Karthikeyan (STFC,DL,HC) < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *petsc-users at mcs.anl.gov > *Subject: *Re: [petsc-users] Hypre AMS usage > > On Fri, Sep 6, 2024 at 11:37?AM Karthikeyan Chockalingam - STFC UKRI via > petsc-users wrote: > > Hello, > > > > I am trying to use the Hypre AMS preconditioner for the first time. > > > > I am following the example problem from Hypre > > > https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L954__;Iw!!G_uCfscf7eWS!fe84MxfHZLtDHZMk46RBJyizvPamDZ_ysHWfHCyD26Z_SGQv2oMDYu7yeAV_WtK5hoqAqUuepnGqGAi69yc_$ > > > > > I have so far successfully set the discrete gradient operator and vertex > co-ordinates, > > > > // Set discrete gradient > > petscErr = PCHYPRESetDiscreteGradient(pc, par_G); > > > > // Set vertex coordinates > > petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, > par_zvec); > > > > Do I need to set the following smoothing options? > > > https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L965__;Iw!!G_uCfscf7eWS!fe84MxfHZLtDHZMk46RBJyizvPamDZ_ysHWfHCyD26Z_SGQv2oMDYu7yeAV_WtK5hoqAqUuepnGqGNAEo0Pf$ > > > > > It cannot hurt. I would set them to begin with. > > > > Also, do I need to convert from MATMPIAIJ to CSR? > > > https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L984__;Iw!!G_uCfscf7eWS!fe84MxfHZLtDHZMk46RBJyizvPamDZ_ysHWfHCyD26Z_SGQv2oMDYu7yeAV_WtK5hoqAqUuepnGqGC8cELGt$ > > > > > No. > > > > What are the other PETSc calls to invoke AMS? Is there an example problem > I can look at? > > > > I do not know. I don't think we have an example. > > > > Thanks, > > > > Matt > > > > Thank you. > > > > Karthik. > > > > -- > > *Karthik Chockalingam, Ph.D.* > > Senior Research Software Engineer > > High Performance Systems Engineering Group > > Hartree Centre | Science and Technology Facilities Council > > karthikeyan.chockalingam at stfc.ac.uk > > > > [image: signature_3970890138] > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fe84MxfHZLtDHZMk46RBJyizvPamDZ_ysHWfHCyD26Z_SGQv2oMDYu7yeAV_WtK5hoqAqUuepnGqGD2aXdkU$ > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fe84MxfHZLtDHZMk46RBJyizvPamDZ_ysHWfHCyD26Z_SGQv2oMDYu7yeAV_WtK5hoqAqUuepnGqGD2aXdkU$ > > > > > > -- > > Stefano > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fe84MxfHZLtDHZMk46RBJyizvPamDZ_ysHWfHCyD26Z_SGQv2oMDYu7yeAV_WtK5hoqAqUuepnGqGD2aXdkU$ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 18270 bytes Desc: not available URL: From karthikeyan.chockalingam at stfc.ac.uk Mon Sep 9 13:18:41 2024 From: karthikeyan.chockalingam at stfc.ac.uk (Karthikeyan Chockalingam - STFC UKRI) Date: Mon, 9 Sep 2024 18:18:41 +0000 Subject: [petsc-users] Hypre AMS usage In-Reply-To: References: Message-ID: Calling the two functions after KSPSetFromOptions() did not work either. Everything works from the command line. I haven?t set KSPSetOperators, not sure if that is an issue. From: Matthew Knepley Date: Monday, 9 September 2024 at 19:09 To: Chockalingam, Karthikeyan (STFC,DL,HC) Cc: Stefano Zampini , petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Hypre AMS usage On Mon, Sep 9, 2024 at 1:21?PM Karthikeyan Chockalingam - STFC UKRI > wrote: Hi Stefano, Thank you. That was helpful. I tried the following: petscErr = PetscOptionsSetValue(NULL,"-ksp_type", "gmres"); petscErr = PetscOptionsSetValue(NULL,"-pc_type", "hypre"); petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_type", "ams"); // Set discrete gradient petscErr = PCHYPRESetDiscreteGradient(pc, par_G); // Set vertex coordinates petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, par_zvec); petscErr = PetscOptionsSetValue(NULL, "-ksp_view", NULL); petscErr = PetscOptionsSetValue(NULL, "-ksp_monitor_true_residual", NULL); petscErr = PetscOptionsSetValue(NULL, "-ksp_converged_reason", NULL); petscErr = PetscOptionsSetValue(NULL, "-options_left", NULL); petscErr = KSPSetFromOptions(ksp); It errored though I have set PCHYPRESetEdgeConstantVectors. My guess is that the "pc" was not yet Hypre, since that type was not set until KSPSetFromOptions() was called. So you need to call those two functions after that call. Thanks, Matt But my program works, without PetscOptionsSetValue and by passing everything on the command line. [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: HYPRE AMS preconditioner needs either the coordinate vectors via PCSetCoordinates() or the edge constant vectors via PCHYPRESetEdgeConstantVectors() or the interpolation matrix via PCHYPRESetInterpolations() [0]PETS C ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc! [0]PETSC ERROR: Option left: name:-ksp_converged_reason (no value) source: code [0]PETSC ERROR: Option left: name:-options_left (no value) source: code [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!b7VHITLFZTpZrADfLB56lWlJTqmwC4JEvZD6echgXfbJQKxpgD25nP6-4KMrqCW_xhtwwYx9uw7yr4sBhLC-uO57fkdRpJfKGDRA$ for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.20.3, unknown [0]PETSC ERROR: ./example-dbg on a arch-moose named HC20210312 by karthikeyan.chockalingam Mon Sep 9 18:10:13 2024 [0]PETSC ERROR: Configure options --with-64-bit-indices --with-cxx-dialect=C++17 --with-debugging=no --with-fortran-bindings=0 --with-mpi=1 --with-openmp=1 --with-shared-libraries=1 --with-sowing=0 --download-fblaslapack=1 --download-hypre=1 --download-metis=1 --download-mumps=1 --download-ptscotch=1 --download-parmetis=1 --download-scalapack=1 --download-slepc=1 --download-strumpack=1 --download-superlu_dist=1 --download-bison=1 --download-hdf5=1 --with-hdf5-fortran-bindings=0 --download-hdf5-configure-arguments="--with-zlib" --with-make-np=4 [0]PETSC ERROR: #1 PCSetUp_HYPRE() at /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/pc/impls/hypre/hypre.c:295 [0]PETSC ERROR: #2 PCSetUp() at /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/pc/interface/precon.c:1080 [0]PETSC ERROR: #3 KSPSetUp() at /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/ksp/interface/itfunc.c:415 [0]PETSC ERROR: #4 KSPSolve_Private() at /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/ksp/interface/itfunc.c:833 [0]PETSC ERROR: #5 KSPSolve() at /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/ksp/interface/itfunc.c:1080 libMesh terminating: HYPRE AMS preconditioner needs either the coordinate vectors via PCSetCoordinates() or the edge constant vectors via PCHYPRESetEdgeConstantVectors() or the interpolation matrix via PCHYPRESetInterpolations() From: Stefano Zampini > Date: Monday, 9 September 2024 at 17:02 To: Matthew Knepley > Cc: Chockalingam, Karthikeyan (STFC,DL,HC) >, petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Hypre AMS usage I would say the best way is to look at the source code https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/hypre/hypre.c?ref_type=heads*L2061__;Iw!!G_uCfscf7eWS!b7VHITLFZTpZrADfLB56lWlJTqmwC4JEvZD6echgXfbJQKxpgD25nP6-4KMrqCW_xhtwwYx9uw7yr4sBhLC-uO57fkdRpBGZO6s9$ https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/hypre/hypre.c?ref_type=heads*L1239__;Iw!!G_uCfscf7eWS!b7VHITLFZTpZrADfLB56lWlJTqmwC4JEvZD6echgXfbJQKxpgD25nP6-4KMrqCW_xhtwwYx9uw7yr4sBhLC-uO57fkdRpLAttthS$ Il giorno lun 9 set 2024 alle ore 18:50 Matthew Knepley > ha scritto: On Mon, Sep 9, 2024 at 10:17?AM Karthikeyan Chockalingam - STFC UKRI > wrote: Hi Matt, You mentioned it doesn?t hurt to set the smoothing flags https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L965__;Iw!!G_uCfscf7eWS!b7VHITLFZTpZrADfLB56lWlJTqmwC4JEvZD6echgXfbJQKxpgD25nP6-4KMrqCW_xhtwwYx9uw7yr4sBhLC-uO57fkdRpPn4JAUM$ Do you know where I can look for the equivalent PETSc commands? Thank you. Yes, this is described here: https://urldefense.us/v3/__https://petsc.org/main/manualpages/PC/PCHYPRE/__;!!G_uCfscf7eWS!b7VHITLFZTpZrADfLB56lWlJTqmwC4JEvZD6echgXfbJQKxpgD25nP6-4KMrqCW_xhtwwYx9uw7yr4sBhLC-uO57fkdRpArP3DS6$ The best way is to run with -help or look at the source. Thanks, Matt Kind regards, Karthik. From: Matthew Knepley > Date: Friday, 6 September 2024 at 17:57 To: Chockalingam, Karthikeyan (STFC,DL,HC) > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Hypre AMS usage On Fri, Sep 6, 2024 at 11:37?AM Karthikeyan Chockalingam - STFC UKRI via petsc-users > wrote: Hello, I am trying to use the Hypre AMS preconditioner for the first time. I am following the example problem from Hypre https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L954__;Iw!!G_uCfscf7eWS!b7VHITLFZTpZrADfLB56lWlJTqmwC4JEvZD6echgXfbJQKxpgD25nP6-4KMrqCW_xhtwwYx9uw7yr4sBhLC-uO57fkdRpMeHg23E$ I have so far successfully set the discrete gradient operator and vertex co-ordinates, // Set discrete gradient petscErr = PCHYPRESetDiscreteGradient(pc, par_G); // Set vertex coordinates petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, par_zvec); Do I need to set the following smoothing options? https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L965__;Iw!!G_uCfscf7eWS!b7VHITLFZTpZrADfLB56lWlJTqmwC4JEvZD6echgXfbJQKxpgD25nP6-4KMrqCW_xhtwwYx9uw7yr4sBhLC-uO57fkdRpPn4JAUM$ It cannot hurt. I would set them to begin with. Also, do I need to convert from MATMPIAIJ to CSR? https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L984__;Iw!!G_uCfscf7eWS!b7VHITLFZTpZrADfLB56lWlJTqmwC4JEvZD6echgXfbJQKxpgD25nP6-4KMrqCW_xhtwwYx9uw7yr4sBhLC-uO57fkdRpH-VQniQ$ No. What are the other PETSc calls to invoke AMS? Is there an example problem I can look at? I do not know. I don't think we have an example. Thanks, Matt Thank you. Karthik. -- Karthik Chockalingam, Ph.D. Senior Research Software Engineer High Performance Systems Engineering Group Hartree Centre | Science and Technology Facilities Council karthikeyan.chockalingam at stfc.ac.uk [signature_3970890138] -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!b7VHITLFZTpZrADfLB56lWlJTqmwC4JEvZD6echgXfbJQKxpgD25nP6-4KMrqCW_xhtwwYx9uw7yr4sBhLC-uO57fkdRpIxqMlea$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!b7VHITLFZTpZrADfLB56lWlJTqmwC4JEvZD6echgXfbJQKxpgD25nP6-4KMrqCW_xhtwwYx9uw7yr4sBhLC-uO57fkdRpIxqMlea$ -- Stefano -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!b7VHITLFZTpZrADfLB56lWlJTqmwC4JEvZD6echgXfbJQKxpgD25nP6-4KMrqCW_xhtwwYx9uw7yr4sBhLC-uO57fkdRpIxqMlea$ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 18270 bytes Desc: image001.png URL: From knepley at gmail.com Mon Sep 9 13:46:40 2024 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 9 Sep 2024 14:46:40 -0400 Subject: [petsc-users] Hypre AMS usage In-Reply-To: References: Message-ID: On Mon, Sep 9, 2024 at 2:18?PM Karthikeyan Chockalingam - STFC UKRI < karthikeyan.chockalingam at stfc.ac.uk> wrote: > Calling the two functions after KSPSetFromOptions() did not work either. > Can you check that the PC has the correct type when you call it? Thanks, Matt > Everything works from the command line. > > I haven?t set KSPSetOperators, not sure if that is an issue. > > > > *From: *Matthew Knepley > *Date: *Monday, 9 September 2024 at 19:09 > *To: *Chockalingam, Karthikeyan (STFC,DL,HC) < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *Stefano Zampini , petsc-users at mcs.anl.gov > > *Subject: *Re: [petsc-users] Hypre AMS usage > > On Mon, Sep 9, 2024 at 1:21?PM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > Hi Stefano, > > > > Thank you. That was helpful. I tried the following: > > > > petscErr = PetscOptionsSetValue(NULL,"-ksp_type", "gmres"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_type", "hypre"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_type", "ams"); > > > > // Set discrete gradient > > petscErr = PCHYPRESetDiscreteGradient(pc, par_G); > > > > // Set vertex coordinates > > petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, > par_zvec); > > > > petscErr = PetscOptionsSetValue(NULL, "-ksp_view", NULL); > > petscErr = PetscOptionsSetValue(NULL, "-ksp_monitor_true_residual", > NULL); > > petscErr = PetscOptionsSetValue(NULL, "-ksp_converged_reason", NULL); > > petscErr = PetscOptionsSetValue(NULL, "-options_left", NULL); > > petscErr = KSPSetFromOptions(ksp); > > > > > > It errored though I have set PCHYPRESetEdgeConstantVectors. > > > > My guess is that the "pc" was not yet Hypre, since that type was not set > until KSPSetFromOptions() was called. So you need > > to call those two functions after that call. > > > > Thanks, > > > > Matt > > > > But my program works, without PetscOptionsSetValue and by passing > everything on the command line. > > > > *[0]PETSC ERROR: --------------------- Error Message > --------------------------------------------------------------* > > [0]PETSC ERROR: HYPRE AMS preconditioner needs either the coordinate > vectors via PCSetCoordinates() or the edge constant vectors via > PCHYPRESetEdgeConstantVectors() or the interpolation matrix via > PCHYPRESetInterpolations() > > [0]PETS C ERROR: WARNING! There are unused option(s) set! Could be the > program crashed before usage or a spelling mistake, etc! > > [0]PETSC ERROR: Option left: name:-ksp_converged_reason (no value) > source: code > > [0]PETSC ERROR: Option left: name:-options_left (no value) source: code > > [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!d7Ibtwj8EdetyxS7Mw_puBNyeptxAZzQX6w-kad_7Pk9sQ8i2I_lWTXeRzcQ081aNcHWfekFUU1zMGa7WR82$ for trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.20.3, unknown > > [0]PETSC ERROR: ./example-dbg on a arch-moose named HC20210312 by > karthikeyan.chockalingam Mon Sep 9 18:10:13 2024 > > [0]PETSC ERROR: Configure options --with-64-bit-indices > --with-cxx-dialect=C++17 --with-debugging=no --with-fortran-bindings=0 > --with-mpi=1 --with-openmp=1 --with-shared-libraries=1 --with-sowing=0 > --download-fblaslapack=1 --download-hypre=1 --download-metis=1 > --download-mumps=1 --download-ptscotch=1 --download-parmetis=1 > --download-scalapack=1 --download-slepc=1 --download-strumpack=1 > --download-superlu_dist=1 --download-bison=1 --download-hdf5=1 > --with-hdf5-fortran-bindings=0 > --download-hdf5-configure-arguments="--with-zlib" --with-make-np=4 > > [0]PETSC ERROR: #1 PCSetUp_HYPRE() at > /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/pc/impls/hypre/hypre.c:295 > > [0]PETSC ERROR: #2 PCSetUp() at > /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/pc/interface/precon.c:1080 > > [0]PETSC ERROR: #3 KSPSetUp() at > /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/ksp/interface/itfunc.c:415 > > [0]PETSC ERROR: #4 KSPSolve_Private() at > /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/ksp/interface/itfunc.c:833 > > [0]PETSC ERROR: #5 KSPSolve() at > /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/ksp/interface/itfunc.c:1080 > > libMesh terminating: > > HYPRE AMS preconditioner needs either the coordinate vectors via > PCSetCoordinates() or the edge constant vectors via > PCHYPRESetEdgeConstantVectors() or the interpolation matrix via > PCHYPRESetInterpolations() > > > > > > *From: *Stefano Zampini > *Date: *Monday, 9 September 2024 at 17:02 > *To: *Matthew Knepley > *Cc: *Chockalingam, Karthikeyan (STFC,DL,HC) < > karthikeyan.chockalingam at stfc.ac.uk>, petsc-users at mcs.anl.gov < > petsc-users at mcs.anl.gov> > *Subject: *Re: [petsc-users] Hypre AMS usage > > I would say the best way is to look at the source code > > > > > https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/hypre/hypre.c?ref_type=heads*L2061__;Iw!!G_uCfscf7eWS!d7Ibtwj8EdetyxS7Mw_puBNyeptxAZzQX6w-kad_7Pk9sQ8i2I_lWTXeRzcQ081aNcHWfekFUU1zMMAuny2o$ > > > https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/hypre/hypre.c?ref_type=heads*L1239__;Iw!!G_uCfscf7eWS!d7Ibtwj8EdetyxS7Mw_puBNyeptxAZzQX6w-kad_7Pk9sQ8i2I_lWTXeRzcQ081aNcHWfekFUU1zMMlytYdT$ > > > > Il giorno lun 9 set 2024 alle ore 18:50 Matthew Knepley > ha scritto: > > On Mon, Sep 9, 2024 at 10:17?AM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > Hi Matt, > > > > You mentioned it doesn?t hurt to set the smoothing flags > > > > > https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L965__;Iw!!G_uCfscf7eWS!d7Ibtwj8EdetyxS7Mw_puBNyeptxAZzQX6w-kad_7Pk9sQ8i2I_lWTXeRzcQ081aNcHWfekFUU1zMPPzNSlj$ > > > Do you know where I can look for the equivalent PETSc commands? Thank you. > > > > Yes, this is described here: > https://urldefense.us/v3/__https://petsc.org/main/manualpages/PC/PCHYPRE/__;!!G_uCfscf7eWS!d7Ibtwj8EdetyxS7Mw_puBNyeptxAZzQX6w-kad_7Pk9sQ8i2I_lWTXeRzcQ081aNcHWfekFUU1zMJsj8yBM$ > > > The best way is to run with -help or look at the source. > > > > Thanks, > > > > Matt > > > > > > Kind regards, > > Karthik. > > > > *From: *Matthew Knepley > *Date: *Friday, 6 September 2024 at 17:57 > *To: *Chockalingam, Karthikeyan (STFC,DL,HC) < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *petsc-users at mcs.anl.gov > *Subject: *Re: [petsc-users] Hypre AMS usage > > On Fri, Sep 6, 2024 at 11:37?AM Karthikeyan Chockalingam - STFC UKRI via > petsc-users wrote: > > Hello, > > > > I am trying to use the Hypre AMS preconditioner for the first time. > > > > I am following the example problem from Hypre > > > https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L954__;Iw!!G_uCfscf7eWS!d7Ibtwj8EdetyxS7Mw_puBNyeptxAZzQX6w-kad_7Pk9sQ8i2I_lWTXeRzcQ081aNcHWfekFUU1zMGGa5X3f$ > > > > > I have so far successfully set the discrete gradient operator and vertex > co-ordinates, > > > > // Set discrete gradient > > petscErr = PCHYPRESetDiscreteGradient(pc, par_G); > > > > // Set vertex coordinates > > petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, > par_zvec); > > > > Do I need to set the following smoothing options? > > > https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L965__;Iw!!G_uCfscf7eWS!d7Ibtwj8EdetyxS7Mw_puBNyeptxAZzQX6w-kad_7Pk9sQ8i2I_lWTXeRzcQ081aNcHWfekFUU1zMPPzNSlj$ > > > > > It cannot hurt. I would set them to begin with. > > > > Also, do I need to convert from MATMPIAIJ to CSR? > > > https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L984__;Iw!!G_uCfscf7eWS!d7Ibtwj8EdetyxS7Mw_puBNyeptxAZzQX6w-kad_7Pk9sQ8i2I_lWTXeRzcQ081aNcHWfekFUU1zMEiNqHdR$ > > > > > No. > > > > What are the other PETSc calls to invoke AMS? Is there an example problem > I can look at? > > > > I do not know. I don't think we have an example. > > > > Thanks, > > > > Matt > > > > Thank you. > > > > Karthik. > > > > -- > > *Karthik Chockalingam, Ph.D.* > > Senior Research Software Engineer > > High Performance Systems Engineering Group > > Hartree Centre | Science and Technology Facilities Council > > karthikeyan.chockalingam at stfc.ac.uk > > > > [image: signature_3970890138] > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!d7Ibtwj8EdetyxS7Mw_puBNyeptxAZzQX6w-kad_7Pk9sQ8i2I_lWTXeRzcQ081aNcHWfekFUU1zMGjOGXd1$ > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!d7Ibtwj8EdetyxS7Mw_puBNyeptxAZzQX6w-kad_7Pk9sQ8i2I_lWTXeRzcQ081aNcHWfekFUU1zMGjOGXd1$ > > > > > > -- > > Stefano > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!d7Ibtwj8EdetyxS7Mw_puBNyeptxAZzQX6w-kad_7Pk9sQ8i2I_lWTXeRzcQ081aNcHWfekFUU1zMGjOGXd1$ > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!d7Ibtwj8EdetyxS7Mw_puBNyeptxAZzQX6w-kad_7Pk9sQ8i2I_lWTXeRzcQ081aNcHWfekFUU1zMGjOGXd1$ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 18270 bytes Desc: not available URL: From karthikeyan.chockalingam at stfc.ac.uk Mon Sep 9 14:37:57 2024 From: karthikeyan.chockalingam at stfc.ac.uk (Karthikeyan Chockalingam - STFC UKRI) Date: Mon, 9 Sep 2024 19:37:57 +0000 Subject: [petsc-users] Hypre AMS usage In-Reply-To: References: Message-ID: I didn?t know how to check the pc type but adding petscErr = PCSetType(pc, "hypre"); before the two functions made it to work. But how does it work from the command line? For summary: KSPCreate(mesh.comm().get(), &ksp); petscErr = PetscOptionsSetValue(NULL,"-ksp_type", "gmres"); petscErr = PetscOptionsSetValue(NULL,"-pc_type", "hypre"); petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_type", "ams"); petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_type", "2"); petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_times", "1"); petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_weigh", "1.0"); petscErr = PetscOptionsSetValue(NULL, "-pc_hypre_ams_omega", "1.0"); petscErr = PetscOptionsSetValue(NULL, "-ksp_view", NULL); petscErr = PetscOptionsSetValue(NULL, "-ksp_monitor_true_residual", NULL); petscErr = PetscOptionsSetValue(NULL, "-ksp_converged_reason", NULL); petscErr = PetscOptionsSetValue(NULL, "-options_left", NULL); petscErr = KSPSetFromOptions(ksp); // Set pc type petscErr = PCSetType(pc, "hypre"); // Set discrete gradient petscErr = PCHYPRESetDiscreteGradient(pc, par_G); // Set vertex coordinates petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, par_zvec); From: Matthew Knepley Date: Monday, 9 September 2024 at 19:46 To: Chockalingam, Karthikeyan (STFC,DL,HC) Cc: Stefano Zampini , petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Hypre AMS usage On Mon, Sep 9, 2024 at 2:18?PM Karthikeyan Chockalingam - STFC UKRI > wrote: Calling the two functions after KSPSetFromOptions() did not work either. Can you check that the PC has the correct type when you call it? Thanks, Matt Everything works from the command line. I haven?t set KSPSetOperators, not sure if that is an issue. From: Matthew Knepley > Date: Monday, 9 September 2024 at 19:09 To: Chockalingam, Karthikeyan (STFC,DL,HC) > Cc: Stefano Zampini >, petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Hypre AMS usage On Mon, Sep 9, 2024 at 1:21?PM Karthikeyan Chockalingam - STFC UKRI > wrote: Hi Stefano, Thank you. That was helpful. I tried the following: petscErr = PetscOptionsSetValue(NULL,"-ksp_type", "gmres"); petscErr = PetscOptionsSetValue(NULL,"-pc_type", "hypre"); petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_type", "ams"); // Set discrete gradient petscErr = PCHYPRESetDiscreteGradient(pc, par_G); // Set vertex coordinates petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, par_zvec); petscErr = PetscOptionsSetValue(NULL, "-ksp_view", NULL); petscErr = PetscOptionsSetValue(NULL, "-ksp_monitor_true_residual", NULL); petscErr = PetscOptionsSetValue(NULL, "-ksp_converged_reason", NULL); petscErr = PetscOptionsSetValue(NULL, "-options_left", NULL); petscErr = KSPSetFromOptions(ksp); It errored though I have set PCHYPRESetEdgeConstantVectors. My guess is that the "pc" was not yet Hypre, since that type was not set until KSPSetFromOptions() was called. So you need to call those two functions after that call. Thanks, Matt But my program works, without PetscOptionsSetValue and by passing everything on the command line. [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: HYPRE AMS preconditioner needs either the coordinate vectors via PCSetCoordinates() or the edge constant vectors via PCHYPRESetEdgeConstantVectors() or the interpolation matrix via PCHYPRESetInterpolations() [0]PETS C ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc! [0]PETSC ERROR: Option left: name:-ksp_converged_reason (no value) source: code [0]PETSC ERROR: Option left: name:-options_left (no value) source: code [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!eYw4lnpu-BbY5WH5sSyH48IiYwLf-FRf0zDii3jJcM2wrJoySnt8upfoeNYLEZPjdr0wBxEDC5QyekoplwHE0AX2XiYI9His52-f$ for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.20.3, unknown [0]PETSC ERROR: ./example-dbg on a arch-moose named HC20210312 by karthikeyan.chockalingam Mon Sep 9 18:10:13 2024 [0]PETSC ERROR: Configure options --with-64-bit-indices --with-cxx-dialect=C++17 --with-debugging=no --with-fortran-bindings=0 --with-mpi=1 --with-openmp=1 --with-shared-libraries=1 --with-sowing=0 --download-fblaslapack=1 --download-hypre=1 --download-metis=1 --download-mumps=1 --download-ptscotch=1 --download-parmetis=1 --download-scalapack=1 --download-slepc=1 --download-strumpack=1 --download-superlu_dist=1 --download-bison=1 --download-hdf5=1 --with-hdf5-fortran-bindings=0 --download-hdf5-configure-arguments="--with-zlib" --with-make-np=4 [0]PETSC ERROR: #1 PCSetUp_HYPRE() at /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/pc/impls/hypre/hypre.c:295 [0]PETSC ERROR: #2 PCSetUp() at /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/pc/interface/precon.c:1080 [0]PETSC ERROR: #3 KSPSetUp() at /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/ksp/interface/itfunc.c:415 [0]PETSC ERROR: #4 KSPSolve_Private() at /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/ksp/interface/itfunc.c:833 [0]PETSC ERROR: #5 KSPSolve() at /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/ksp/interface/itfunc.c:1080 libMesh terminating: HYPRE AMS preconditioner needs either the coordinate vectors via PCSetCoordinates() or the edge constant vectors via PCHYPRESetEdgeConstantVectors() or the interpolation matrix via PCHYPRESetInterpolations() From: Stefano Zampini > Date: Monday, 9 September 2024 at 17:02 To: Matthew Knepley > Cc: Chockalingam, Karthikeyan (STFC,DL,HC) >, petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Hypre AMS usage I would say the best way is to look at the source code https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/hypre/hypre.c?ref_type=heads*L2061__;Iw!!G_uCfscf7eWS!eYw4lnpu-BbY5WH5sSyH48IiYwLf-FRf0zDii3jJcM2wrJoySnt8upfoeNYLEZPjdr0wBxEDC5QyekoplwHE0AX2XiYI9NmuR5ZD$ https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/hypre/hypre.c?ref_type=heads*L1239__;Iw!!G_uCfscf7eWS!eYw4lnpu-BbY5WH5sSyH48IiYwLf-FRf0zDii3jJcM2wrJoySnt8upfoeNYLEZPjdr0wBxEDC5QyekoplwHE0AX2XiYI9A9pZjHy$ Il giorno lun 9 set 2024 alle ore 18:50 Matthew Knepley > ha scritto: On Mon, Sep 9, 2024 at 10:17?AM Karthikeyan Chockalingam - STFC UKRI > wrote: Hi Matt, You mentioned it doesn?t hurt to set the smoothing flags https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L965__;Iw!!G_uCfscf7eWS!eYw4lnpu-BbY5WH5sSyH48IiYwLf-FRf0zDii3jJcM2wrJoySnt8upfoeNYLEZPjdr0wBxEDC5QyekoplwHE0AX2XiYI9Me7mhsS$ Do you know where I can look for the equivalent PETSc commands? Thank you. Yes, this is described here: https://urldefense.us/v3/__https://petsc.org/main/manualpages/PC/PCHYPRE/__;!!G_uCfscf7eWS!eYw4lnpu-BbY5WH5sSyH48IiYwLf-FRf0zDii3jJcM2wrJoySnt8upfoeNYLEZPjdr0wBxEDC5QyekoplwHE0AX2XiYI9Dq956S-$ The best way is to run with -help or look at the source. Thanks, Matt Kind regards, Karthik. From: Matthew Knepley > Date: Friday, 6 September 2024 at 17:57 To: Chockalingam, Karthikeyan (STFC,DL,HC) > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Hypre AMS usage On Fri, Sep 6, 2024 at 11:37?AM Karthikeyan Chockalingam - STFC UKRI via petsc-users > wrote: Hello, I am trying to use the Hypre AMS preconditioner for the first time. I am following the example problem from Hypre https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L954__;Iw!!G_uCfscf7eWS!eYw4lnpu-BbY5WH5sSyH48IiYwLf-FRf0zDii3jJcM2wrJoySnt8upfoeNYLEZPjdr0wBxEDC5QyekoplwHE0AX2XiYI9HilXNIt$ I have so far successfully set the discrete gradient operator and vertex co-ordinates, // Set discrete gradient petscErr = PCHYPRESetDiscreteGradient(pc, par_G); // Set vertex coordinates petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, par_zvec); Do I need to set the following smoothing options? https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L965__;Iw!!G_uCfscf7eWS!eYw4lnpu-BbY5WH5sSyH48IiYwLf-FRf0zDii3jJcM2wrJoySnt8upfoeNYLEZPjdr0wBxEDC5QyekoplwHE0AX2XiYI9Me7mhsS$ It cannot hurt. I would set them to begin with. Also, do I need to convert from MATMPIAIJ to CSR? https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L984__;Iw!!G_uCfscf7eWS!eYw4lnpu-BbY5WH5sSyH48IiYwLf-FRf0zDii3jJcM2wrJoySnt8upfoeNYLEZPjdr0wBxEDC5QyekoplwHE0AX2XiYI9Ew9m3by$ No. What are the other PETSc calls to invoke AMS? Is there an example problem I can look at? I do not know. I don't think we have an example. Thanks, Matt Thank you. Karthik. -- Karthik Chockalingam, Ph.D. Senior Research Software Engineer High Performance Systems Engineering Group Hartree Centre | Science and Technology Facilities Council karthikeyan.chockalingam at stfc.ac.uk [signature_3970890138] -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!eYw4lnpu-BbY5WH5sSyH48IiYwLf-FRf0zDii3jJcM2wrJoySnt8upfoeNYLEZPjdr0wBxEDC5QyekoplwHE0AX2XiYI9PETlFak$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!eYw4lnpu-BbY5WH5sSyH48IiYwLf-FRf0zDii3jJcM2wrJoySnt8upfoeNYLEZPjdr0wBxEDC5QyekoplwHE0AX2XiYI9PETlFak$ -- Stefano -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!eYw4lnpu-BbY5WH5sSyH48IiYwLf-FRf0zDii3jJcM2wrJoySnt8upfoeNYLEZPjdr0wBxEDC5QyekoplwHE0AX2XiYI9PETlFak$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!eYw4lnpu-BbY5WH5sSyH48IiYwLf-FRf0zDii3jJcM2wrJoySnt8upfoeNYLEZPjdr0wBxEDC5QyekoplwHE0AX2XiYI9PETlFak$ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 18270 bytes Desc: image001.png URL: From knepley at gmail.com Mon Sep 9 14:56:56 2024 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 9 Sep 2024 15:56:56 -0400 Subject: [petsc-users] Hypre AMS usage In-Reply-To: References: Message-ID: On Mon, Sep 9, 2024 at 3:38?PM Karthikeyan Chockalingam - STFC UKRI < karthikeyan.chockalingam at stfc.ac.uk> wrote: > I didn?t know how to check the pc type but adding > > petscErr = PCSetType(pc, "hypre"); > > before the two functions made it to work. > > > > But how does it work from the command line? > > > > For summary: > > > > KSPCreate(mesh.comm().get(), &ksp); > > petscErr = PetscOptionsSetValue(NULL,"-ksp_type", "gmres"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_type", "hypre"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_type", "ams"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_type", "2"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_times", "1"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_weigh", "1.0"); > > petscErr = PetscOptionsSetValue(NULL, "-pc_hypre_ams_omega", "1.0"); > > petscErr = PetscOptionsSetValue(NULL, "-ksp_view", NULL); > > petscErr = PetscOptionsSetValue(NULL, "-ksp_monitor_true_residual", > NULL); > > petscErr = PetscOptionsSetValue(NULL, "-ksp_converged_reason", NULL); > > petscErr = PetscOptionsSetValue(NULL, "-options_left", NULL); > > petscErr = KSPSetFromOptions(ksp); > Where is the KSPGetPC() call? Thanks, Matt > // Set pc type > > petscErr = PCSetType(pc, "hypre"); > > > > // Set discrete gradient > > petscErr = PCHYPRESetDiscreteGradient(pc, par_G); > > > > // Set vertex coordinates > > petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, > par_zvec); > > > > > > *From: *Matthew Knepley > *Date: *Monday, 9 September 2024 at 19:46 > *To: *Chockalingam, Karthikeyan (STFC,DL,HC) < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *Stefano Zampini , petsc-users at mcs.anl.gov > > *Subject: *Re: [petsc-users] Hypre AMS usage > > On Mon, Sep 9, 2024 at 2:18?PM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > Calling the two functions after KSPSetFromOptions() did not work either. > > > > Can you check that the PC has the correct type when you call it? > > > > Thanks, > > > > Matt > > > > Everything works from the command line. > > I haven?t set KSPSetOperators, not sure if that is an issue. > > > > *From: *Matthew Knepley > *Date: *Monday, 9 September 2024 at 19:09 > *To: *Chockalingam, Karthikeyan (STFC,DL,HC) < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *Stefano Zampini , petsc-users at mcs.anl.gov > > *Subject: *Re: [petsc-users] Hypre AMS usage > > On Mon, Sep 9, 2024 at 1:21?PM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > Hi Stefano, > > > > Thank you. That was helpful. I tried the following: > > > > petscErr = PetscOptionsSetValue(NULL,"-ksp_type", "gmres"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_type", "hypre"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_type", "ams"); > > > > // Set discrete gradient > > petscErr = PCHYPRESetDiscreteGradient(pc, par_G); > > > > // Set vertex coordinates > > petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, > par_zvec); > > > > petscErr = PetscOptionsSetValue(NULL, "-ksp_view", NULL); > > petscErr = PetscOptionsSetValue(NULL, "-ksp_monitor_true_residual", > NULL); > > petscErr = PetscOptionsSetValue(NULL, "-ksp_converged_reason", NULL); > > petscErr = PetscOptionsSetValue(NULL, "-options_left", NULL); > > petscErr = KSPSetFromOptions(ksp); > > > > > > It errored though I have set PCHYPRESetEdgeConstantVectors. > > > > My guess is that the "pc" was not yet Hypre, since that type was not set > until KSPSetFromOptions() was called. So you need > > to call those two functions after that call. > > > > Thanks, > > > > Matt > > > > But my program works, without PetscOptionsSetValue and by passing > everything on the command line. > > > > *[0]PETSC ERROR: --------------------- Error Message > --------------------------------------------------------------* > > [0]PETSC ERROR: HYPRE AMS preconditioner needs either the coordinate > vectors via PCSetCoordinates() or the edge constant vectors via > PCHYPRESetEdgeConstantVectors() or the interpolation matrix via > PCHYPRESetInterpolations() > > [0]PETS C ERROR: WARNING! There are unused option(s) set! Could be the > program crashed before usage or a spelling mistake, etc! > > [0]PETSC ERROR: Option left: name:-ksp_converged_reason (no value) > source: code > > [0]PETSC ERROR: Option left: name:-options_left (no value) source: code > > [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!bPCXhLGg7jBKQ8N7LCSSnf6NovqslSKRmN9Okpj0J-2kMV1H_p1J2k5WL29WsxTzFFOjJqVptu6mtcUECUOq$ for trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.20.3, unknown > > [0]PETSC ERROR: ./example-dbg on a arch-moose named HC20210312 by > karthikeyan.chockalingam Mon Sep 9 18:10:13 2024 > > [0]PETSC ERROR: Configure options --with-64-bit-indices > --with-cxx-dialect=C++17 --with-debugging=no --with-fortran-bindings=0 > --with-mpi=1 --with-openmp=1 --with-shared-libraries=1 --with-sowing=0 > --download-fblaslapack=1 --download-hypre=1 --download-metis=1 > --download-mumps=1 --download-ptscotch=1 --download-parmetis=1 > --download-scalapack=1 --download-slepc=1 --download-strumpack=1 > --download-superlu_dist=1 --download-bison=1 --download-hdf5=1 > --with-hdf5-fortran-bindings=0 > --download-hdf5-configure-arguments="--with-zlib" --with-make-np=4 > > [0]PETSC ERROR: #1 PCSetUp_HYPRE() at > /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/pc/impls/hypre/hypre.c:295 > > [0]PETSC ERROR: #2 PCSetUp() at > /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/pc/interface/precon.c:1080 > > [0]PETSC ERROR: #3 KSPSetUp() at > /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/ksp/interface/itfunc.c:415 > > [0]PETSC ERROR: #4 KSPSolve_Private() at > /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/ksp/interface/itfunc.c:833 > > [0]PETSC ERROR: #5 KSPSolve() at > /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/ksp/interface/itfunc.c:1080 > > libMesh terminating: > > HYPRE AMS preconditioner needs either the coordinate vectors via > PCSetCoordinates() or the edge constant vectors via > PCHYPRESetEdgeConstantVectors() or the interpolation matrix via > PCHYPRESetInterpolations() > > > > > > *From: *Stefano Zampini > *Date: *Monday, 9 September 2024 at 17:02 > *To: *Matthew Knepley > *Cc: *Chockalingam, Karthikeyan (STFC,DL,HC) < > karthikeyan.chockalingam at stfc.ac.uk>, petsc-users at mcs.anl.gov < > petsc-users at mcs.anl.gov> > *Subject: *Re: [petsc-users] Hypre AMS usage > > I would say the best way is to look at the source code > > > > > https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/hypre/hypre.c?ref_type=heads*L2061__;Iw!!G_uCfscf7eWS!bPCXhLGg7jBKQ8N7LCSSnf6NovqslSKRmN9Okpj0J-2kMV1H_p1J2k5WL29WsxTzFFOjJqVptu6mtcheDSS5$ > > > https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/hypre/hypre.c?ref_type=heads*L1239__;Iw!!G_uCfscf7eWS!bPCXhLGg7jBKQ8N7LCSSnf6NovqslSKRmN9Okpj0J-2kMV1H_p1J2k5WL29WsxTzFFOjJqVptu6mtbmu07n0$ > > > > Il giorno lun 9 set 2024 alle ore 18:50 Matthew Knepley > ha scritto: > > On Mon, Sep 9, 2024 at 10:17?AM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > Hi Matt, > > > > You mentioned it doesn?t hurt to set the smoothing flags > > > > > https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L965__;Iw!!G_uCfscf7eWS!bPCXhLGg7jBKQ8N7LCSSnf6NovqslSKRmN9Okpj0J-2kMV1H_p1J2k5WL29WsxTzFFOjJqVptu6mtQ3XnN-5$ > > > Do you know where I can look for the equivalent PETSc commands? Thank you. > > > > Yes, this is described here: > https://urldefense.us/v3/__https://petsc.org/main/manualpages/PC/PCHYPRE/__;!!G_uCfscf7eWS!bPCXhLGg7jBKQ8N7LCSSnf6NovqslSKRmN9Okpj0J-2kMV1H_p1J2k5WL29WsxTzFFOjJqVptu6mtTAEu7DZ$ > > > The best way is to run with -help or look at the source. > > > > Thanks, > > > > Matt > > > > > > Kind regards, > > Karthik. > > > > *From: *Matthew Knepley > *Date: *Friday, 6 September 2024 at 17:57 > *To: *Chockalingam, Karthikeyan (STFC,DL,HC) < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *petsc-users at mcs.anl.gov > *Subject: *Re: [petsc-users] Hypre AMS usage > > On Fri, Sep 6, 2024 at 11:37?AM Karthikeyan Chockalingam - STFC UKRI via > petsc-users wrote: > > Hello, > > > > I am trying to use the Hypre AMS preconditioner for the first time. > > > > I am following the example problem from Hypre > > > https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L954__;Iw!!G_uCfscf7eWS!bPCXhLGg7jBKQ8N7LCSSnf6NovqslSKRmN9Okpj0J-2kMV1H_p1J2k5WL29WsxTzFFOjJqVptu6mtXIoB5oo$ > > > > > I have so far successfully set the discrete gradient operator and vertex > co-ordinates, > > > > // Set discrete gradient > > petscErr = PCHYPRESetDiscreteGradient(pc, par_G); > > > > // Set vertex coordinates > > petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, > par_zvec); > > > > Do I need to set the following smoothing options? > > > https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L965__;Iw!!G_uCfscf7eWS!bPCXhLGg7jBKQ8N7LCSSnf6NovqslSKRmN9Okpj0J-2kMV1H_p1J2k5WL29WsxTzFFOjJqVptu6mtQ3XnN-5$ > > > > > It cannot hurt. I would set them to begin with. > > > > Also, do I need to convert from MATMPIAIJ to CSR? > > > https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L984__;Iw!!G_uCfscf7eWS!bPCXhLGg7jBKQ8N7LCSSnf6NovqslSKRmN9Okpj0J-2kMV1H_p1J2k5WL29WsxTzFFOjJqVptu6mtTwjkoKg$ > > > > > No. > > > > What are the other PETSc calls to invoke AMS? Is there an example problem > I can look at? > > > > I do not know. I don't think we have an example. > > > > Thanks, > > > > Matt > > > > Thank you. > > > > Karthik. > > > > -- > > *Karthik Chockalingam, Ph.D.* > > Senior Research Software Engineer > > High Performance Systems Engineering Group > > Hartree Centre | Science and Technology Facilities Council > > karthikeyan.chockalingam at stfc.ac.uk > > > > [image: signature_3970890138] > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bPCXhLGg7jBKQ8N7LCSSnf6NovqslSKRmN9Okpj0J-2kMV1H_p1J2k5WL29WsxTzFFOjJqVptu6mtXHp7Z0r$ > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bPCXhLGg7jBKQ8N7LCSSnf6NovqslSKRmN9Okpj0J-2kMV1H_p1J2k5WL29WsxTzFFOjJqVptu6mtXHp7Z0r$ > > > > > > -- > > Stefano > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bPCXhLGg7jBKQ8N7LCSSnf6NovqslSKRmN9Okpj0J-2kMV1H_p1J2k5WL29WsxTzFFOjJqVptu6mtXHp7Z0r$ > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bPCXhLGg7jBKQ8N7LCSSnf6NovqslSKRmN9Okpj0J-2kMV1H_p1J2k5WL29WsxTzFFOjJqVptu6mtXHp7Z0r$ > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bPCXhLGg7jBKQ8N7LCSSnf6NovqslSKRmN9Okpj0J-2kMV1H_p1J2k5WL29WsxTzFFOjJqVptu6mtXHp7Z0r$ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 18270 bytes Desc: not available URL: From karthikeyan.chockalingam at stfc.ac.uk Mon Sep 9 15:03:22 2024 From: karthikeyan.chockalingam at stfc.ac.uk (Karthikeyan Chockalingam - STFC UKRI) Date: Mon, 9 Sep 2024 20:03:22 +0000 Subject: [petsc-users] Hypre AMS usage In-Reply-To: References: Message-ID: I never explicitly called KSPGetPC(). I am embedding AMS in a libmesh (fem) example. So, I only got the reference to pc from libmesh and set petscErr = PCHYPRESetDiscreteGradient(pc, par_G); From: Matthew Knepley Date: Monday, 9 September 2024 at 20:57 To: Chockalingam, Karthikeyan (STFC,DL,HC) Cc: Stefano Zampini , petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Hypre AMS usage On Mon, Sep 9, 2024 at 3:38?PM Karthikeyan Chockalingam - STFC UKRI > wrote: I didn?t know how to check the pc type but adding petscErr = PCSetType(pc, "hypre"); before the two functions made it to work. But how does it work from the command line? For summary: KSPCreate(mesh.comm().get(), &ksp); petscErr = PetscOptionsSetValue(NULL,"-ksp_type", "gmres"); petscErr = PetscOptionsSetValue(NULL,"-pc_type", "hypre"); petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_type", "ams"); petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_type", "2"); petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_times", "1"); petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_weigh", "1.0"); petscErr = PetscOptionsSetValue(NULL, "-pc_hypre_ams_omega", "1.0"); petscErr = PetscOptionsSetValue(NULL, "-ksp_view", NULL); petscErr = PetscOptionsSetValue(NULL, "-ksp_monitor_true_residual", NULL); petscErr = PetscOptionsSetValue(NULL, "-ksp_converged_reason", NULL); petscErr = PetscOptionsSetValue(NULL, "-options_left", NULL); petscErr = KSPSetFromOptions(ksp); Where is the KSPGetPC() call? Thanks, Matt // Set pc type petscErr = PCSetType(pc, "hypre"); // Set discrete gradient petscErr = PCHYPRESetDiscreteGradient(pc, par_G); // Set vertex coordinates petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, par_zvec); From: Matthew Knepley > Date: Monday, 9 September 2024 at 19:46 To: Chockalingam, Karthikeyan (STFC,DL,HC) > Cc: Stefano Zampini >, petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Hypre AMS usage On Mon, Sep 9, 2024 at 2:18?PM Karthikeyan Chockalingam - STFC UKRI > wrote: Calling the two functions after KSPSetFromOptions() did not work either. Can you check that the PC has the correct type when you call it? Thanks, Matt Everything works from the command line. I haven?t set KSPSetOperators, not sure if that is an issue. From: Matthew Knepley > Date: Monday, 9 September 2024 at 19:09 To: Chockalingam, Karthikeyan (STFC,DL,HC) > Cc: Stefano Zampini >, petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Hypre AMS usage On Mon, Sep 9, 2024 at 1:21?PM Karthikeyan Chockalingam - STFC UKRI > wrote: Hi Stefano, Thank you. That was helpful. I tried the following: petscErr = PetscOptionsSetValue(NULL,"-ksp_type", "gmres"); petscErr = PetscOptionsSetValue(NULL,"-pc_type", "hypre"); petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_type", "ams"); // Set discrete gradient petscErr = PCHYPRESetDiscreteGradient(pc, par_G); // Set vertex coordinates petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, par_zvec); petscErr = PetscOptionsSetValue(NULL, "-ksp_view", NULL); petscErr = PetscOptionsSetValue(NULL, "-ksp_monitor_true_residual", NULL); petscErr = PetscOptionsSetValue(NULL, "-ksp_converged_reason", NULL); petscErr = PetscOptionsSetValue(NULL, "-options_left", NULL); petscErr = KSPSetFromOptions(ksp); It errored though I have set PCHYPRESetEdgeConstantVectors. My guess is that the "pc" was not yet Hypre, since that type was not set until KSPSetFromOptions() was called. So you need to call those two functions after that call. Thanks, Matt But my program works, without PetscOptionsSetValue and by passing everything on the command line. [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: HYPRE AMS preconditioner needs either the coordinate vectors via PCSetCoordinates() or the edge constant vectors via PCHYPRESetEdgeConstantVectors() or the interpolation matrix via PCHYPRESetInterpolations() [0]PETS C ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc! [0]PETSC ERROR: Option left: name:-ksp_converged_reason (no value) source: code [0]PETSC ERROR: Option left: name:-options_left (no value) source: code [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!dkm28NStZmKXBlf2nSMgCMkJC8CQ8T2tnmhAwsND0Bz74mxG_LyGyHWDGWF6E85XLcebgv75DeJV7kbMV7Pq8IjFgcgXBlaDxg3B$ for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.20.3, unknown [0]PETSC ERROR: ./example-dbg on a arch-moose named HC20210312 by karthikeyan.chockalingam Mon Sep 9 18:10:13 2024 [0]PETSC ERROR: Configure options --with-64-bit-indices --with-cxx-dialect=C++17 --with-debugging=no --with-fortran-bindings=0 --with-mpi=1 --with-openmp=1 --with-shared-libraries=1 --with-sowing=0 --download-fblaslapack=1 --download-hypre=1 --download-metis=1 --download-mumps=1 --download-ptscotch=1 --download-parmetis=1 --download-scalapack=1 --download-slepc=1 --download-strumpack=1 --download-superlu_dist=1 --download-bison=1 --download-hdf5=1 --with-hdf5-fortran-bindings=0 --download-hdf5-configure-arguments="--with-zlib" --with-make-np=4 [0]PETSC ERROR: #1 PCSetUp_HYPRE() at /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/pc/impls/hypre/hypre.c:295 [0]PETSC ERROR: #2 PCSetUp() at /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/pc/interface/precon.c:1080 [0]PETSC ERROR: #3 KSPSetUp() at /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/ksp/interface/itfunc.c:415 [0]PETSC ERROR: #4 KSPSolve_Private() at /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/ksp/interface/itfunc.c:833 [0]PETSC ERROR: #5 KSPSolve() at /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/ksp/interface/itfunc.c:1080 libMesh terminating: HYPRE AMS preconditioner needs either the coordinate vectors via PCSetCoordinates() or the edge constant vectors via PCHYPRESetEdgeConstantVectors() or the interpolation matrix via PCHYPRESetInterpolations() From: Stefano Zampini > Date: Monday, 9 September 2024 at 17:02 To: Matthew Knepley > Cc: Chockalingam, Karthikeyan (STFC,DL,HC) >, petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Hypre AMS usage I would say the best way is to look at the source code https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/hypre/hypre.c?ref_type=heads*L2061__;Iw!!G_uCfscf7eWS!dkm28NStZmKXBlf2nSMgCMkJC8CQ8T2tnmhAwsND0Bz74mxG_LyGyHWDGWF6E85XLcebgv75DeJV7kbMV7Pq8IjFgcgXBiuQiYeB$ https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/hypre/hypre.c?ref_type=heads*L1239__;Iw!!G_uCfscf7eWS!dkm28NStZmKXBlf2nSMgCMkJC8CQ8T2tnmhAwsND0Bz74mxG_LyGyHWDGWF6E85XLcebgv75DeJV7kbMV7Pq8IjFgcgXBpXp4zIJ$ Il giorno lun 9 set 2024 alle ore 18:50 Matthew Knepley > ha scritto: On Mon, Sep 9, 2024 at 10:17?AM Karthikeyan Chockalingam - STFC UKRI > wrote: Hi Matt, You mentioned it doesn?t hurt to set the smoothing flags https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L965__;Iw!!G_uCfscf7eWS!dkm28NStZmKXBlf2nSMgCMkJC8CQ8T2tnmhAwsND0Bz74mxG_LyGyHWDGWF6E85XLcebgv75DeJV7kbMV7Pq8IjFgcgXBs_nEdNZ$ Do you know where I can look for the equivalent PETSc commands? Thank you. Yes, this is described here: https://urldefense.us/v3/__https://petsc.org/main/manualpages/PC/PCHYPRE/__;!!G_uCfscf7eWS!dkm28NStZmKXBlf2nSMgCMkJC8CQ8T2tnmhAwsND0Bz74mxG_LyGyHWDGWF6E85XLcebgv75DeJV7kbMV7Pq8IjFgcgXBt3EToc2$ The best way is to run with -help or look at the source. Thanks, Matt Kind regards, Karthik. From: Matthew Knepley > Date: Friday, 6 September 2024 at 17:57 To: Chockalingam, Karthikeyan (STFC,DL,HC) > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Hypre AMS usage On Fri, Sep 6, 2024 at 11:37?AM Karthikeyan Chockalingam - STFC UKRI via petsc-users > wrote: Hello, I am trying to use the Hypre AMS preconditioner for the first time. I am following the example problem from Hypre https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L954__;Iw!!G_uCfscf7eWS!dkm28NStZmKXBlf2nSMgCMkJC8CQ8T2tnmhAwsND0Bz74mxG_LyGyHWDGWF6E85XLcebgv75DeJV7kbMV7Pq8IjFgcgXBp6LJT5B$ I have so far successfully set the discrete gradient operator and vertex co-ordinates, // Set discrete gradient petscErr = PCHYPRESetDiscreteGradient(pc, par_G); // Set vertex coordinates petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, par_zvec); Do I need to set the following smoothing options? https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L965__;Iw!!G_uCfscf7eWS!dkm28NStZmKXBlf2nSMgCMkJC8CQ8T2tnmhAwsND0Bz74mxG_LyGyHWDGWF6E85XLcebgv75DeJV7kbMV7Pq8IjFgcgXBs_nEdNZ$ It cannot hurt. I would set them to begin with. Also, do I need to convert from MATMPIAIJ to CSR? https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L984__;Iw!!G_uCfscf7eWS!dkm28NStZmKXBlf2nSMgCMkJC8CQ8T2tnmhAwsND0Bz74mxG_LyGyHWDGWF6E85XLcebgv75DeJV7kbMV7Pq8IjFgcgXBlPLfOch$ No. What are the other PETSc calls to invoke AMS? Is there an example problem I can look at? I do not know. I don't think we have an example. Thanks, Matt Thank you. Karthik. -- Karthik Chockalingam, Ph.D. Senior Research Software Engineer High Performance Systems Engineering Group Hartree Centre | Science and Technology Facilities Council karthikeyan.chockalingam at stfc.ac.uk [signature_3970890138] -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!dkm28NStZmKXBlf2nSMgCMkJC8CQ8T2tnmhAwsND0Bz74mxG_LyGyHWDGWF6E85XLcebgv75DeJV7kbMV7Pq8IjFgcgXBoVYxr6r$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!dkm28NStZmKXBlf2nSMgCMkJC8CQ8T2tnmhAwsND0Bz74mxG_LyGyHWDGWF6E85XLcebgv75DeJV7kbMV7Pq8IjFgcgXBoVYxr6r$ -- Stefano -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!dkm28NStZmKXBlf2nSMgCMkJC8CQ8T2tnmhAwsND0Bz74mxG_LyGyHWDGWF6E85XLcebgv75DeJV7kbMV7Pq8IjFgcgXBoVYxr6r$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!dkm28NStZmKXBlf2nSMgCMkJC8CQ8T2tnmhAwsND0Bz74mxG_LyGyHWDGWF6E85XLcebgv75DeJV7kbMV7Pq8IjFgcgXBoVYxr6r$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!dkm28NStZmKXBlf2nSMgCMkJC8CQ8T2tnmhAwsND0Bz74mxG_LyGyHWDGWF6E85XLcebgv75DeJV7kbMV7Pq8IjFgcgXBoVYxr6r$ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 18270 bytes Desc: image001.png URL: From knepley at gmail.com Mon Sep 9 15:11:33 2024 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 9 Sep 2024 16:11:33 -0400 Subject: [petsc-users] Hypre AMS usage In-Reply-To: References: Message-ID: On Mon, Sep 9, 2024 at 4:03?PM Karthikeyan Chockalingam - STFC UKRI < karthikeyan.chockalingam at stfc.ac.uk> wrote: > I never explicitly called KSPGetPC(). I am embedding AMS in a libmesh > (fem) example. So, I only got the reference to pc from libmesh and set petscErr > = PCHYPRESetDiscreteGradient(pc, par_G); > However, you are creating a _new_ KSP, so your PC will not match it. This is not doing what you want. Thanks, Matt > > > *From: *Matthew Knepley > *Date: *Monday, 9 September 2024 at 20:57 > *To: *Chockalingam, Karthikeyan (STFC,DL,HC) < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *Stefano Zampini , petsc-users at mcs.anl.gov > > *Subject: *Re: [petsc-users] Hypre AMS usage > > On Mon, Sep 9, 2024 at 3:38?PM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > I didn?t know how to check the pc type but adding > > petscErr = PCSetType(pc, "hypre"); > > before the two functions made it to work. > > > > But how does it work from the command line? > > > > For summary: > > > > KSPCreate(mesh.comm().get(), &ksp); > > petscErr = PetscOptionsSetValue(NULL,"-ksp_type", "gmres"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_type", "hypre"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_type", "ams"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_type", "2"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_times", "1"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_weigh", "1.0"); > > petscErr = PetscOptionsSetValue(NULL, "-pc_hypre_ams_omega", "1.0"); > > petscErr = PetscOptionsSetValue(NULL, "-ksp_view", NULL); > > petscErr = PetscOptionsSetValue(NULL, "-ksp_monitor_true_residual", > NULL); > > petscErr = PetscOptionsSetValue(NULL, "-ksp_converged_reason", NULL); > > petscErr = PetscOptionsSetValue(NULL, "-options_left", NULL); > > petscErr = KSPSetFromOptions(ksp); > > > > Where is the KSPGetPC() call? > > > > Thanks, > > > > Matt > > > > // Set pc type > > petscErr = PCSetType(pc, "hypre"); > > > > // Set discrete gradient > > petscErr = PCHYPRESetDiscreteGradient(pc, par_G); > > > > // Set vertex coordinates > > petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, > par_zvec); > > > > > > *From: *Matthew Knepley > *Date: *Monday, 9 September 2024 at 19:46 > *To: *Chockalingam, Karthikeyan (STFC,DL,HC) < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *Stefano Zampini , petsc-users at mcs.anl.gov > > *Subject: *Re: [petsc-users] Hypre AMS usage > > On Mon, Sep 9, 2024 at 2:18?PM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > Calling the two functions after KSPSetFromOptions() did not work either. > > > > Can you check that the PC has the correct type when you call it? > > > > Thanks, > > > > Matt > > > > Everything works from the command line. > > I haven?t set KSPSetOperators, not sure if that is an issue. > > > > *From: *Matthew Knepley > *Date: *Monday, 9 September 2024 at 19:09 > *To: *Chockalingam, Karthikeyan (STFC,DL,HC) < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *Stefano Zampini , petsc-users at mcs.anl.gov > > *Subject: *Re: [petsc-users] Hypre AMS usage > > On Mon, Sep 9, 2024 at 1:21?PM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > Hi Stefano, > > > > Thank you. That was helpful. I tried the following: > > > > petscErr = PetscOptionsSetValue(NULL,"-ksp_type", "gmres"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_type", "hypre"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_type", "ams"); > > > > // Set discrete gradient > > petscErr = PCHYPRESetDiscreteGradient(pc, par_G); > > > > // Set vertex coordinates > > petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, > par_zvec); > > > > petscErr = PetscOptionsSetValue(NULL, "-ksp_view", NULL); > > petscErr = PetscOptionsSetValue(NULL, "-ksp_monitor_true_residual", > NULL); > > petscErr = PetscOptionsSetValue(NULL, "-ksp_converged_reason", NULL); > > petscErr = PetscOptionsSetValue(NULL, "-options_left", NULL); > > petscErr = KSPSetFromOptions(ksp); > > > > > > It errored though I have set PCHYPRESetEdgeConstantVectors. > > > > My guess is that the "pc" was not yet Hypre, since that type was not set > until KSPSetFromOptions() was called. So you need > > to call those two functions after that call. > > > > Thanks, > > > > Matt > > > > But my program works, without PetscOptionsSetValue and by passing > everything on the command line. > > > > *[0]PETSC ERROR: --------------------- Error Message > --------------------------------------------------------------* > > [0]PETSC ERROR: HYPRE AMS preconditioner needs either the coordinate > vectors via PCSetCoordinates() or the edge constant vectors via > PCHYPRESetEdgeConstantVectors() or the interpolation matrix via > PCHYPRESetInterpolations() > > [0]PETS C ERROR: WARNING! There are unused option(s) set! Could be the > program crashed before usage or a spelling mistake, etc! > > [0]PETSC ERROR: Option left: name:-ksp_converged_reason (no value) > source: code > > [0]PETSC ERROR: Option left: name:-options_left (no value) source: code > > [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!YvXIaD60U5fExpCVoRjdXjjoV3VW5D3N2JFqzAjilDUo9FNFNiJnU9HYgan9BFSEge5t9geaE6sHV1MxMQLo$ for trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.20.3, unknown > > [0]PETSC ERROR: ./example-dbg on a arch-moose named HC20210312 by > karthikeyan.chockalingam Mon Sep 9 18:10:13 2024 > > [0]PETSC ERROR: Configure options --with-64-bit-indices > --with-cxx-dialect=C++17 --with-debugging=no --with-fortran-bindings=0 > --with-mpi=1 --with-openmp=1 --with-shared-libraries=1 --with-sowing=0 > --download-fblaslapack=1 --download-hypre=1 --download-metis=1 > --download-mumps=1 --download-ptscotch=1 --download-parmetis=1 > --download-scalapack=1 --download-slepc=1 --download-strumpack=1 > --download-superlu_dist=1 --download-bison=1 --download-hdf5=1 > --with-hdf5-fortran-bindings=0 > --download-hdf5-configure-arguments="--with-zlib" --with-make-np=4 > > [0]PETSC ERROR: #1 PCSetUp_HYPRE() at > /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/pc/impls/hypre/hypre.c:295 > > [0]PETSC ERROR: #2 PCSetUp() at > /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/pc/interface/precon.c:1080 > > [0]PETSC ERROR: #3 KSPSetUp() at > /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/ksp/interface/itfunc.c:415 > > [0]PETSC ERROR: #4 KSPSolve_Private() at > /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/ksp/interface/itfunc.c:833 > > [0]PETSC ERROR: #5 KSPSolve() at > /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/ksp/interface/itfunc.c:1080 > > libMesh terminating: > > HYPRE AMS preconditioner needs either the coordinate vectors via > PCSetCoordinates() or the edge constant vectors via > PCHYPRESetEdgeConstantVectors() or the interpolation matrix via > PCHYPRESetInterpolations() > > > > > > *From: *Stefano Zampini > *Date: *Monday, 9 September 2024 at 17:02 > *To: *Matthew Knepley > *Cc: *Chockalingam, Karthikeyan (STFC,DL,HC) < > karthikeyan.chockalingam at stfc.ac.uk>, petsc-users at mcs.anl.gov < > petsc-users at mcs.anl.gov> > *Subject: *Re: [petsc-users] Hypre AMS usage > > I would say the best way is to look at the source code > > > > > https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/hypre/hypre.c?ref_type=heads*L2061__;Iw!!G_uCfscf7eWS!YvXIaD60U5fExpCVoRjdXjjoV3VW5D3N2JFqzAjilDUo9FNFNiJnU9HYgan9BFSEge5t9geaE6sHV2lmCywz$ > > > https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/hypre/hypre.c?ref_type=heads*L1239__;Iw!!G_uCfscf7eWS!YvXIaD60U5fExpCVoRjdXjjoV3VW5D3N2JFqzAjilDUo9FNFNiJnU9HYgan9BFSEge5t9geaE6sHV3Fuvspu$ > > > > Il giorno lun 9 set 2024 alle ore 18:50 Matthew Knepley > ha scritto: > > On Mon, Sep 9, 2024 at 10:17?AM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > Hi Matt, > > > > You mentioned it doesn?t hurt to set the smoothing flags > > > > > https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L965__;Iw!!G_uCfscf7eWS!YvXIaD60U5fExpCVoRjdXjjoV3VW5D3N2JFqzAjilDUo9FNFNiJnU9HYgan9BFSEge5t9geaE6sHV4ecyZkn$ > > > Do you know where I can look for the equivalent PETSc commands? Thank you. > > > > Yes, this is described here: > https://urldefense.us/v3/__https://petsc.org/main/manualpages/PC/PCHYPRE/__;!!G_uCfscf7eWS!YvXIaD60U5fExpCVoRjdXjjoV3VW5D3N2JFqzAjilDUo9FNFNiJnU9HYgan9BFSEge5t9geaE6sHV77iGsAg$ > > > The best way is to run with -help or look at the source. > > > > Thanks, > > > > Matt > > > > > > Kind regards, > > Karthik. > > > > *From: *Matthew Knepley > *Date: *Friday, 6 September 2024 at 17:57 > *To: *Chockalingam, Karthikeyan (STFC,DL,HC) < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *petsc-users at mcs.anl.gov > *Subject: *Re: [petsc-users] Hypre AMS usage > > On Fri, Sep 6, 2024 at 11:37?AM Karthikeyan Chockalingam - STFC UKRI via > petsc-users wrote: > > Hello, > > > > I am trying to use the Hypre AMS preconditioner for the first time. > > > > I am following the example problem from Hypre > > > https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L954__;Iw!!G_uCfscf7eWS!YvXIaD60U5fExpCVoRjdXjjoV3VW5D3N2JFqzAjilDUo9FNFNiJnU9HYgan9BFSEge5t9geaE6sHV_FzBmeV$ > > > > > I have so far successfully set the discrete gradient operator and vertex > co-ordinates, > > > > // Set discrete gradient > > petscErr = PCHYPRESetDiscreteGradient(pc, par_G); > > > > // Set vertex coordinates > > petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, > par_zvec); > > > > Do I need to set the following smoothing options? > > > https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L965__;Iw!!G_uCfscf7eWS!YvXIaD60U5fExpCVoRjdXjjoV3VW5D3N2JFqzAjilDUo9FNFNiJnU9HYgan9BFSEge5t9geaE6sHV4ecyZkn$ > > > > > It cannot hurt. I would set them to begin with. > > > > Also, do I need to convert from MATMPIAIJ to CSR? > > > https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L984__;Iw!!G_uCfscf7eWS!YvXIaD60U5fExpCVoRjdXjjoV3VW5D3N2JFqzAjilDUo9FNFNiJnU9HYgan9BFSEge5t9geaE6sHVw2wjHgs$ > > > > > No. > > > > What are the other PETSc calls to invoke AMS? Is there an example problem > I can look at? > > > > I do not know. I don't think we have an example. > > > > Thanks, > > > > Matt > > > > Thank you. > > > > Karthik. > > > > -- > > *Karthik Chockalingam, Ph.D.* > > Senior Research Software Engineer > > High Performance Systems Engineering Group > > Hartree Centre | Science and Technology Facilities Council > > karthikeyan.chockalingam at stfc.ac.uk > > > > [image: signature_3970890138] > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YvXIaD60U5fExpCVoRjdXjjoV3VW5D3N2JFqzAjilDUo9FNFNiJnU9HYgan9BFSEge5t9geaE6sHV75eLTpr$ > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YvXIaD60U5fExpCVoRjdXjjoV3VW5D3N2JFqzAjilDUo9FNFNiJnU9HYgan9BFSEge5t9geaE6sHV75eLTpr$ > > > > > > -- > > Stefano > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YvXIaD60U5fExpCVoRjdXjjoV3VW5D3N2JFqzAjilDUo9FNFNiJnU9HYgan9BFSEge5t9geaE6sHV75eLTpr$ > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YvXIaD60U5fExpCVoRjdXjjoV3VW5D3N2JFqzAjilDUo9FNFNiJnU9HYgan9BFSEge5t9geaE6sHV75eLTpr$ > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YvXIaD60U5fExpCVoRjdXjjoV3VW5D3N2JFqzAjilDUo9FNFNiJnU9HYgan9BFSEge5t9geaE6sHV75eLTpr$ > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YvXIaD60U5fExpCVoRjdXjjoV3VW5D3N2JFqzAjilDUo9FNFNiJnU9HYgan9BFSEge5t9geaE6sHV75eLTpr$ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 18270 bytes Desc: not available URL: From karthikeyan.chockalingam at stfc.ac.uk Mon Sep 9 15:32:21 2024 From: karthikeyan.chockalingam at stfc.ac.uk (Karthikeyan Chockalingam - STFC UKRI) Date: Mon, 9 Sep 2024 20:32:21 +0000 Subject: [petsc-users] Hypre AMS usage In-Reply-To: References: Message-ID: Great. Thank you for letting me know. I got the reference to KSP as well from libmesh and I am not creating a new KSP. This time around, I didn?t have to set the PC type and it seems to work. petscErr = PetscOptionsSetValue(NULL,"-ksp_type", "gmres"); petscErr = PetscOptionsSetValue(NULL,"-pc_type", "hypre"); petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_type", "ams"); petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_type", "2"); petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_times", "1"); petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_weight", "1.0"); petscErr = PetscOptionsSetValue(NULL, "-pc_hypre_ams_omega", "1.0"); petscErr = PetscOptionsSetValue(NULL, "-ksp_view", NULL); petscErr = PetscOptionsSetValue(NULL, "-ksp_monitor_true_residual", NULL); petscErr = PetscOptionsSetValue(NULL, "-ksp_converged_reason", NULL); petscErr = PetscOptionsSetValue(NULL, "-options_left", NULL); petscErr = KSPSetFromOptions(ksp); // Set discrete gradient petscErr = PCHYPRESetDiscreteGradient(pc, par_G); // Set vertex coordinates petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, par_zvec); From: Matthew Knepley Date: Monday, 9 September 2024 at 21:11 To: Chockalingam, Karthikeyan (STFC,DL,HC) Cc: Stefano Zampini , petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Hypre AMS usage On Mon, Sep 9, 2024 at 4:03?PM Karthikeyan Chockalingam - STFC UKRI > wrote: I never explicitly called KSPGetPC(). I am embedding AMS in a libmesh (fem) example. So, I only got the reference to pc from libmesh and set petscErr = PCHYPRESetDiscreteGradient(pc, par_G); However, you are creating a _new_ KSP, so your PC will not match it. This is not doing what you want. Thanks, Matt From: Matthew Knepley > Date: Monday, 9 September 2024 at 20:57 To: Chockalingam, Karthikeyan (STFC,DL,HC) > Cc: Stefano Zampini >, petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Hypre AMS usage On Mon, Sep 9, 2024 at 3:38?PM Karthikeyan Chockalingam - STFC UKRI > wrote: I didn?t know how to check the pc type but adding petscErr = PCSetType(pc, "hypre"); before the two functions made it to work. But how does it work from the command line? For summary: KSPCreate(mesh.comm().get(), &ksp); petscErr = PetscOptionsSetValue(NULL,"-ksp_type", "gmres"); petscErr = PetscOptionsSetValue(NULL,"-pc_type", "hypre"); petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_type", "ams"); petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_type", "2"); petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_times", "1"); petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_weigh", "1.0"); petscErr = PetscOptionsSetValue(NULL, "-pc_hypre_ams_omega", "1.0"); petscErr = PetscOptionsSetValue(NULL, "-ksp_view", NULL); petscErr = PetscOptionsSetValue(NULL, "-ksp_monitor_true_residual", NULL); petscErr = PetscOptionsSetValue(NULL, "-ksp_converged_reason", NULL); petscErr = PetscOptionsSetValue(NULL, "-options_left", NULL); petscErr = KSPSetFromOptions(ksp); Where is the KSPGetPC() call? Thanks, Matt // Set pc type petscErr = PCSetType(pc, "hypre"); // Set discrete gradient petscErr = PCHYPRESetDiscreteGradient(pc, par_G); // Set vertex coordinates petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, par_zvec); From: Matthew Knepley > Date: Monday, 9 September 2024 at 19:46 To: Chockalingam, Karthikeyan (STFC,DL,HC) > Cc: Stefano Zampini >, petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Hypre AMS usage On Mon, Sep 9, 2024 at 2:18?PM Karthikeyan Chockalingam - STFC UKRI > wrote: Calling the two functions after KSPSetFromOptions() did not work either. Can you check that the PC has the correct type when you call it? Thanks, Matt Everything works from the command line. I haven?t set KSPSetOperators, not sure if that is an issue. From: Matthew Knepley > Date: Monday, 9 September 2024 at 19:09 To: Chockalingam, Karthikeyan (STFC,DL,HC) > Cc: Stefano Zampini >, petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Hypre AMS usage On Mon, Sep 9, 2024 at 1:21?PM Karthikeyan Chockalingam - STFC UKRI > wrote: Hi Stefano, Thank you. That was helpful. I tried the following: petscErr = PetscOptionsSetValue(NULL,"-ksp_type", "gmres"); petscErr = PetscOptionsSetValue(NULL,"-pc_type", "hypre"); petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_type", "ams"); // Set discrete gradient petscErr = PCHYPRESetDiscreteGradient(pc, par_G); // Set vertex coordinates petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, par_zvec); petscErr = PetscOptionsSetValue(NULL, "-ksp_view", NULL); petscErr = PetscOptionsSetValue(NULL, "-ksp_monitor_true_residual", NULL); petscErr = PetscOptionsSetValue(NULL, "-ksp_converged_reason", NULL); petscErr = PetscOptionsSetValue(NULL, "-options_left", NULL); petscErr = KSPSetFromOptions(ksp); It errored though I have set PCHYPRESetEdgeConstantVectors. My guess is that the "pc" was not yet Hypre, since that type was not set until KSPSetFromOptions() was called. So you need to call those two functions after that call. Thanks, Matt But my program works, without PetscOptionsSetValue and by passing everything on the command line. [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: HYPRE AMS preconditioner needs either the coordinate vectors via PCSetCoordinates() or the edge constant vectors via PCHYPRESetEdgeConstantVectors() or the interpolation matrix via PCHYPRESetInterpolations() [0]PETS C ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc! [0]PETSC ERROR: Option left: name:-ksp_converged_reason (no value) source: code [0]PETSC ERROR: Option left: name:-options_left (no value) source: code [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!a5Zwb43X7PZGzxpqU7jilYgIMqB2zsEqJbHn7VhNidCgapU8XJf_cyVQ0MLldjneo_O6KL-PXk0i9LhgTVl_G5ICFiivGuWhrhe0$ for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.20.3, unknown [0]PETSC ERROR: ./example-dbg on a arch-moose named HC20210312 by karthikeyan.chockalingam Mon Sep 9 18:10:13 2024 [0]PETSC ERROR: Configure options --with-64-bit-indices --with-cxx-dialect=C++17 --with-debugging=no --with-fortran-bindings=0 --with-mpi=1 --with-openmp=1 --with-shared-libraries=1 --with-sowing=0 --download-fblaslapack=1 --download-hypre=1 --download-metis=1 --download-mumps=1 --download-ptscotch=1 --download-parmetis=1 --download-scalapack=1 --download-slepc=1 --download-strumpack=1 --download-superlu_dist=1 --download-bison=1 --download-hdf5=1 --with-hdf5-fortran-bindings=0 --download-hdf5-configure-arguments="--with-zlib" --with-make-np=4 [0]PETSC ERROR: #1 PCSetUp_HYPRE() at /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/pc/impls/hypre/hypre.c:295 [0]PETSC ERROR: #2 PCSetUp() at /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/pc/interface/precon.c:1080 [0]PETSC ERROR: #3 KSPSetUp() at /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/ksp/interface/itfunc.c:415 [0]PETSC ERROR: #4 KSPSolve_Private() at /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/ksp/interface/itfunc.c:833 [0]PETSC ERROR: #5 KSPSolve() at /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/ksp/interface/itfunc.c:1080 libMesh terminating: HYPRE AMS preconditioner needs either the coordinate vectors via PCSetCoordinates() or the edge constant vectors via PCHYPRESetEdgeConstantVectors() or the interpolation matrix via PCHYPRESetInterpolations() From: Stefano Zampini > Date: Monday, 9 September 2024 at 17:02 To: Matthew Knepley > Cc: Chockalingam, Karthikeyan (STFC,DL,HC) >, petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Hypre AMS usage I would say the best way is to look at the source code https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/hypre/hypre.c?ref_type=heads*L2061__;Iw!!G_uCfscf7eWS!a5Zwb43X7PZGzxpqU7jilYgIMqB2zsEqJbHn7VhNidCgapU8XJf_cyVQ0MLldjneo_O6KL-PXk0i9LhgTVl_G5ICFiivGsFBV2kS$ https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/hypre/hypre.c?ref_type=heads*L1239__;Iw!!G_uCfscf7eWS!a5Zwb43X7PZGzxpqU7jilYgIMqB2zsEqJbHn7VhNidCgapU8XJf_cyVQ0MLldjneo_O6KL-PXk0i9LhgTVl_G5ICFiivGpsUG5OL$ Il giorno lun 9 set 2024 alle ore 18:50 Matthew Knepley > ha scritto: On Mon, Sep 9, 2024 at 10:17?AM Karthikeyan Chockalingam - STFC UKRI > wrote: Hi Matt, You mentioned it doesn?t hurt to set the smoothing flags https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L965__;Iw!!G_uCfscf7eWS!a5Zwb43X7PZGzxpqU7jilYgIMqB2zsEqJbHn7VhNidCgapU8XJf_cyVQ0MLldjneo_O6KL-PXk0i9LhgTVl_G5ICFiivGi2Ptk3R$ Do you know where I can look for the equivalent PETSc commands? Thank you. Yes, this is described here: https://urldefense.us/v3/__https://petsc.org/main/manualpages/PC/PCHYPRE/__;!!G_uCfscf7eWS!a5Zwb43X7PZGzxpqU7jilYgIMqB2zsEqJbHn7VhNidCgapU8XJf_cyVQ0MLldjneo_O6KL-PXk0i9LhgTVl_G5ICFiivGj1Vl6P_$ The best way is to run with -help or look at the source. Thanks, Matt Kind regards, Karthik. From: Matthew Knepley > Date: Friday, 6 September 2024 at 17:57 To: Chockalingam, Karthikeyan (STFC,DL,HC) > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Hypre AMS usage On Fri, Sep 6, 2024 at 11:37?AM Karthikeyan Chockalingam - STFC UKRI via petsc-users > wrote: Hello, I am trying to use the Hypre AMS preconditioner for the first time. I am following the example problem from Hypre https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L954__;Iw!!G_uCfscf7eWS!a5Zwb43X7PZGzxpqU7jilYgIMqB2zsEqJbHn7VhNidCgapU8XJf_cyVQ0MLldjneo_O6KL-PXk0i9LhgTVl_G5ICFiivGtimJYNe$ I have so far successfully set the discrete gradient operator and vertex co-ordinates, // Set discrete gradient petscErr = PCHYPRESetDiscreteGradient(pc, par_G); // Set vertex coordinates petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, par_zvec); Do I need to set the following smoothing options? https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L965__;Iw!!G_uCfscf7eWS!a5Zwb43X7PZGzxpqU7jilYgIMqB2zsEqJbHn7VhNidCgapU8XJf_cyVQ0MLldjneo_O6KL-PXk0i9LhgTVl_G5ICFiivGi2Ptk3R$ It cannot hurt. I would set them to begin with. Also, do I need to convert from MATMPIAIJ to CSR? https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L984__;Iw!!G_uCfscf7eWS!a5Zwb43X7PZGzxpqU7jilYgIMqB2zsEqJbHn7VhNidCgapU8XJf_cyVQ0MLldjneo_O6KL-PXk0i9LhgTVl_G5ICFiivGktWzffK$ No. What are the other PETSc calls to invoke AMS? Is there an example problem I can look at? I do not know. I don't think we have an example. Thanks, Matt Thank you. Karthik. -- Karthik Chockalingam, Ph.D. Senior Research Software Engineer High Performance Systems Engineering Group Hartree Centre | Science and Technology Facilities Council karthikeyan.chockalingam at stfc.ac.uk [signature_3970890138] -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!a5Zwb43X7PZGzxpqU7jilYgIMqB2zsEqJbHn7VhNidCgapU8XJf_cyVQ0MLldjneo_O6KL-PXk0i9LhgTVl_G5ICFiivGs2jyxEL$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!a5Zwb43X7PZGzxpqU7jilYgIMqB2zsEqJbHn7VhNidCgapU8XJf_cyVQ0MLldjneo_O6KL-PXk0i9LhgTVl_G5ICFiivGs2jyxEL$ -- Stefano -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!a5Zwb43X7PZGzxpqU7jilYgIMqB2zsEqJbHn7VhNidCgapU8XJf_cyVQ0MLldjneo_O6KL-PXk0i9LhgTVl_G5ICFiivGs2jyxEL$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!a5Zwb43X7PZGzxpqU7jilYgIMqB2zsEqJbHn7VhNidCgapU8XJf_cyVQ0MLldjneo_O6KL-PXk0i9LhgTVl_G5ICFiivGs2jyxEL$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!a5Zwb43X7PZGzxpqU7jilYgIMqB2zsEqJbHn7VhNidCgapU8XJf_cyVQ0MLldjneo_O6KL-PXk0i9LhgTVl_G5ICFiivGs2jyxEL$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!a5Zwb43X7PZGzxpqU7jilYgIMqB2zsEqJbHn7VhNidCgapU8XJf_cyVQ0MLldjneo_O6KL-PXk0i9LhgTVl_G5ICFiivGs2jyxEL$ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 18270 bytes Desc: image001.png URL: From knepley at gmail.com Mon Sep 9 16:00:01 2024 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 9 Sep 2024 17:00:01 -0400 Subject: [petsc-users] Hypre AMS usage In-Reply-To: References: Message-ID: On Mon, Sep 9, 2024 at 4:32?PM Karthikeyan Chockalingam - STFC UKRI < karthikeyan.chockalingam at stfc.ac.uk> wrote: > Great. Thank you for letting me know. > > I got the reference to KSP as well from libmesh and I am not creating a > new KSP. > > This time around, I didn?t have to set the PC type and it seems to work. > Excellent. Thanks, Matt > petscErr = PetscOptionsSetValue(NULL,"-ksp_type", "gmres"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_type", "hypre"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_type", "ams"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_type", "2"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_times", "1"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_weight", > "1.0"); > > petscErr = PetscOptionsSetValue(NULL, "-pc_hypre_ams_omega", "1.0"); > > petscErr = PetscOptionsSetValue(NULL, "-ksp_view", NULL); > > petscErr = PetscOptionsSetValue(NULL, "-ksp_monitor_true_residual", > NULL); > > petscErr = PetscOptionsSetValue(NULL, "-ksp_converged_reason", NULL); > > petscErr = PetscOptionsSetValue(NULL, "-options_left", NULL); > > petscErr = KSPSetFromOptions(ksp); > > > > // Set discrete gradient > > petscErr = PCHYPRESetDiscreteGradient(pc, par_G); > > > > // Set vertex coordinates > > petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, > par_zvec); > > > > > > *From: *Matthew Knepley > *Date: *Monday, 9 September 2024 at 21:11 > *To: *Chockalingam, Karthikeyan (STFC,DL,HC) < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *Stefano Zampini , petsc-users at mcs.anl.gov > > *Subject: *Re: [petsc-users] Hypre AMS usage > > On Mon, Sep 9, 2024 at 4:03?PM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > I never explicitly called KSPGetPC(). I am embedding AMS in a libmesh > (fem) example. So, I only got the reference to pc from libmesh and set > petscErr = PCHYPRESetDiscreteGradient(pc, par_G); > > > > However, you are creating a _new_ KSP, so your PC will not match it. This > is not doing what you want. > > > > Thanks, > > > > Matt > > > > > > *From: *Matthew Knepley > *Date: *Monday, 9 September 2024 at 20:57 > *To: *Chockalingam, Karthikeyan (STFC,DL,HC) < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *Stefano Zampini , petsc-users at mcs.anl.gov > > *Subject: *Re: [petsc-users] Hypre AMS usage > > On Mon, Sep 9, 2024 at 3:38?PM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > I didn?t know how to check the pc type but adding > > petscErr = PCSetType(pc, "hypre"); > > before the two functions made it to work. > > > > But how does it work from the command line? > > > > For summary: > > > > KSPCreate(mesh.comm().get(), &ksp); > > petscErr = PetscOptionsSetValue(NULL,"-ksp_type", "gmres"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_type", "hypre"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_type", "ams"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_type", "2"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_times", "1"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_weigh", "1.0"); > > petscErr = PetscOptionsSetValue(NULL, "-pc_hypre_ams_omega", "1.0"); > > petscErr = PetscOptionsSetValue(NULL, "-ksp_view", NULL); > > petscErr = PetscOptionsSetValue(NULL, "-ksp_monitor_true_residual", > NULL); > > petscErr = PetscOptionsSetValue(NULL, "-ksp_converged_reason", NULL); > > petscErr = PetscOptionsSetValue(NULL, "-options_left", NULL); > > petscErr = KSPSetFromOptions(ksp); > > > > Where is the KSPGetPC() call? > > > > Thanks, > > > > Matt > > > > // Set pc type > > petscErr = PCSetType(pc, "hypre"); > > > > // Set discrete gradient > > petscErr = PCHYPRESetDiscreteGradient(pc, par_G); > > > > // Set vertex coordinates > > petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, > par_zvec); > > > > > > *From: *Matthew Knepley > *Date: *Monday, 9 September 2024 at 19:46 > *To: *Chockalingam, Karthikeyan (STFC,DL,HC) < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *Stefano Zampini , petsc-users at mcs.anl.gov > > *Subject: *Re: [petsc-users] Hypre AMS usage > > On Mon, Sep 9, 2024 at 2:18?PM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > Calling the two functions after KSPSetFromOptions() did not work either. > > > > Can you check that the PC has the correct type when you call it? > > > > Thanks, > > > > Matt > > > > Everything works from the command line. > > I haven?t set KSPSetOperators, not sure if that is an issue. > > > > *From: *Matthew Knepley > *Date: *Monday, 9 September 2024 at 19:09 > *To: *Chockalingam, Karthikeyan (STFC,DL,HC) < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *Stefano Zampini , petsc-users at mcs.anl.gov > > *Subject: *Re: [petsc-users] Hypre AMS usage > > On Mon, Sep 9, 2024 at 1:21?PM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > Hi Stefano, > > > > Thank you. That was helpful. I tried the following: > > > > petscErr = PetscOptionsSetValue(NULL,"-ksp_type", "gmres"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_type", "hypre"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_type", "ams"); > > > > // Set discrete gradient > > petscErr = PCHYPRESetDiscreteGradient(pc, par_G); > > > > // Set vertex coordinates > > petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, > par_zvec); > > > > petscErr = PetscOptionsSetValue(NULL, "-ksp_view", NULL); > > petscErr = PetscOptionsSetValue(NULL, "-ksp_monitor_true_residual", > NULL); > > petscErr = PetscOptionsSetValue(NULL, "-ksp_converged_reason", NULL); > > petscErr = PetscOptionsSetValue(NULL, "-options_left", NULL); > > petscErr = KSPSetFromOptions(ksp); > > > > > > It errored though I have set PCHYPRESetEdgeConstantVectors. > > > > My guess is that the "pc" was not yet Hypre, since that type was not set > until KSPSetFromOptions() was called. So you need > > to call those two functions after that call. > > > > Thanks, > > > > Matt > > > > But my program works, without PetscOptionsSetValue and by passing > everything on the command line. > > > > *[0]PETSC ERROR: --------------------- Error Message > --------------------------------------------------------------* > > [0]PETSC ERROR: HYPRE AMS preconditioner needs either the coordinate > vectors via PCSetCoordinates() or the edge constant vectors via > PCHYPRESetEdgeConstantVectors() or the interpolation matrix via > PCHYPRESetInterpolations() > > [0]PETS C ERROR: WARNING! There are unused option(s) set! Could be the > program crashed before usage or a spelling mistake, etc! > > [0]PETSC ERROR: Option left: name:-ksp_converged_reason (no value) > source: code > > [0]PETSC ERROR: Option left: name:-options_left (no value) source: code > > [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!a3KatxLoaJS_7gFrJE3eLM-5OYXL5RWDjxI8DUxhWhHgbHxnV001g2y3V66HmFadvh7wZTxwCIxRQd2a--EC$ for trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.20.3, unknown > > [0]PETSC ERROR: ./example-dbg on a arch-moose named HC20210312 by > karthikeyan.chockalingam Mon Sep 9 18:10:13 2024 > > [0]PETSC ERROR: Configure options --with-64-bit-indices > --with-cxx-dialect=C++17 --with-debugging=no --with-fortran-bindings=0 > --with-mpi=1 --with-openmp=1 --with-shared-libraries=1 --with-sowing=0 > --download-fblaslapack=1 --download-hypre=1 --download-metis=1 > --download-mumps=1 --download-ptscotch=1 --download-parmetis=1 > --download-scalapack=1 --download-slepc=1 --download-strumpack=1 > --download-superlu_dist=1 --download-bison=1 --download-hdf5=1 > --with-hdf5-fortran-bindings=0 > --download-hdf5-configure-arguments="--with-zlib" --with-make-np=4 > > [0]PETSC ERROR: #1 PCSetUp_HYPRE() at > /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/pc/impls/hypre/hypre.c:295 > > [0]PETSC ERROR: #2 PCSetUp() at > /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/pc/interface/precon.c:1080 > > [0]PETSC ERROR: #3 KSPSetUp() at > /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/ksp/interface/itfunc.c:415 > > [0]PETSC ERROR: #4 KSPSolve_Private() at > /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/ksp/interface/itfunc.c:833 > > [0]PETSC ERROR: #5 KSPSolve() at > /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/ksp/interface/itfunc.c:1080 > > libMesh terminating: > > HYPRE AMS preconditioner needs either the coordinate vectors via > PCSetCoordinates() or the edge constant vectors via > PCHYPRESetEdgeConstantVectors() or the interpolation matrix via > PCHYPRESetInterpolations() > > > > > > *From: *Stefano Zampini > *Date: *Monday, 9 September 2024 at 17:02 > *To: *Matthew Knepley > *Cc: *Chockalingam, Karthikeyan (STFC,DL,HC) < > karthikeyan.chockalingam at stfc.ac.uk>, petsc-users at mcs.anl.gov < > petsc-users at mcs.anl.gov> > *Subject: *Re: [petsc-users] Hypre AMS usage > > I would say the best way is to look at the source code > > > > > https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/hypre/hypre.c?ref_type=heads*L2061__;Iw!!G_uCfscf7eWS!a3KatxLoaJS_7gFrJE3eLM-5OYXL5RWDjxI8DUxhWhHgbHxnV001g2y3V66HmFadvh7wZTxwCIxRQaqtNtFF$ > > > https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/hypre/hypre.c?ref_type=heads*L1239__;Iw!!G_uCfscf7eWS!a3KatxLoaJS_7gFrJE3eLM-5OYXL5RWDjxI8DUxhWhHgbHxnV001g2y3V66HmFadvh7wZTxwCIxRQRV5wMat$ > > > > Il giorno lun 9 set 2024 alle ore 18:50 Matthew Knepley > ha scritto: > > On Mon, Sep 9, 2024 at 10:17?AM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > Hi Matt, > > > > You mentioned it doesn?t hurt to set the smoothing flags > > > > > https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L965__;Iw!!G_uCfscf7eWS!a3KatxLoaJS_7gFrJE3eLM-5OYXL5RWDjxI8DUxhWhHgbHxnV001g2y3V66HmFadvh7wZTxwCIxRQTfXNe_z$ > > > Do you know where I can look for the equivalent PETSc commands? Thank you. > > > > Yes, this is described here: > https://urldefense.us/v3/__https://petsc.org/main/manualpages/PC/PCHYPRE/__;!!G_uCfscf7eWS!a3KatxLoaJS_7gFrJE3eLM-5OYXL5RWDjxI8DUxhWhHgbHxnV001g2y3V66HmFadvh7wZTxwCIxRQe1TTiy6$ > > > The best way is to run with -help or look at the source. > > > > Thanks, > > > > Matt > > > > > > Kind regards, > > Karthik. > > > > *From: *Matthew Knepley > *Date: *Friday, 6 September 2024 at 17:57 > *To: *Chockalingam, Karthikeyan (STFC,DL,HC) < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *petsc-users at mcs.anl.gov > *Subject: *Re: [petsc-users] Hypre AMS usage > > On Fri, Sep 6, 2024 at 11:37?AM Karthikeyan Chockalingam - STFC UKRI via > petsc-users wrote: > > Hello, > > > > I am trying to use the Hypre AMS preconditioner for the first time. > > > > I am following the example problem from Hypre > > > https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L954__;Iw!!G_uCfscf7eWS!a3KatxLoaJS_7gFrJE3eLM-5OYXL5RWDjxI8DUxhWhHgbHxnV001g2y3V66HmFadvh7wZTxwCIxRQV3ZXlVA$ > > > > > I have so far successfully set the discrete gradient operator and vertex > co-ordinates, > > > > // Set discrete gradient > > petscErr = PCHYPRESetDiscreteGradient(pc, par_G); > > > > // Set vertex coordinates > > petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, > par_zvec); > > > > Do I need to set the following smoothing options? > > > https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L965__;Iw!!G_uCfscf7eWS!a3KatxLoaJS_7gFrJE3eLM-5OYXL5RWDjxI8DUxhWhHgbHxnV001g2y3V66HmFadvh7wZTxwCIxRQTfXNe_z$ > > > > > It cannot hurt. I would set them to begin with. > > > > Also, do I need to convert from MATMPIAIJ to CSR? > > > https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L984__;Iw!!G_uCfscf7eWS!a3KatxLoaJS_7gFrJE3eLM-5OYXL5RWDjxI8DUxhWhHgbHxnV001g2y3V66HmFadvh7wZTxwCIxRQZM2zjIn$ > > > > > No. > > > > What are the other PETSc calls to invoke AMS? Is there an example problem > I can look at? > > > > I do not know. I don't think we have an example. > > > > Thanks, > > > > Matt > > > > Thank you. > > > > Karthik. > > > > -- > > *Karthik Chockalingam, Ph.D.* > > Senior Research Software Engineer > > High Performance Systems Engineering Group > > Hartree Centre | Science and Technology Facilities Council > > karthikeyan.chockalingam at stfc.ac.uk > > > > [image: signature_3970890138] > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!a3KatxLoaJS_7gFrJE3eLM-5OYXL5RWDjxI8DUxhWhHgbHxnV001g2y3V66HmFadvh7wZTxwCIxRQSYHgPtC$ > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!a3KatxLoaJS_7gFrJE3eLM-5OYXL5RWDjxI8DUxhWhHgbHxnV001g2y3V66HmFadvh7wZTxwCIxRQSYHgPtC$ > > > > > > -- > > Stefano > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!a3KatxLoaJS_7gFrJE3eLM-5OYXL5RWDjxI8DUxhWhHgbHxnV001g2y3V66HmFadvh7wZTxwCIxRQSYHgPtC$ > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!a3KatxLoaJS_7gFrJE3eLM-5OYXL5RWDjxI8DUxhWhHgbHxnV001g2y3V66HmFadvh7wZTxwCIxRQSYHgPtC$ > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!a3KatxLoaJS_7gFrJE3eLM-5OYXL5RWDjxI8DUxhWhHgbHxnV001g2y3V66HmFadvh7wZTxwCIxRQSYHgPtC$ > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!a3KatxLoaJS_7gFrJE3eLM-5OYXL5RWDjxI8DUxhWhHgbHxnV001g2y3V66HmFadvh7wZTxwCIxRQSYHgPtC$ > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!a3KatxLoaJS_7gFrJE3eLM-5OYXL5RWDjxI8DUxhWhHgbHxnV001g2y3V66HmFadvh7wZTxwCIxRQSYHgPtC$ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 18270 bytes Desc: not available URL: From C.Klaij at marin.nl Tue Sep 10 06:36:07 2024 From: C.Klaij at marin.nl (Klaij, Christiaan) Date: Tue, 10 Sep 2024 11:36:07 +0000 Subject: [petsc-users] how to visualize the matordering? Message-ID: I'm saving a petsc mat to file and loading it into python to make a spy plot of its sparsity pattern, so far so good. Now I would like to compare with a reordened pattern, say rcm. I've notice the option -mat_view_ordering draw, but do not have X Windows on this machine. What is the recommended way to get the reordened matrix for inspection? Chris dr. ir. Christiaan Klaij | Senior Researcher | Research & Development T +31 317 49 33 44 | C.Klaij at marin.nl | https://urldefense.us/v3/__http://www.marin.nl__;!!G_uCfscf7eWS!ZlpBqrxu1xOtAsOWM9zgbuOyULFV7FmZJfrJqgSnScyCr8z-tfyvuf3b2aXiy8qER3gQEm4Qu44oMudMz-xqPtY$ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image859244.png Type: image/png Size: 5004 bytes Desc: image859244.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image612326.png Type: image/png Size: 487 bytes Desc: image612326.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image769878.png Type: image/png Size: 504 bytes Desc: image769878.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image635531.png Type: image/png Size: 482 bytes Desc: image635531.png URL: From bsmith at petsc.dev Tue Sep 10 08:09:08 2024 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 10 Sep 2024 09:09:08 -0400 Subject: [petsc-users] how to visualize the matordering? In-Reply-To: References: Message-ID: You can save the plot in an image file that does not require X. For example -mat_view_ordering draw:image:filename Several image formats are available. Barry > On Sep 10, 2024, at 7:36?AM, Klaij, Christiaan wrote: > > I'm saving a petsc mat to file and loading it into python to make a spy plot of its sparsity pattern, so far so good. > > Now I would like to compare with a reordened pattern, say rcm. I've notice the option -mat_view_ordering draw, but do not have X Windows on this machine. What is the recommended way to get the reordened matrix for inspection? > > Chris > > dr. ir.???? Christiaan Klaij > | Senior Researcher | Research & Development > T +31?317?49?33?44 | C.Klaij at marin.nl | https://urldefense.us/v3/__http://www.marin.nl__;!!G_uCfscf7eWS!cB3QbQAW-a4JBnC6ienjVNxxG6Tu93rh2v5D0sKWoMilcEnEoHpZ5t6yP-XPhz9FiAGj8ULbp-EHk-ndDfGJ_wo$ > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From karthikeyan.chockalingam at stfc.ac.uk Tue Sep 10 12:16:53 2024 From: karthikeyan.chockalingam at stfc.ac.uk (Karthikeyan Chockalingam - STFC UKRI) Date: Tue, 10 Sep 2024 17:16:53 +0000 Subject: [petsc-users] Hypre AMS usage In-Reply-To: References: Message-ID: Hi Matt, I am not sure, if I understand how to read the source code, let?s take the below line PetscCall(PetscOptionsReal("-pc_hypre_ams_relax_weight", "Relaxation weight for AMS smoother", "None", jac->as_relax_weight, &jac->as_relax_weight, &flag3)) (Q1) Am I doing it right by setting as in the line below? How does the line below match up to the above? Since the above line has more arguments petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_weight", "1.0") (Q2) How do I go about setting the four parameters below using PetscOptionsSetValue? PetscCall(PetscOptionsIntArray("-pc_hypre_ams_amg_alpha_options", "AMG options for vector Poisson", "None", jac->as_amg_alpha_opts, &n, &flag2)); if (flag || flag2) { PetscCallExternal(HYPRE_AMSSetAlphaAMGOptions, jac->hsolver, jac->as_amg_alpha_opts[0], /* AMG coarsen type */ jac->as_amg_alpha_opts[1], /* AMG agg_levels */ jac->as_amg_alpha_opts[2], /* AMG relax_type */ jac->as_amg_alpha_theta, jac->as_amg_alpha_opts[3], /* AMG interp_type */ jac->as_amg_alpha_opts[4]); /* AMG Pmax */ } Kind regards, Karthik. From: Matthew Knepley Date: Monday, 9 September 2024 at 22:00 To: Chockalingam, Karthikeyan (STFC,DL,HC) Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Hypre AMS usage On Mon, Sep 9, 2024 at 4:32?PM Karthikeyan Chockalingam - STFC UKRI > wrote: Great. Thank you for letting me know. I got the reference to KSP as well from libmesh and I am not creating a new KSP. This time around, I didn?t have to set the PC type and it seems to work. Excellent. Thanks, Matt petscErr = PetscOptionsSetValue(NULL,"-ksp_type", "gmres"); petscErr = PetscOptionsSetValue(NULL,"-pc_type", "hypre"); petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_type", "ams"); petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_type", "2"); petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_times", "1"); petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_weight", "1.0"); petscErr = PetscOptionsSetValue(NULL, "-pc_hypre_ams_omega", "1.0"); petscErr = PetscOptionsSetValue(NULL, "-ksp_view", NULL); petscErr = PetscOptionsSetValue(NULL, "-ksp_monitor_true_residual", NULL); petscErr = PetscOptionsSetValue(NULL, "-ksp_converged_reason", NULL); petscErr = PetscOptionsSetValue(NULL, "-options_left", NULL); petscErr = KSPSetFromOptions(ksp); // Set discrete gradient petscErr = PCHYPRESetDiscreteGradient(pc, par_G); // Set vertex coordinates petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, par_zvec); From: Matthew Knepley > Date: Monday, 9 September 2024 at 21:11 To: Chockalingam, Karthikeyan (STFC,DL,HC) > Cc: Stefano Zampini >, petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Hypre AMS usage On Mon, Sep 9, 2024 at 4:03?PM Karthikeyan Chockalingam - STFC UKRI > wrote: I never explicitly called KSPGetPC(). I am embedding AMS in a libmesh (fem) example. So, I only got the reference to pc from libmesh and set petscErr = PCHYPRESetDiscreteGradient(pc, par_G); However, you are creating a _new_ KSP, so your PC will not match it. This is not doing what you want. Thanks, Matt From: Matthew Knepley > Date: Monday, 9 September 2024 at 20:57 To: Chockalingam, Karthikeyan (STFC,DL,HC) > Cc: Stefano Zampini >, petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Hypre AMS usage On Mon, Sep 9, 2024 at 3:38?PM Karthikeyan Chockalingam - STFC UKRI > wrote: I didn?t know how to check the pc type but adding petscErr = PCSetType(pc, "hypre"); before the two functions made it to work. But how does it work from the command line? For summary: KSPCreate(mesh.comm().get(), &ksp); petscErr = PetscOptionsSetValue(NULL,"-ksp_type", "gmres"); petscErr = PetscOptionsSetValue(NULL,"-pc_type", "hypre"); petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_type", "ams"); petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_type", "2"); petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_times", "1"); petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_weigh", "1.0"); petscErr = PetscOptionsSetValue(NULL, "-pc_hypre_ams_omega", "1.0"); petscErr = PetscOptionsSetValue(NULL, "-ksp_view", NULL); petscErr = PetscOptionsSetValue(NULL, "-ksp_monitor_true_residual", NULL); petscErr = PetscOptionsSetValue(NULL, "-ksp_converged_reason", NULL); petscErr = PetscOptionsSetValue(NULL, "-options_left", NULL); petscErr = KSPSetFromOptions(ksp); Where is the KSPGetPC() call? Thanks, Matt // Set pc type petscErr = PCSetType(pc, "hypre"); // Set discrete gradient petscErr = PCHYPRESetDiscreteGradient(pc, par_G); // Set vertex coordinates petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, par_zvec); From: Matthew Knepley > Date: Monday, 9 September 2024 at 19:46 To: Chockalingam, Karthikeyan (STFC,DL,HC) > Cc: Stefano Zampini >, petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Hypre AMS usage On Mon, Sep 9, 2024 at 2:18?PM Karthikeyan Chockalingam - STFC UKRI > wrote: Calling the two functions after KSPSetFromOptions() did not work either. Can you check that the PC has the correct type when you call it? Thanks, Matt Everything works from the command line. I haven?t set KSPSetOperators, not sure if that is an issue. From: Matthew Knepley > Date: Monday, 9 September 2024 at 19:09 To: Chockalingam, Karthikeyan (STFC,DL,HC) > Cc: Stefano Zampini >, petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Hypre AMS usage On Mon, Sep 9, 2024 at 1:21?PM Karthikeyan Chockalingam - STFC UKRI > wrote: Hi Stefano, Thank you. That was helpful. I tried the following: petscErr = PetscOptionsSetValue(NULL,"-ksp_type", "gmres"); petscErr = PetscOptionsSetValue(NULL,"-pc_type", "hypre"); petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_type", "ams"); // Set discrete gradient petscErr = PCHYPRESetDiscreteGradient(pc, par_G); // Set vertex coordinates petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, par_zvec); petscErr = PetscOptionsSetValue(NULL, "-ksp_view", NULL); petscErr = PetscOptionsSetValue(NULL, "-ksp_monitor_true_residual", NULL); petscErr = PetscOptionsSetValue(NULL, "-ksp_converged_reason", NULL); petscErr = PetscOptionsSetValue(NULL, "-options_left", NULL); petscErr = KSPSetFromOptions(ksp); It errored though I have set PCHYPRESetEdgeConstantVectors. My guess is that the "pc" was not yet Hypre, since that type was not set until KSPSetFromOptions() was called. So you need to call those two functions after that call. Thanks, Matt But my program works, without PetscOptionsSetValue and by passing everything on the command line. [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: HYPRE AMS preconditioner needs either the coordinate vectors via PCSetCoordinates() or the edge constant vectors via PCHYPRESetEdgeConstantVectors() or the interpolation matrix via PCHYPRESetInterpolations() [0]PETS C ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc! [0]PETSC ERROR: Option left: name:-ksp_converged_reason (no value) source: code [0]PETSC ERROR: Option left: name:-options_left (no value) source: code [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!YLWMpSx0H9uhaBMkNdt_SaKxHEab3a-U9xdt6315FLGVN37kf7VGm3fThdy1j1KCH5SPP8ynAhL3YjRHwz2e9puxcSU7dFRi6cIw$ for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.20.3, unknown [0]PETSC ERROR: ./example-dbg on a arch-moose named HC20210312 by karthikeyan.chockalingam Mon Sep 9 18:10:13 2024 [0]PETSC ERROR: Configure options --with-64-bit-indices --with-cxx-dialect=C++17 --with-debugging=no --with-fortran-bindings=0 --with-mpi=1 --with-openmp=1 --with-shared-libraries=1 --with-sowing=0 --download-fblaslapack=1 --download-hypre=1 --download-metis=1 --download-mumps=1 --download-ptscotch=1 --download-parmetis=1 --download-scalapack=1 --download-slepc=1 --download-strumpack=1 --download-superlu_dist=1 --download-bison=1 --download-hdf5=1 --with-hdf5-fortran-bindings=0 --download-hdf5-configure-arguments="--with-zlib" --with-make-np=4 [0]PETSC ERROR: #1 PCSetUp_HYPRE() at /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/pc/impls/hypre/hypre.c:295 [0]PETSC ERROR: #2 PCSetUp() at /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/pc/interface/precon.c:1080 [0]PETSC ERROR: #3 KSPSetUp() at /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/ksp/interface/itfunc.c:415 [0]PETSC ERROR: #4 KSPSolve_Private() at /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/ksp/interface/itfunc.c:833 [0]PETSC ERROR: #5 KSPSolve() at /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/ksp/interface/itfunc.c:1080 libMesh terminating: HYPRE AMS preconditioner needs either the coordinate vectors via PCSetCoordinates() or the edge constant vectors via PCHYPRESetEdgeConstantVectors() or the interpolation matrix via PCHYPRESetInterpolations() From: Stefano Zampini > Date: Monday, 9 September 2024 at 17:02 To: Matthew Knepley > Cc: Chockalingam, Karthikeyan (STFC,DL,HC) >, petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Hypre AMS usage I would say the best way is to look at the source code https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/hypre/hypre.c?ref_type=heads*L2061__;Iw!!G_uCfscf7eWS!YLWMpSx0H9uhaBMkNdt_SaKxHEab3a-U9xdt6315FLGVN37kf7VGm3fThdy1j1KCH5SPP8ynAhL3YjRHwz2e9puxcSU7dA_V2Zrh$ https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/hypre/hypre.c?ref_type=heads*L1239__;Iw!!G_uCfscf7eWS!YLWMpSx0H9uhaBMkNdt_SaKxHEab3a-U9xdt6315FLGVN37kf7VGm3fThdy1j1KCH5SPP8ynAhL3YjRHwz2e9puxcSU7dCD6gvmG$ Il giorno lun 9 set 2024 alle ore 18:50 Matthew Knepley > ha scritto: On Mon, Sep 9, 2024 at 10:17?AM Karthikeyan Chockalingam - STFC UKRI > wrote: Hi Matt, You mentioned it doesn?t hurt to set the smoothing flags https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L965__;Iw!!G_uCfscf7eWS!YLWMpSx0H9uhaBMkNdt_SaKxHEab3a-U9xdt6315FLGVN37kf7VGm3fThdy1j1KCH5SPP8ynAhL3YjRHwz2e9puxcSU7dA76B8UI$ Do you know where I can look for the equivalent PETSc commands? Thank you. Yes, this is described here: https://urldefense.us/v3/__https://petsc.org/main/manualpages/PC/PCHYPRE/__;!!G_uCfscf7eWS!YLWMpSx0H9uhaBMkNdt_SaKxHEab3a-U9xdt6315FLGVN37kf7VGm3fThdy1j1KCH5SPP8ynAhL3YjRHwz2e9puxcSU7dC7ch0OK$ The best way is to run with -help or look at the source. Thanks, Matt Kind regards, Karthik. From: Matthew Knepley > Date: Friday, 6 September 2024 at 17:57 To: Chockalingam, Karthikeyan (STFC,DL,HC) > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Hypre AMS usage On Fri, Sep 6, 2024 at 11:37?AM Karthikeyan Chockalingam - STFC UKRI via petsc-users > wrote: Hello, I am trying to use the Hypre AMS preconditioner for the first time. I am following the example problem from Hypre https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L954__;Iw!!G_uCfscf7eWS!YLWMpSx0H9uhaBMkNdt_SaKxHEab3a-U9xdt6315FLGVN37kf7VGm3fThdy1j1KCH5SPP8ynAhL3YjRHwz2e9puxcSU7dN3BQIpW$ I have so far successfully set the discrete gradient operator and vertex co-ordinates, // Set discrete gradient petscErr = PCHYPRESetDiscreteGradient(pc, par_G); // Set vertex coordinates petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, par_zvec); Do I need to set the following smoothing options? https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L965__;Iw!!G_uCfscf7eWS!YLWMpSx0H9uhaBMkNdt_SaKxHEab3a-U9xdt6315FLGVN37kf7VGm3fThdy1j1KCH5SPP8ynAhL3YjRHwz2e9puxcSU7dA76B8UI$ It cannot hurt. I would set them to begin with. Also, do I need to convert from MATMPIAIJ to CSR? https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L984__;Iw!!G_uCfscf7eWS!YLWMpSx0H9uhaBMkNdt_SaKxHEab3a-U9xdt6315FLGVN37kf7VGm3fThdy1j1KCH5SPP8ynAhL3YjRHwz2e9puxcSU7dGRl6keR$ No. What are the other PETSc calls to invoke AMS? Is there an example problem I can look at? I do not know. I don't think we have an example. Thanks, Matt Thank you. Karthik. -- Karthik Chockalingam, Ph.D. Senior Research Software Engineer High Performance Systems Engineering Group Hartree Centre | Science and Technology Facilities Council karthikeyan.chockalingam at stfc.ac.uk [signature_3970890138] -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YLWMpSx0H9uhaBMkNdt_SaKxHEab3a-U9xdt6315FLGVN37kf7VGm3fThdy1j1KCH5SPP8ynAhL3YjRHwz2e9puxcSU7dEmst4km$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YLWMpSx0H9uhaBMkNdt_SaKxHEab3a-U9xdt6315FLGVN37kf7VGm3fThdy1j1KCH5SPP8ynAhL3YjRHwz2e9puxcSU7dEmst4km$ -- Stefano -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YLWMpSx0H9uhaBMkNdt_SaKxHEab3a-U9xdt6315FLGVN37kf7VGm3fThdy1j1KCH5SPP8ynAhL3YjRHwz2e9puxcSU7dEmst4km$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YLWMpSx0H9uhaBMkNdt_SaKxHEab3a-U9xdt6315FLGVN37kf7VGm3fThdy1j1KCH5SPP8ynAhL3YjRHwz2e9puxcSU7dEmst4km$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YLWMpSx0H9uhaBMkNdt_SaKxHEab3a-U9xdt6315FLGVN37kf7VGm3fThdy1j1KCH5SPP8ynAhL3YjRHwz2e9puxcSU7dEmst4km$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YLWMpSx0H9uhaBMkNdt_SaKxHEab3a-U9xdt6315FLGVN37kf7VGm3fThdy1j1KCH5SPP8ynAhL3YjRHwz2e9puxcSU7dEmst4km$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YLWMpSx0H9uhaBMkNdt_SaKxHEab3a-U9xdt6315FLGVN37kf7VGm3fThdy1j1KCH5SPP8ynAhL3YjRHwz2e9puxcSU7dEmst4km$ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 18270 bytes Desc: image001.png URL: From knepley at gmail.com Tue Sep 10 13:22:23 2024 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 10 Sep 2024 14:22:23 -0400 Subject: [petsc-users] Hypre AMS usage In-Reply-To: References: Message-ID: On Tue, Sep 10, 2024 at 1:16?PM Karthikeyan Chockalingam - STFC UKRI < karthikeyan.chockalingam at stfc.ac.uk> wrote: > Hi Matt, > > > > I am not sure, if I understand how to read the source code, let?s take the > below line > > > > PetscCall(PetscOptionsReal("-pc_hypre_ams_relax_weight", "Relaxation > weight for AMS smoother", "None", jac->as_relax_weight, > &jac->as_relax_weight, &flag3)) > > (Q1) Am I doing it right by setting as in the line below? How does the > line below match up to the above? Since the above line has more arguments > > petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_weight", "1.0") > Yes. All options take a single string argument. The other arguments to the function help present it to the user (say through a GUI). > (Q2) How do I go about setting the four parameters below using > PetscOptionsSetValue? > It looks like you have 5 parameters, and you would call petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_amg_alpha_options", "10,21,32,43,54") Thanks, Matt > PetscCall(PetscOptionsIntArray("-pc_hypre_ams_amg_alpha_options", "AMG > options for vector Poisson", "None", jac->as_amg_alpha_opts, &n, &flag2)); > > if (flag || flag2) { > > PetscCallExternal(HYPRE_AMSSetAlphaAMGOptions, jac->hsolver, > jac->as_amg_alpha_opts[0], /* AMG coarsen type */ > > > jac->as_amg_alpha_opts[1], /* > AMG agg_levels */ > > jac->as_amg_alpha_opts[2], > /* AMG relax_type */ > > jac->as_amg_alpha_theta, > jac->as_amg_alpha_opts[3], /* AMG interp_type */ > > > jac->as_amg_alpha_opts[4]); /* > AMG Pmax */ > > } > > Kind regards, > > Karthik. > > > > > > > > *From: *Matthew Knepley > *Date: *Monday, 9 September 2024 at 22:00 > *To: *Chockalingam, Karthikeyan (STFC,DL,HC) < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *petsc-users at mcs.anl.gov > *Subject: *Re: [petsc-users] Hypre AMS usage > > On Mon, Sep 9, 2024 at 4:32?PM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > Great. Thank you for letting me know. > > I got the reference to KSP as well from libmesh and I am not creating a > new KSP. > > This time around, I didn?t have to set the PC type and it seems to work. > > > > Excellent. > > > > Thanks, > > > > Matt > > > > petscErr = PetscOptionsSetValue(NULL,"-ksp_type", "gmres"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_type", "hypre"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_type", "ams"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_type", "2"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_times", "1"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_weight", > "1.0"); > > petscErr = PetscOptionsSetValue(NULL, "-pc_hypre_ams_omega", "1.0"); > > petscErr = PetscOptionsSetValue(NULL, "-ksp_view", NULL); > > petscErr = PetscOptionsSetValue(NULL, "-ksp_monitor_true_residual", > NULL); > > petscErr = PetscOptionsSetValue(NULL, "-ksp_converged_reason", NULL); > > petscErr = PetscOptionsSetValue(NULL, "-options_left", NULL); > > petscErr = KSPSetFromOptions(ksp); > > > > // Set discrete gradient > > petscErr = PCHYPRESetDiscreteGradient(pc, par_G); > > > > // Set vertex coordinates > > petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, > par_zvec); > > > > > > *From: *Matthew Knepley > *Date: *Monday, 9 September 2024 at 21:11 > *To: *Chockalingam, Karthikeyan (STFC,DL,HC) < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *Stefano Zampini , petsc-users at mcs.anl.gov > > *Subject: *Re: [petsc-users] Hypre AMS usage > > On Mon, Sep 9, 2024 at 4:03?PM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > I never explicitly called KSPGetPC(). I am embedding AMS in a libmesh > (fem) example. So, I only got the reference to pc from libmesh and set > petscErr = PCHYPRESetDiscreteGradient(pc, par_G); > > > > However, you are creating a _new_ KSP, so your PC will not match it. This > is not doing what you want. > > > > Thanks, > > > > Matt > > > > > > *From: *Matthew Knepley > *Date: *Monday, 9 September 2024 at 20:57 > *To: *Chockalingam, Karthikeyan (STFC,DL,HC) < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *Stefano Zampini , petsc-users at mcs.anl.gov > > *Subject: *Re: [petsc-users] Hypre AMS usage > > On Mon, Sep 9, 2024 at 3:38?PM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > I didn?t know how to check the pc type but adding > > petscErr = PCSetType(pc, "hypre"); > > before the two functions made it to work. > > > > But how does it work from the command line? > > > > For summary: > > > > KSPCreate(mesh.comm().get(), &ksp); > > petscErr = PetscOptionsSetValue(NULL,"-ksp_type", "gmres"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_type", "hypre"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_type", "ams"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_type", "2"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_times", "1"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_weigh", "1.0"); > > petscErr = PetscOptionsSetValue(NULL, "-pc_hypre_ams_omega", "1.0"); > > petscErr = PetscOptionsSetValue(NULL, "-ksp_view", NULL); > > petscErr = PetscOptionsSetValue(NULL, "-ksp_monitor_true_residual", > NULL); > > petscErr = PetscOptionsSetValue(NULL, "-ksp_converged_reason", NULL); > > petscErr = PetscOptionsSetValue(NULL, "-options_left", NULL); > > petscErr = KSPSetFromOptions(ksp); > > > > Where is the KSPGetPC() call? > > > > Thanks, > > > > Matt > > > > // Set pc type > > petscErr = PCSetType(pc, "hypre"); > > > > // Set discrete gradient > > petscErr = PCHYPRESetDiscreteGradient(pc, par_G); > > > > // Set vertex coordinates > > petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, > par_zvec); > > > > > > *From: *Matthew Knepley > *Date: *Monday, 9 September 2024 at 19:46 > *To: *Chockalingam, Karthikeyan (STFC,DL,HC) < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *Stefano Zampini , petsc-users at mcs.anl.gov > > *Subject: *Re: [petsc-users] Hypre AMS usage > > On Mon, Sep 9, 2024 at 2:18?PM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > Calling the two functions after KSPSetFromOptions() did not work either. > > > > Can you check that the PC has the correct type when you call it? > > > > Thanks, > > > > Matt > > > > Everything works from the command line. > > I haven?t set KSPSetOperators, not sure if that is an issue. > > > > *From: *Matthew Knepley > *Date: *Monday, 9 September 2024 at 19:09 > *To: *Chockalingam, Karthikeyan (STFC,DL,HC) < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *Stefano Zampini , petsc-users at mcs.anl.gov > > *Subject: *Re: [petsc-users] Hypre AMS usage > > On Mon, Sep 9, 2024 at 1:21?PM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > Hi Stefano, > > > > Thank you. That was helpful. I tried the following: > > > > petscErr = PetscOptionsSetValue(NULL,"-ksp_type", "gmres"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_type", "hypre"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_type", "ams"); > > > > // Set discrete gradient > > petscErr = PCHYPRESetDiscreteGradient(pc, par_G); > > > > // Set vertex coordinates > > petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, > par_zvec); > > > > petscErr = PetscOptionsSetValue(NULL, "-ksp_view", NULL); > > petscErr = PetscOptionsSetValue(NULL, "-ksp_monitor_true_residual", > NULL); > > petscErr = PetscOptionsSetValue(NULL, "-ksp_converged_reason", NULL); > > petscErr = PetscOptionsSetValue(NULL, "-options_left", NULL); > > petscErr = KSPSetFromOptions(ksp); > > > > > > It errored though I have set PCHYPRESetEdgeConstantVectors. > > > > My guess is that the "pc" was not yet Hypre, since that type was not set > until KSPSetFromOptions() was called. So you need > > to call those two functions after that call. > > > > Thanks, > > > > Matt > > > > But my program works, without PetscOptionsSetValue and by passing > everything on the command line. > > > > *[0]PETSC ERROR: --------------------- Error Message > --------------------------------------------------------------* > > [0]PETSC ERROR: HYPRE AMS preconditioner needs either the coordinate > vectors via PCSetCoordinates() or the edge constant vectors via > PCHYPRESetEdgeConstantVectors() or the interpolation matrix via > PCHYPRESetInterpolations() > > [0]PETS C ERROR: WARNING! There are unused option(s) set! Could be the > program crashed before usage or a spelling mistake, etc! > > [0]PETSC ERROR: Option left: name:-ksp_converged_reason (no value) > source: code > > [0]PETSC ERROR: Option left: name:-options_left (no value) source: code > > [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!fzNdjKwlSE6S1482kG8uLzgyvhIjtNTWlib31rKGN57mOsU51yP6t9MkwWHKKx5J_BOV6XeIS4ibNFOiY6cO$ for trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.20.3, unknown > > [0]PETSC ERROR: ./example-dbg on a arch-moose named HC20210312 by > karthikeyan.chockalingam Mon Sep 9 18:10:13 2024 > > [0]PETSC ERROR: Configure options --with-64-bit-indices > --with-cxx-dialect=C++17 --with-debugging=no --with-fortran-bindings=0 > --with-mpi=1 --with-openmp=1 --with-shared-libraries=1 --with-sowing=0 > --download-fblaslapack=1 --download-hypre=1 --download-metis=1 > --download-mumps=1 --download-ptscotch=1 --download-parmetis=1 > --download-scalapack=1 --download-slepc=1 --download-strumpack=1 > --download-superlu_dist=1 --download-bison=1 --download-hdf5=1 > --with-hdf5-fortran-bindings=0 > --download-hdf5-configure-arguments="--with-zlib" --with-make-np=4 > > [0]PETSC ERROR: #1 PCSetUp_HYPRE() at > /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/pc/impls/hypre/hypre.c:295 > > [0]PETSC ERROR: #2 PCSetUp() at > /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/pc/interface/precon.c:1080 > > [0]PETSC ERROR: #3 KSPSetUp() at > /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/ksp/interface/itfunc.c:415 > > [0]PETSC ERROR: #4 KSPSolve_Private() at > /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/ksp/interface/itfunc.c:833 > > [0]PETSC ERROR: #5 KSPSolve() at > /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/ksp/interface/itfunc.c:1080 > > libMesh terminating: > > HYPRE AMS preconditioner needs either the coordinate vectors via > PCSetCoordinates() or the edge constant vectors via > PCHYPRESetEdgeConstantVectors() or the interpolation matrix via > PCHYPRESetInterpolations() > > > > > > *From: *Stefano Zampini > *Date: *Monday, 9 September 2024 at 17:02 > *To: *Matthew Knepley > *Cc: *Chockalingam, Karthikeyan (STFC,DL,HC) < > karthikeyan.chockalingam at stfc.ac.uk>, petsc-users at mcs.anl.gov < > petsc-users at mcs.anl.gov> > *Subject: *Re: [petsc-users] Hypre AMS usage > > I would say the best way is to look at the source code > > > > > https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/hypre/hypre.c?ref_type=heads*L2061__;Iw!!G_uCfscf7eWS!fzNdjKwlSE6S1482kG8uLzgyvhIjtNTWlib31rKGN57mOsU51yP6t9MkwWHKKx5J_BOV6XeIS4ibNPqq4coE$ > > > https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/hypre/hypre.c?ref_type=heads*L1239__;Iw!!G_uCfscf7eWS!fzNdjKwlSE6S1482kG8uLzgyvhIjtNTWlib31rKGN57mOsU51yP6t9MkwWHKKx5J_BOV6XeIS4ibNJC3V4RB$ > > > > Il giorno lun 9 set 2024 alle ore 18:50 Matthew Knepley > ha scritto: > > On Mon, Sep 9, 2024 at 10:17?AM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > Hi Matt, > > > > You mentioned it doesn?t hurt to set the smoothing flags > > > > > https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L965__;Iw!!G_uCfscf7eWS!fzNdjKwlSE6S1482kG8uLzgyvhIjtNTWlib31rKGN57mOsU51yP6t9MkwWHKKx5J_BOV6XeIS4ibNNfmla18$ > > > Do you know where I can look for the equivalent PETSc commands? Thank you. > > > > Yes, this is described here: > https://urldefense.us/v3/__https://petsc.org/main/manualpages/PC/PCHYPRE/__;!!G_uCfscf7eWS!fzNdjKwlSE6S1482kG8uLzgyvhIjtNTWlib31rKGN57mOsU51yP6t9MkwWHKKx5J_BOV6XeIS4ibNJpowz1n$ > > > The best way is to run with -help or look at the source. > > > > Thanks, > > > > Matt > > > > > > Kind regards, > > Karthik. > > > > *From: *Matthew Knepley > *Date: *Friday, 6 September 2024 at 17:57 > *To: *Chockalingam, Karthikeyan (STFC,DL,HC) < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *petsc-users at mcs.anl.gov > *Subject: *Re: [petsc-users] Hypre AMS usage > > On Fri, Sep 6, 2024 at 11:37?AM Karthikeyan Chockalingam - STFC UKRI via > petsc-users wrote: > > Hello, > > > > I am trying to use the Hypre AMS preconditioner for the first time. > > > > I am following the example problem from Hypre > > > https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L954__;Iw!!G_uCfscf7eWS!fzNdjKwlSE6S1482kG8uLzgyvhIjtNTWlib31rKGN57mOsU51yP6t9MkwWHKKx5J_BOV6XeIS4ibNNbhOKrb$ > > > > > I have so far successfully set the discrete gradient operator and vertex > co-ordinates, > > > > // Set discrete gradient > > petscErr = PCHYPRESetDiscreteGradient(pc, par_G); > > > > // Set vertex coordinates > > petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, > par_zvec); > > > > Do I need to set the following smoothing options? > > > https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L965__;Iw!!G_uCfscf7eWS!fzNdjKwlSE6S1482kG8uLzgyvhIjtNTWlib31rKGN57mOsU51yP6t9MkwWHKKx5J_BOV6XeIS4ibNNfmla18$ > > > > > It cannot hurt. I would set them to begin with. > > > > Also, do I need to convert from MATMPIAIJ to CSR? > > > https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L984__;Iw!!G_uCfscf7eWS!fzNdjKwlSE6S1482kG8uLzgyvhIjtNTWlib31rKGN57mOsU51yP6t9MkwWHKKx5J_BOV6XeIS4ibNEr3SNHH$ > > > > > No. > > > > What are the other PETSc calls to invoke AMS? Is there an example problem > I can look at? > > > > I do not know. I don't think we have an example. > > > > Thanks, > > > > Matt > > > > Thank you. > > > > Karthik. > > > > -- > > *Karthik Chockalingam, Ph.D.* > > Senior Research Software Engineer > > High Performance Systems Engineering Group > > Hartree Centre | Science and Technology Facilities Council > > karthikeyan.chockalingam at stfc.ac.uk > > > > [image: signature_3970890138] > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fzNdjKwlSE6S1482kG8uLzgyvhIjtNTWlib31rKGN57mOsU51yP6t9MkwWHKKx5J_BOV6XeIS4ibNHy1GP_5$ > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fzNdjKwlSE6S1482kG8uLzgyvhIjtNTWlib31rKGN57mOsU51yP6t9MkwWHKKx5J_BOV6XeIS4ibNHy1GP_5$ > > > > > > -- > > Stefano > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fzNdjKwlSE6S1482kG8uLzgyvhIjtNTWlib31rKGN57mOsU51yP6t9MkwWHKKx5J_BOV6XeIS4ibNHy1GP_5$ > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fzNdjKwlSE6S1482kG8uLzgyvhIjtNTWlib31rKGN57mOsU51yP6t9MkwWHKKx5J_BOV6XeIS4ibNHy1GP_5$ > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fzNdjKwlSE6S1482kG8uLzgyvhIjtNTWlib31rKGN57mOsU51yP6t9MkwWHKKx5J_BOV6XeIS4ibNHy1GP_5$ > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fzNdjKwlSE6S1482kG8uLzgyvhIjtNTWlib31rKGN57mOsU51yP6t9MkwWHKKx5J_BOV6XeIS4ibNHy1GP_5$ > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fzNdjKwlSE6S1482kG8uLzgyvhIjtNTWlib31rKGN57mOsU51yP6t9MkwWHKKx5J_BOV6XeIS4ibNHy1GP_5$ > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fzNdjKwlSE6S1482kG8uLzgyvhIjtNTWlib31rKGN57mOsU51yP6t9MkwWHKKx5J_BOV6XeIS4ibNHy1GP_5$ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 18270 bytes Desc: not available URL: From liufield at gmail.com Wed Sep 11 05:03:52 2024 From: liufield at gmail.com (neil liu) Date: Wed, 11 Sep 2024 06:03:52 -0400 Subject: [petsc-users] Inquiry about metric-based adaptation using MMG in petsc Message-ID: Dear Petsc developers and users, I am trying to explore adaptive mesh refinement (tetradedra) with Petsc. It seems case 12 ($PETSC DIR/src/snes/tutorials/ex12.c, from paper, https://urldefense.us/v3/__https://arxiv.org/pdf/2201.02806__;!!G_uCfscf7eWS!ZEwaHkvYc_IrjycFr8LiNi-nZcFqu0ZpsydAAomxptQ6C0xkBs3qhn5ba31Z4vipKf4mTrqDm8A5S65DDD_xFg$ ) is a good example. However when I tried, ./ex12 -run_type full -dm_plex_dim 3 -dm_distribute -dm_plex_box_faces 10,10,10 -bc_type dirichlet -petscspace_degree 1 -variable_coefficient ball -snes_converged _reason ::ascii_info_detail -ksp_type cg -pc_type sor -snes_adapt_sequence 3 -adaptor_target_num 1000 -dm_plex_metric_h_max 0.5 -dm_adaptor mmg L_2 Error: 1.55486 Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE iterations 2 it shows the following error, [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Argument out of range [0]PETSC ERROR: Global metric normalization factor must be in (0, inf). Is the input metric positive-definite? Do you have any suggestions here? Thanks, Xiaodong -------------- next part -------------- An HTML attachment was scrubbed... URL: From C.Klaij at marin.nl Fri Sep 13 01:39:56 2024 From: C.Klaij at marin.nl (Klaij, Christiaan) Date: Fri, 13 Sep 2024 06:39:56 +0000 Subject: [petsc-users] how to visualize the matordering? In-Reply-To: References: Message-ID: Thanks Barry, that works fine. Any chance to access the reordered matrix behind the plot? Chris ________________________________________ From: Barry Smith Sent: Tuesday, September 10, 2024 3:09 PM To: Klaij, Christiaan Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] how to visualize the matordering? You don't often get email from bsmith at petsc.dev. Learn why this is important You can save the plot in an image file that does not require X. For example -mat_view_ordering draw:image:filename Several image formats are available. Barry On Sep 10, 2024, at 7:36?AM, Klaij, Christiaan wrote: I'm saving a petsc mat to file and loading it into python to make a spy plot of its sparsity pattern, so far so good. Now I would like to compare with a reordened pattern, say rcm. I've notice the option -mat_view_ordering draw, but do not have X Windows on this machine. What is the recommended way to get the reordened matrix for inspection? Chris dr. ir.???? Christiaan Klaij | Senior Researcher | Research & Development T +31 317 49 33 44 | C.Klaij at marin.nl | https://urldefense.us/v3/__http://www.marin.nl__;!!G_uCfscf7eWS!fbUhaFTo3vSK-SuD9ZJ8eCD-wcGaA2dp_iFxbwerHHOlCS7JVWRhadkbjdVxrrInqzkq9zYDYfR4UOeiAGVa6nA$ From knepley at gmail.com Fri Sep 13 06:34:01 2024 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 13 Sep 2024 07:34:01 -0400 Subject: [petsc-users] how to visualize the matordering? In-Reply-To: References: Message-ID: On Fri, Sep 13, 2024 at 2:40?AM Klaij, Christiaan wrote: > Thanks Barry, that works fine. Any chance to access the reordered matrix > behind the plot? > You can make it with MatPermute(). Thanks, Matt > Chris > > ________________________________________ > From: Barry Smith > Sent: Tuesday, September 10, 2024 3:09 PM > To: Klaij, Christiaan > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] how to visualize the matordering? > > You don't often get email from bsmith at petsc.dev. Learn why this is > important< > https://urldefense.us/v3/__https://aka.ms/LearnAboutSenderIdentification__;!!G_uCfscf7eWS!fbUhaFTo3vSK-SuD9ZJ8eCD-wcGaA2dp_iFxbwerHHOlCS7JVWRhadkbjdVxrrInqzkq9zYDYfR4UOeiWk_xAmk$ > > > > You can save the plot in an image file that does not require X. For > example > > -mat_view_ordering draw:image:filename > > Several image formats are available. > > > Barry > > > On Sep 10, 2024, at 7:36?AM, Klaij, Christiaan wrote: > > I'm saving a petsc mat to file and loading it into python to make a spy > plot of its sparsity pattern, so far so good. > > Now I would like to compare with a reordened pattern, say rcm. I've notice > the option -mat_view_ordering draw, but do not have X Windows on this > machine. What is the recommended way to get the reordened matrix for > inspection? > > Chris > > dr. ir. Christiaan Klaij > | Senior Researcher | Research & Development > T +31 317 49 33 44 | > C.Klaij at marin.nl | > https://urldefense.us/v3/__http://www.marin.nl__;!!G_uCfscf7eWS!fbUhaFTo3vSK-SuD9ZJ8eCD-wcGaA2dp_iFxbwerHHOlCS7JVWRhadkbjdVxrrInqzkq9zYDYfR4UOeiAGVa6nA$ > < > https://urldefense.us/v3/__https://www.marin.nl/__;!!G_uCfscf7eWS!ZlpBqrxu1xOtAsOWM9zgbuOyULFV7FmZJfrJqgSnScyCr8z-tfyvuf3b2aXiy8qER3gQEm4Qu44oMudMb3ZvYBA$ > > > < > https://urldefense.us/v3/__https://www.facebook.com/marin.wageningen__;!!G_uCfscf7eWS!ZlpBqrxu1xOtAsOWM9zgbuOyULFV7FmZJfrJqgSnScyCr8z-tfyvuf3b2aXiy8qER3gQEm4Qu44oMudMrq6nsrQ$ > > > < > https://urldefense.us/v3/__https://www.linkedin.com/company/marin__;!!G_uCfscf7eWS!ZlpBqrxu1xOtAsOWM9zgbuOyULFV7FmZJfrJqgSnScyCr8z-tfyvuf3b2aXiy8qER3gQEm4Qu44oMudMuNr3Ucw$ > > > < > https://urldefense.us/v3/__https://www.youtube.com/marinmultimedia__;!!G_uCfscf7eWS!ZlpBqrxu1xOtAsOWM9zgbuOyULFV7FmZJfrJqgSnScyCr8z-tfyvuf3b2aXiy8qER3gQEm4Qu44oMudMadib6bE$ > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cqVEpFXzlZDy9mxC3QVCfvcTsImy_ZZq-sc3ewVbiGF-PD6JhuYL6Yonrh06ghc5fTXhH0dOYpci2Sb8_vJr$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Fri Sep 13 13:22:54 2024 From: jed at jedbrown.org (Jed Brown) Date: Fri, 13 Sep 2024 12:22:54 -0600 Subject: [petsc-users] Inquiry about metric-based adaptation using MMG in petsc In-Reply-To: References: Message-ID: <87o74rfm69.fsf@jedbrown.org> This error message is coming from the following: $ rg -B4 'Global metric normalization factor must be in' src/dm/impls/plex/plexmetric.c 1308- PetscCall(PetscDSSetObjective(ds, 0, detMFunc)); 1309- PetscCall(DMPlexComputeIntegralFEM(dmDet, determinant, &integral, NULL)); 1310- } 1311- realIntegral = PetscRealPart(integral); 1312: PetscCheck(realIntegral > 1.0e-30, PETSC_COMM_SELF, PETSC_ERR_ARG_OUTOFRANGE, "Global metric normalization factor must be in (0, inf). Is the input metric positive-definite?"); Perhaps you can independently check what integrand is being provided. It's probably zero or negative. You could apply this patch so the error message will report a value. diff --git i/src/dm/impls/plex/plexmetric.c w/src/dm/impls/plex/plexmetric.c index 61caeed28de..906cb394027 100644 --- i/src/dm/impls/plex/plexmetric.c +++ w/src/dm/impls/plex/plexmetric.c @@ -1309,7 +1309,7 @@ PetscErrorCode DMPlexMetricNormalize(DM dm, Vec metricIn, PetscBool restrictSize PetscCall(DMPlexComputeIntegralFEM(dmDet, determinant, &integral, NULL)); } realIntegral = PetscRealPart(integral); - PetscCheck(realIntegral > 1.0e-30, PETSC_COMM_SELF, PETSC_ERR_ARG_OUTOFRANGE, "Global metric normalization factor must be in (0, inf). Is the input metric positive-definite?"); + PetscCheck(realIntegral > 1.0e-30, PETSC_COMM_SELF, PETSC_ERR_ARG_OUTOFRANGE, "Global metric normalization factor %g must be in (0, inf). Is the input metric positive-definite?", (double)realIntegral); factGlob = PetscPowReal(target / realIntegral, 2.0 / dim); /* Apply local scaling */ neil liu writes: > Dear Petsc developers and users, > > I am trying to explore adaptive mesh refinement (tetradedra) with Petsc. > It seems case 12 ($PETSC DIR/src/snes/tutorials/ex12.c, from paper, > https://urldefense.us/v3/__https://arxiv.org/pdf/2201.02806__;!!G_uCfscf7eWS!ZEwaHkvYc_IrjycFr8LiNi-nZcFqu0ZpsydAAomxptQ6C0xkBs3qhn5ba31Z4vipKf4mTrqDm8A5S65DDD_xFg$ ) is a good example. > However when I tried, > ./ex12 -run_type full -dm_plex_dim 3 -dm_distribute -dm_plex_box_faces > 10,10,10 -bc_type dirichlet -petscspace_degree 1 -variable_coefficient ball > -snes_converged > _reason ::ascii_info_detail -ksp_type cg -pc_type sor -snes_adapt_sequence > 3 -adaptor_target_num 1000 -dm_plex_metric_h_max 0.5 -dm_adaptor mmg > L_2 Error: 1.55486 > Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE iterations 2 > > it shows the following error, > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Argument out of range > [0]PETSC ERROR: Global metric normalization factor must be in (0, inf). Is > the input metric positive-definite? > > Do you have any suggestions here? > > Thanks, > > Xiaodong From karthikeyan.chockalingam at stfc.ac.uk Fri Sep 13 15:56:10 2024 From: karthikeyan.chockalingam at stfc.ac.uk (Karthikeyan Chockalingam - STFC UKRI) Date: Fri, 13 Sep 2024 20:56:10 +0000 Subject: [petsc-users] Hypre AMS usage In-Reply-To: References: Message-ID: I would like to call HYPRE_AMSSetBetaPoissonMatrix from Hypre https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/hypre/hypre.c?ref_type=heads*L325__;Iw!!G_uCfscf7eWS!bIl8jeOv0ElEZcZXJyokC5HmHRy0GFJNcnG8ExyfYDjdBSaxJWWWHin2XKHiY7DpmldbOK-4mpaY8_vj_P5iyuvRStY3ovLAZK_D$ https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/hypre/hypre.c?ref_type=heads*L329__;Iw!!G_uCfscf7eWS!bIl8jeOv0ElEZcZXJyokC5HmHRy0GFJNcnG8ExyfYDjdBSaxJWWWHin2XKHiY7DpmldbOK-4mpaY8_vj_P5iyuvRStY3opNPSwe9$ Look like ams_beta_is_zero has to be true So would I end up calling PCHYPRESetPoissonMatrix_HYPRE with isalpha being false? https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/hypre/hypre.c?ref_type=heads*L1640__;Iw!!G_uCfscf7eWS!bIl8jeOv0ElEZcZXJyokC5HmHRy0GFJNcnG8ExyfYDjdBSaxJWWWHin2XKHiY7DpmldbOK-4mpaY8_vj_P5iyuvRStY3olhlOhKr$ Kind regards, Karthik. From: Matthew Knepley Date: Tuesday, 10 September 2024 at 19:22 To: Chockalingam, Karthikeyan (STFC,DL,HC) Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Hypre AMS usage On Tue, Sep 10, 2024 at 1:16?PM Karthikeyan Chockalingam - STFC UKRI > wrote: Hi Matt, I am not sure, if I understand how to read the source code, let?s take the below line PetscCall(PetscOptionsReal("-pc_hypre_ams_relax_weight", "Relaxation weight for AMS smoother", "None", jac->as_relax_weight, &jac->as_relax_weight, &flag3)) (Q1) Am I doing it right by setting as in the line below? How does the line below match up to the above? Since the above line has more arguments petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_weight", "1.0") Yes. All options take a single string argument. The other arguments to the function help present it to the user (say through a GUI). (Q2) How do I go about setting the four parameters below using PetscOptionsSetValue? It looks like you have 5 parameters, and you would call petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_amg_alpha_options", "10,21,32,43,54") Thanks, Matt PetscCall(PetscOptionsIntArray("-pc_hypre_ams_amg_alpha_options", "AMG options for vector Poisson", "None", jac->as_amg_alpha_opts, &n, &flag2)); if (flag || flag2) { PetscCallExternal(HYPRE_AMSSetAlphaAMGOptions, jac->hsolver, jac->as_amg_alpha_opts[0], /* AMG coarsen type */ jac->as_amg_alpha_opts[1], /* AMG agg_levels */ jac->as_amg_alpha_opts[2], /* AMG relax_type */ jac->as_amg_alpha_theta, jac->as_amg_alpha_opts[3], /* AMG interp_type */ jac->as_amg_alpha_opts[4]); /* AMG Pmax */ } Kind regards, Karthik. From: Matthew Knepley > Date: Monday, 9 September 2024 at 22:00 To: Chockalingam, Karthikeyan (STFC,DL,HC) > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Hypre AMS usage On Mon, Sep 9, 2024 at 4:32?PM Karthikeyan Chockalingam - STFC UKRI > wrote: Great. Thank you for letting me know. I got the reference to KSP as well from libmesh and I am not creating a new KSP. This time around, I didn?t have to set the PC type and it seems to work. Excellent. Thanks, Matt petscErr = PetscOptionsSetValue(NULL,"-ksp_type", "gmres"); petscErr = PetscOptionsSetValue(NULL,"-pc_type", "hypre"); petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_type", "ams"); petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_type", "2"); petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_times", "1"); petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_weight", "1.0"); petscErr = PetscOptionsSetValue(NULL, "-pc_hypre_ams_omega", "1.0"); petscErr = PetscOptionsSetValue(NULL, "-ksp_view", NULL); petscErr = PetscOptionsSetValue(NULL, "-ksp_monitor_true_residual", NULL); petscErr = PetscOptionsSetValue(NULL, "-ksp_converged_reason", NULL); petscErr = PetscOptionsSetValue(NULL, "-options_left", NULL); petscErr = KSPSetFromOptions(ksp); // Set discrete gradient petscErr = PCHYPRESetDiscreteGradient(pc, par_G); // Set vertex coordinates petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, par_zvec); From: Matthew Knepley > Date: Monday, 9 September 2024 at 21:11 To: Chockalingam, Karthikeyan (STFC,DL,HC) > Cc: Stefano Zampini >, petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Hypre AMS usage On Mon, Sep 9, 2024 at 4:03?PM Karthikeyan Chockalingam - STFC UKRI > wrote: I never explicitly called KSPGetPC(). I am embedding AMS in a libmesh (fem) example. So, I only got the reference to pc from libmesh and set petscErr = PCHYPRESetDiscreteGradient(pc, par_G); However, you are creating a _new_ KSP, so your PC will not match it. This is not doing what you want. Thanks, Matt From: Matthew Knepley > Date: Monday, 9 September 2024 at 20:57 To: Chockalingam, Karthikeyan (STFC,DL,HC) > Cc: Stefano Zampini >, petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Hypre AMS usage On Mon, Sep 9, 2024 at 3:38?PM Karthikeyan Chockalingam - STFC UKRI > wrote: I didn?t know how to check the pc type but adding petscErr = PCSetType(pc, "hypre"); before the two functions made it to work. But how does it work from the command line? For summary: KSPCreate(mesh.comm().get(), &ksp); petscErr = PetscOptionsSetValue(NULL,"-ksp_type", "gmres"); petscErr = PetscOptionsSetValue(NULL,"-pc_type", "hypre"); petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_type", "ams"); petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_type", "2"); petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_times", "1"); petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_weigh", "1.0"); petscErr = PetscOptionsSetValue(NULL, "-pc_hypre_ams_omega", "1.0"); petscErr = PetscOptionsSetValue(NULL, "-ksp_view", NULL); petscErr = PetscOptionsSetValue(NULL, "-ksp_monitor_true_residual", NULL); petscErr = PetscOptionsSetValue(NULL, "-ksp_converged_reason", NULL); petscErr = PetscOptionsSetValue(NULL, "-options_left", NULL); petscErr = KSPSetFromOptions(ksp); Where is the KSPGetPC() call? Thanks, Matt // Set pc type petscErr = PCSetType(pc, "hypre"); // Set discrete gradient petscErr = PCHYPRESetDiscreteGradient(pc, par_G); // Set vertex coordinates petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, par_zvec); From: Matthew Knepley > Date: Monday, 9 September 2024 at 19:46 To: Chockalingam, Karthikeyan (STFC,DL,HC) > Cc: Stefano Zampini >, petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Hypre AMS usage On Mon, Sep 9, 2024 at 2:18?PM Karthikeyan Chockalingam - STFC UKRI > wrote: Calling the two functions after KSPSetFromOptions() did not work either. Can you check that the PC has the correct type when you call it? Thanks, Matt Everything works from the command line. I haven?t set KSPSetOperators, not sure if that is an issue. From: Matthew Knepley > Date: Monday, 9 September 2024 at 19:09 To: Chockalingam, Karthikeyan (STFC,DL,HC) > Cc: Stefano Zampini >, petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Hypre AMS usage On Mon, Sep 9, 2024 at 1:21?PM Karthikeyan Chockalingam - STFC UKRI > wrote: Hi Stefano, Thank you. That was helpful. I tried the following: petscErr = PetscOptionsSetValue(NULL,"-ksp_type", "gmres"); petscErr = PetscOptionsSetValue(NULL,"-pc_type", "hypre"); petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_type", "ams"); // Set discrete gradient petscErr = PCHYPRESetDiscreteGradient(pc, par_G); // Set vertex coordinates petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, par_zvec); petscErr = PetscOptionsSetValue(NULL, "-ksp_view", NULL); petscErr = PetscOptionsSetValue(NULL, "-ksp_monitor_true_residual", NULL); petscErr = PetscOptionsSetValue(NULL, "-ksp_converged_reason", NULL); petscErr = PetscOptionsSetValue(NULL, "-options_left", NULL); petscErr = KSPSetFromOptions(ksp); It errored though I have set PCHYPRESetEdgeConstantVectors. My guess is that the "pc" was not yet Hypre, since that type was not set until KSPSetFromOptions() was called. So you need to call those two functions after that call. Thanks, Matt But my program works, without PetscOptionsSetValue and by passing everything on the command line. [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: HYPRE AMS preconditioner needs either the coordinate vectors via PCSetCoordinates() or the edge constant vectors via PCHYPRESetEdgeConstantVectors() or the interpolation matrix via PCHYPRESetInterpolations() [0]PETS C ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc! [0]PETSC ERROR: Option left: name:-ksp_converged_reason (no value) source: code [0]PETSC ERROR: Option left: name:-options_left (no value) source: code [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!bIl8jeOv0ElEZcZXJyokC5HmHRy0GFJNcnG8ExyfYDjdBSaxJWWWHin2XKHiY7DpmldbOK-4mpaY8_vj_P5iyuvRStY3oiqi8BDe$ for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.20.3, unknown [0]PETSC ERROR: ./example-dbg on a arch-moose named HC20210312 by karthikeyan.chockalingam Mon Sep 9 18:10:13 2024 [0]PETSC ERROR: Configure options --with-64-bit-indices --with-cxx-dialect=C++17 --with-debugging=no --with-fortran-bindings=0 --with-mpi=1 --with-openmp=1 --with-shared-libraries=1 --with-sowing=0 --download-fblaslapack=1 --download-hypre=1 --download-metis=1 --download-mumps=1 --download-ptscotch=1 --download-parmetis=1 --download-scalapack=1 --download-slepc=1 --download-strumpack=1 --download-superlu_dist=1 --download-bison=1 --download-hdf5=1 --with-hdf5-fortran-bindings=0 --download-hdf5-configure-arguments="--with-zlib" --with-make-np=4 [0]PETSC ERROR: #1 PCSetUp_HYPRE() at /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/pc/impls/hypre/hypre.c:295 [0]PETSC ERROR: #2 PCSetUp() at /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/pc/interface/precon.c:1080 [0]PETSC ERROR: #3 KSPSetUp() at /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/ksp/interface/itfunc.c:415 [0]PETSC ERROR: #4 KSPSolve_Private() at /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/ksp/interface/itfunc.c:833 [0]PETSC ERROR: #5 KSPSolve() at /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/ksp/interface/itfunc.c:1080 libMesh terminating: HYPRE AMS preconditioner needs either the coordinate vectors via PCSetCoordinates() or the edge constant vectors via PCHYPRESetEdgeConstantVectors() or the interpolation matrix via PCHYPRESetInterpolations() From: Stefano Zampini > Date: Monday, 9 September 2024 at 17:02 To: Matthew Knepley > Cc: Chockalingam, Karthikeyan (STFC,DL,HC) >, petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Hypre AMS usage I would say the best way is to look at the source code https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/hypre/hypre.c?ref_type=heads*L2061__;Iw!!G_uCfscf7eWS!bIl8jeOv0ElEZcZXJyokC5HmHRy0GFJNcnG8ExyfYDjdBSaxJWWWHin2XKHiY7DpmldbOK-4mpaY8_vj_P5iyuvRStY3ouf-twyh$ https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/hypre/hypre.c?ref_type=heads*L1239__;Iw!!G_uCfscf7eWS!bIl8jeOv0ElEZcZXJyokC5HmHRy0GFJNcnG8ExyfYDjdBSaxJWWWHin2XKHiY7DpmldbOK-4mpaY8_vj_P5iyuvRStY3ol7J1lmL$ Il giorno lun 9 set 2024 alle ore 18:50 Matthew Knepley > ha scritto: On Mon, Sep 9, 2024 at 10:17?AM Karthikeyan Chockalingam - STFC UKRI > wrote: Hi Matt, You mentioned it doesn?t hurt to set the smoothing flags https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L965__;Iw!!G_uCfscf7eWS!bIl8jeOv0ElEZcZXJyokC5HmHRy0GFJNcnG8ExyfYDjdBSaxJWWWHin2XKHiY7DpmldbOK-4mpaY8_vj_P5iyuvRStY3ohk1YJ2e$ Do you know where I can look for the equivalent PETSc commands? Thank you. Yes, this is described here: https://urldefense.us/v3/__https://petsc.org/main/manualpages/PC/PCHYPRE/__;!!G_uCfscf7eWS!bIl8jeOv0ElEZcZXJyokC5HmHRy0GFJNcnG8ExyfYDjdBSaxJWWWHin2XKHiY7DpmldbOK-4mpaY8_vj_P5iyuvRStY3otbGLpuh$ The best way is to run with -help or look at the source. Thanks, Matt Kind regards, Karthik. From: Matthew Knepley > Date: Friday, 6 September 2024 at 17:57 To: Chockalingam, Karthikeyan (STFC,DL,HC) > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Hypre AMS usage On Fri, Sep 6, 2024 at 11:37?AM Karthikeyan Chockalingam - STFC UKRI via petsc-users > wrote: Hello, I am trying to use the Hypre AMS preconditioner for the first time. I am following the example problem from Hypre https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L954__;Iw!!G_uCfscf7eWS!bIl8jeOv0ElEZcZXJyokC5HmHRy0GFJNcnG8ExyfYDjdBSaxJWWWHin2XKHiY7DpmldbOK-4mpaY8_vj_P5iyuvRStY3otpKxOId$ I have so far successfully set the discrete gradient operator and vertex co-ordinates, // Set discrete gradient petscErr = PCHYPRESetDiscreteGradient(pc, par_G); // Set vertex coordinates petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, par_zvec); Do I need to set the following smoothing options? https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L965__;Iw!!G_uCfscf7eWS!bIl8jeOv0ElEZcZXJyokC5HmHRy0GFJNcnG8ExyfYDjdBSaxJWWWHin2XKHiY7DpmldbOK-4mpaY8_vj_P5iyuvRStY3ohk1YJ2e$ It cannot hurt. I would set them to begin with. Also, do I need to convert from MATMPIAIJ to CSR? https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L984__;Iw!!G_uCfscf7eWS!bIl8jeOv0ElEZcZXJyokC5HmHRy0GFJNcnG8ExyfYDjdBSaxJWWWHin2XKHiY7DpmldbOK-4mpaY8_vj_P5iyuvRStY3olMmf5mz$ No. What are the other PETSc calls to invoke AMS? Is there an example problem I can look at? I do not know. I don't think we have an example. Thanks, Matt Thank you. Karthik. -- Karthik Chockalingam, Ph.D. Senior Research Software Engineer High Performance Systems Engineering Group Hartree Centre | Science and Technology Facilities Council karthikeyan.chockalingam at stfc.ac.uk [signature_3970890138] -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bIl8jeOv0ElEZcZXJyokC5HmHRy0GFJNcnG8ExyfYDjdBSaxJWWWHin2XKHiY7DpmldbOK-4mpaY8_vj_P5iyuvRStY3oiGwv7RL$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bIl8jeOv0ElEZcZXJyokC5HmHRy0GFJNcnG8ExyfYDjdBSaxJWWWHin2XKHiY7DpmldbOK-4mpaY8_vj_P5iyuvRStY3oiGwv7RL$ -- Stefano -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bIl8jeOv0ElEZcZXJyokC5HmHRy0GFJNcnG8ExyfYDjdBSaxJWWWHin2XKHiY7DpmldbOK-4mpaY8_vj_P5iyuvRStY3oiGwv7RL$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bIl8jeOv0ElEZcZXJyokC5HmHRy0GFJNcnG8ExyfYDjdBSaxJWWWHin2XKHiY7DpmldbOK-4mpaY8_vj_P5iyuvRStY3oiGwv7RL$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bIl8jeOv0ElEZcZXJyokC5HmHRy0GFJNcnG8ExyfYDjdBSaxJWWWHin2XKHiY7DpmldbOK-4mpaY8_vj_P5iyuvRStY3oiGwv7RL$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bIl8jeOv0ElEZcZXJyokC5HmHRy0GFJNcnG8ExyfYDjdBSaxJWWWHin2XKHiY7DpmldbOK-4mpaY8_vj_P5iyuvRStY3oiGwv7RL$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bIl8jeOv0ElEZcZXJyokC5HmHRy0GFJNcnG8ExyfYDjdBSaxJWWWHin2XKHiY7DpmldbOK-4mpaY8_vj_P5iyuvRStY3oiGwv7RL$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bIl8jeOv0ElEZcZXJyokC5HmHRy0GFJNcnG8ExyfYDjdBSaxJWWWHin2XKHiY7DpmldbOK-4mpaY8_vj_P5iyuvRStY3oiGwv7RL$ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 18270 bytes Desc: image001.png URL: From knepley at gmail.com Fri Sep 13 15:59:52 2024 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 13 Sep 2024 16:59:52 -0400 Subject: [petsc-users] Hypre AMS usage In-Reply-To: References: Message-ID: On Fri, Sep 13, 2024 at 4:56?PM Karthikeyan Chockalingam - STFC UKRI < karthikeyan.chockalingam at stfc.ac.uk> wrote: > I would like to call HYPRE_AMSSetBetaPoissonMatrix from Hypre > > > https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/hypre/hypre.c?ref_type=heads*L325__;Iw!!G_uCfscf7eWS!ZJD-6o5hrlVpL8ybDZph6UhwQZfSRsXh6qrc8JSZFe5X5WVUdp4E5IL8wq6uq6DL0z1mDj78-_2eOZtqQC8-$ > > > https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/hypre/hypre.c?ref_type=heads*L329__;Iw!!G_uCfscf7eWS!ZJD-6o5hrlVpL8ybDZph6UhwQZfSRsXh6qrc8JSZFe5X5WVUdp4E5IL8wq6uq6DL0z1mDj78-_2eOV6Ojr7J$ > > > > Look like ams_beta_is_zero has to be true > I don't think that is true: https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blame/main/src/ksp/pc/impls/hypre/hypre.c?ref_type=heads*L329__;Iw!!G_uCfscf7eWS!ZJD-6o5hrlVpL8ybDZph6UhwQZfSRsXh6qrc8JSZFe5X5WVUdp4E5IL8wq6uq6DL0z1mDj78-_2eOdRLti-4$ Thanks, Matt > > So would I end up calling PCHYPRESetPoissonMatrix_HYPRE with isalpha being > false? > > > https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/hypre/hypre.c?ref_type=heads*L1640__;Iw!!G_uCfscf7eWS!ZJD-6o5hrlVpL8ybDZph6UhwQZfSRsXh6qrc8JSZFe5X5WVUdp4E5IL8wq6uq6DL0z1mDj78-_2eOfWoAnkR$ > > > > Kind regards, > > Karthik. > > > > *From: *Matthew Knepley > *Date: *Tuesday, 10 September 2024 at 19:22 > *To: *Chockalingam, Karthikeyan (STFC,DL,HC) < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *petsc-users at mcs.anl.gov > *Subject: *Re: [petsc-users] Hypre AMS usage > > On Tue, Sep 10, 2024 at 1:16?PM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > Hi Matt, > > > > I am not sure, if I understand how to read the source code, let?s take the > below line > > > > PetscCall(PetscOptionsReal("-pc_hypre_ams_relax_weight", "Relaxation > weight for AMS smoother", "None", jac->as_relax_weight, > &jac->as_relax_weight, &flag3)) > > (Q1) Am I doing it right by setting as in the line below? How does the > line below match up to the above? Since the above line has more arguments > > petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_weight", "1.0") > > > > Yes. All options take a single string argument. The other arguments to the > function help present it to the user (say through a GUI). > > > > (Q2) How do I go about setting the four parameters below using > PetscOptionsSetValue? > > > > It looks like you have 5 parameters, and you would call > > > > petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_amg_alpha_options", > "10,21,32,43,54") > > > > Thanks, > > > > Matt > > > > PetscCall(PetscOptionsIntArray("-pc_hypre_ams_amg_alpha_options", "AMG > options for vector Poisson", "None", jac->as_amg_alpha_opts, &n, &flag2)); > > if (flag || flag2) { > > PetscCallExternal(HYPRE_AMSSetAlphaAMGOptions, jac->hsolver, > jac->as_amg_alpha_opts[0], /* AMG coarsen type */ > > > jac->as_amg_alpha_opts[1], /* > AMG agg_levels */ > > jac->as_amg_alpha_opts[2], > /* AMG relax_type */ > > jac->as_amg_alpha_theta, > jac->as_amg_alpha_opts[3], /* AMG interp_type */ > > > jac->as_amg_alpha_opts[4]); /* > AMG Pmax */ > > } > > Kind regards, > > Karthik. > > > > > > > > *From: *Matthew Knepley > *Date: *Monday, 9 September 2024 at 22:00 > *To: *Chockalingam, Karthikeyan (STFC,DL,HC) < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *petsc-users at mcs.anl.gov > *Subject: *Re: [petsc-users] Hypre AMS usage > > On Mon, Sep 9, 2024 at 4:32?PM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > Great. Thank you for letting me know. > > I got the reference to KSP as well from libmesh and I am not creating a > new KSP. > > This time around, I didn?t have to set the PC type and it seems to work. > > > > Excellent. > > > > Thanks, > > > > Matt > > > > petscErr = PetscOptionsSetValue(NULL,"-ksp_type", "gmres"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_type", "hypre"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_type", "ams"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_type", "2"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_times", "1"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_weight", > "1.0"); > > petscErr = PetscOptionsSetValue(NULL, "-pc_hypre_ams_omega", "1.0"); > > petscErr = PetscOptionsSetValue(NULL, "-ksp_view", NULL); > > petscErr = PetscOptionsSetValue(NULL, "-ksp_monitor_true_residual", > NULL); > > petscErr = PetscOptionsSetValue(NULL, "-ksp_converged_reason", NULL); > > petscErr = PetscOptionsSetValue(NULL, "-options_left", NULL); > > petscErr = KSPSetFromOptions(ksp); > > > > // Set discrete gradient > > petscErr = PCHYPRESetDiscreteGradient(pc, par_G); > > > > // Set vertex coordinates > > petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, > par_zvec); > > > > > > *From: *Matthew Knepley > *Date: *Monday, 9 September 2024 at 21:11 > *To: *Chockalingam, Karthikeyan (STFC,DL,HC) < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *Stefano Zampini , petsc-users at mcs.anl.gov > > *Subject: *Re: [petsc-users] Hypre AMS usage > > On Mon, Sep 9, 2024 at 4:03?PM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > I never explicitly called KSPGetPC(). I am embedding AMS in a libmesh > (fem) example. So, I only got the reference to pc from libmesh and set > petscErr = PCHYPRESetDiscreteGradient(pc, par_G); > > > > However, you are creating a _new_ KSP, so your PC will not match it. This > is not doing what you want. > > > > Thanks, > > > > Matt > > > > > > *From: *Matthew Knepley > *Date: *Monday, 9 September 2024 at 20:57 > *To: *Chockalingam, Karthikeyan (STFC,DL,HC) < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *Stefano Zampini , petsc-users at mcs.anl.gov > > *Subject: *Re: [petsc-users] Hypre AMS usage > > On Mon, Sep 9, 2024 at 3:38?PM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > I didn?t know how to check the pc type but adding > > petscErr = PCSetType(pc, "hypre"); > > before the two functions made it to work. > > > > But how does it work from the command line? > > > > For summary: > > > > KSPCreate(mesh.comm().get(), &ksp); > > petscErr = PetscOptionsSetValue(NULL,"-ksp_type", "gmres"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_type", "hypre"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_type", "ams"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_type", "2"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_times", "1"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_weigh", "1.0"); > > petscErr = PetscOptionsSetValue(NULL, "-pc_hypre_ams_omega", "1.0"); > > petscErr = PetscOptionsSetValue(NULL, "-ksp_view", NULL); > > petscErr = PetscOptionsSetValue(NULL, "-ksp_monitor_true_residual", > NULL); > > petscErr = PetscOptionsSetValue(NULL, "-ksp_converged_reason", NULL); > > petscErr = PetscOptionsSetValue(NULL, "-options_left", NULL); > > petscErr = KSPSetFromOptions(ksp); > > > > Where is the KSPGetPC() call? > > > > Thanks, > > > > Matt > > > > // Set pc type > > petscErr = PCSetType(pc, "hypre"); > > > > // Set discrete gradient > > petscErr = PCHYPRESetDiscreteGradient(pc, par_G); > > > > // Set vertex coordinates > > petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, > par_zvec); > > > > > > *From: *Matthew Knepley > *Date: *Monday, 9 September 2024 at 19:46 > *To: *Chockalingam, Karthikeyan (STFC,DL,HC) < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *Stefano Zampini , petsc-users at mcs.anl.gov > > *Subject: *Re: [petsc-users] Hypre AMS usage > > On Mon, Sep 9, 2024 at 2:18?PM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > Calling the two functions after KSPSetFromOptions() did not work either. > > > > Can you check that the PC has the correct type when you call it? > > > > Thanks, > > > > Matt > > > > Everything works from the command line. > > I haven?t set KSPSetOperators, not sure if that is an issue. > > > > *From: *Matthew Knepley > *Date: *Monday, 9 September 2024 at 19:09 > *To: *Chockalingam, Karthikeyan (STFC,DL,HC) < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *Stefano Zampini , petsc-users at mcs.anl.gov > > *Subject: *Re: [petsc-users] Hypre AMS usage > > On Mon, Sep 9, 2024 at 1:21?PM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > Hi Stefano, > > > > Thank you. That was helpful. I tried the following: > > > > petscErr = PetscOptionsSetValue(NULL,"-ksp_type", "gmres"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_type", "hypre"); > > petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_type", "ams"); > > > > // Set discrete gradient > > petscErr = PCHYPRESetDiscreteGradient(pc, par_G); > > > > // Set vertex coordinates > > petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, > par_zvec); > > > > petscErr = PetscOptionsSetValue(NULL, "-ksp_view", NULL); > > petscErr = PetscOptionsSetValue(NULL, "-ksp_monitor_true_residual", > NULL); > > petscErr = PetscOptionsSetValue(NULL, "-ksp_converged_reason", NULL); > > petscErr = PetscOptionsSetValue(NULL, "-options_left", NULL); > > petscErr = KSPSetFromOptions(ksp); > > > > > > It errored though I have set PCHYPRESetEdgeConstantVectors. > > > > My guess is that the "pc" was not yet Hypre, since that type was not set > until KSPSetFromOptions() was called. So you need > > to call those two functions after that call. > > > > Thanks, > > > > Matt > > > > But my program works, without PetscOptionsSetValue and by passing > everything on the command line. > > > > *[0]PETSC ERROR: --------------------- Error Message > --------------------------------------------------------------* > > [0]PETSC ERROR: HYPRE AMS preconditioner needs either the coordinate > vectors via PCSetCoordinates() or the edge constant vectors via > PCHYPRESetEdgeConstantVectors() or the interpolation matrix via > PCHYPRESetInterpolations() > > [0]PETS C ERROR: WARNING! There are unused option(s) set! Could be the > program crashed before usage or a spelling mistake, etc! > > [0]PETSC ERROR: Option left: name:-ksp_converged_reason (no value) > source: code > > [0]PETSC ERROR: Option left: name:-options_left (no value) source: code > > [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!ZJD-6o5hrlVpL8ybDZph6UhwQZfSRsXh6qrc8JSZFe5X5WVUdp4E5IL8wq6uq6DL0z1mDj78-_2eOYewD_HW$ for trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.20.3, unknown > > [0]PETSC ERROR: ./example-dbg on a arch-moose named HC20210312 by > karthikeyan.chockalingam Mon Sep 9 18:10:13 2024 > > [0]PETSC ERROR: Configure options --with-64-bit-indices > --with-cxx-dialect=C++17 --with-debugging=no --with-fortran-bindings=0 > --with-mpi=1 --with-openmp=1 --with-shared-libraries=1 --with-sowing=0 > --download-fblaslapack=1 --download-hypre=1 --download-metis=1 > --download-mumps=1 --download-ptscotch=1 --download-parmetis=1 > --download-scalapack=1 --download-slepc=1 --download-strumpack=1 > --download-superlu_dist=1 --download-bison=1 --download-hdf5=1 > --with-hdf5-fortran-bindings=0 > --download-hdf5-configure-arguments="--with-zlib" --with-make-np=4 > > [0]PETSC ERROR: #1 PCSetUp_HYPRE() at > /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/pc/impls/hypre/hypre.c:295 > > [0]PETSC ERROR: #2 PCSetUp() at > /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/pc/interface/precon.c:1080 > > [0]PETSC ERROR: #3 KSPSetUp() at > /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/ksp/interface/itfunc.c:415 > > [0]PETSC ERROR: #4 KSPSolve_Private() at > /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/ksp/interface/itfunc.c:833 > > [0]PETSC ERROR: #5 KSPSolve() at > /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/ksp/interface/itfunc.c:1080 > > libMesh terminating: > > HYPRE AMS preconditioner needs either the coordinate vectors via > PCSetCoordinates() or the edge constant vectors via > PCHYPRESetEdgeConstantVectors() or the interpolation matrix via > PCHYPRESetInterpolations() > > > > > > *From: *Stefano Zampini > *Date: *Monday, 9 September 2024 at 17:02 > *To: *Matthew Knepley > *Cc: *Chockalingam, Karthikeyan (STFC,DL,HC) < > karthikeyan.chockalingam at stfc.ac.uk>, petsc-users at mcs.anl.gov < > petsc-users at mcs.anl.gov> > *Subject: *Re: [petsc-users] Hypre AMS usage > > I would say the best way is to look at the source code > > > > > https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/hypre/hypre.c?ref_type=heads*L2061__;Iw!!G_uCfscf7eWS!ZJD-6o5hrlVpL8ybDZph6UhwQZfSRsXh6qrc8JSZFe5X5WVUdp4E5IL8wq6uq6DL0z1mDj78-_2eOTtuLRbw$ > > > https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/hypre/hypre.c?ref_type=heads*L1239__;Iw!!G_uCfscf7eWS!ZJD-6o5hrlVpL8ybDZph6UhwQZfSRsXh6qrc8JSZFe5X5WVUdp4E5IL8wq6uq6DL0z1mDj78-_2eOafqSNH2$ > > > > Il giorno lun 9 set 2024 alle ore 18:50 Matthew Knepley > ha scritto: > > On Mon, Sep 9, 2024 at 10:17?AM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > Hi Matt, > > > > You mentioned it doesn?t hurt to set the smoothing flags > > > > > https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L965__;Iw!!G_uCfscf7eWS!ZJD-6o5hrlVpL8ybDZph6UhwQZfSRsXh6qrc8JSZFe5X5WVUdp4E5IL8wq6uq6DL0z1mDj78-_2eOXdBGmyD$ > > > Do you know where I can look for the equivalent PETSc commands? Thank you. > > > > Yes, this is described here: > https://urldefense.us/v3/__https://petsc.org/main/manualpages/PC/PCHYPRE/__;!!G_uCfscf7eWS!ZJD-6o5hrlVpL8ybDZph6UhwQZfSRsXh6qrc8JSZFe5X5WVUdp4E5IL8wq6uq6DL0z1mDj78-_2eOet4C2K_$ > > > The best way is to run with -help or look at the source. > > > > Thanks, > > > > Matt > > > > > > Kind regards, > > Karthik. > > > > *From: *Matthew Knepley > *Date: *Friday, 6 September 2024 at 17:57 > *To: *Chockalingam, Karthikeyan (STFC,DL,HC) < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *petsc-users at mcs.anl.gov > *Subject: *Re: [petsc-users] Hypre AMS usage > > On Fri, Sep 6, 2024 at 11:37?AM Karthikeyan Chockalingam - STFC UKRI via > petsc-users wrote: > > Hello, > > > > I am trying to use the Hypre AMS preconditioner for the first time. > > > > I am following the example problem from Hypre > > > https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L954__;Iw!!G_uCfscf7eWS!ZJD-6o5hrlVpL8ybDZph6UhwQZfSRsXh6qrc8JSZFe5X5WVUdp4E5IL8wq6uq6DL0z1mDj78-_2eOYORksiX$ > > > > > I have so far successfully set the discrete gradient operator and vertex > co-ordinates, > > > > // Set discrete gradient > > petscErr = PCHYPRESetDiscreteGradient(pc, par_G); > > > > // Set vertex coordinates > > petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, > par_zvec); > > > > Do I need to set the following smoothing options? > > > https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L965__;Iw!!G_uCfscf7eWS!ZJD-6o5hrlVpL8ybDZph6UhwQZfSRsXh6qrc8JSZFe5X5WVUdp4E5IL8wq6uq6DL0z1mDj78-_2eOXdBGmyD$ > > > > > It cannot hurt. I would set them to begin with. > > > > Also, do I need to convert from MATMPIAIJ to CSR? > > > https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L984__;Iw!!G_uCfscf7eWS!ZJD-6o5hrlVpL8ybDZph6UhwQZfSRsXh6qrc8JSZFe5X5WVUdp4E5IL8wq6uq6DL0z1mDj78-_2eOTzVXwDj$ > > > > > No. > > > > What are the other PETSc calls to invoke AMS? Is there an example problem > I can look at? > > > > I do not know. I don't think we have an example. > > > > Thanks, > > > > Matt > > > > Thank you. > > > > Karthik. > > > > -- > > *Karthik Chockalingam, Ph.D.* > > Senior Research Software Engineer > > High Performance Systems Engineering Group > > Hartree Centre | Science and Technology Facilities Council > > karthikeyan.chockalingam at stfc.ac.uk > > > > [image: signature_3970890138] > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!ZJD-6o5hrlVpL8ybDZph6UhwQZfSRsXh6qrc8JSZFe5X5WVUdp4E5IL8wq6uq6DL0z1mDj78-_2eOe6JDc03$ > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!ZJD-6o5hrlVpL8ybDZph6UhwQZfSRsXh6qrc8JSZFe5X5WVUdp4E5IL8wq6uq6DL0z1mDj78-_2eOe6JDc03$ > > > > > > -- > > Stefano > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!ZJD-6o5hrlVpL8ybDZph6UhwQZfSRsXh6qrc8JSZFe5X5WVUdp4E5IL8wq6uq6DL0z1mDj78-_2eOe6JDc03$ > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!ZJD-6o5hrlVpL8ybDZph6UhwQZfSRsXh6qrc8JSZFe5X5WVUdp4E5IL8wq6uq6DL0z1mDj78-_2eOe6JDc03$ > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!ZJD-6o5hrlVpL8ybDZph6UhwQZfSRsXh6qrc8JSZFe5X5WVUdp4E5IL8wq6uq6DL0z1mDj78-_2eOe6JDc03$ > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!ZJD-6o5hrlVpL8ybDZph6UhwQZfSRsXh6qrc8JSZFe5X5WVUdp4E5IL8wq6uq6DL0z1mDj78-_2eOe6JDc03$ > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!ZJD-6o5hrlVpL8ybDZph6UhwQZfSRsXh6qrc8JSZFe5X5WVUdp4E5IL8wq6uq6DL0z1mDj78-_2eOe6JDc03$ > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!ZJD-6o5hrlVpL8ybDZph6UhwQZfSRsXh6qrc8JSZFe5X5WVUdp4E5IL8wq6uq6DL0z1mDj78-_2eOe6JDc03$ > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!ZJD-6o5hrlVpL8ybDZph6UhwQZfSRsXh6qrc8JSZFe5X5WVUdp4E5IL8wq6uq6DL0z1mDj78-_2eOe6JDc03$ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 18270 bytes Desc: not available URL: From karthikeyan.chockalingam at stfc.ac.uk Fri Sep 13 16:05:56 2024 From: karthikeyan.chockalingam at stfc.ac.uk (Karthikeyan Chockalingam - STFC UKRI) Date: Fri, 13 Sep 2024 21:05:56 +0000 Subject: [petsc-users] Hypre AMS usage In-Reply-To: References: Message-ID: Agreed but I would like to HYPRE_AMSSetBetaPoissonMatrix(ams, NULL); So it does look like ams_beta_is_zero has to be true https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/hypre/hypre.c?ref_type=heads*L325__;Iw!!G_uCfscf7eWS!fGbqNXg5CfhwXeT3V0Drg4Pm22hSYJSfMQZxLlhNo9ax2s9XDXVSUY8t_5ly57te4gqalgJL4pn9XWZ6tJJXfxXCJLV895Ooco1g$ Same question again So would I end up calling PCHYPRESetPoissonMatrix_HYPRE with isalpha being false? https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/hypre/hypre.c?ref_type=heads*L1640__;Iw!!G_uCfscf7eWS!fGbqNXg5CfhwXeT3V0Drg4Pm22hSYJSfMQZxLlhNo9ax2s9XDXVSUY8t_5ly57te4gqalgJL4pn9XWZ6tJJXfxXCJLV8924nlFrx$ From: Matthew Knepley Date: Friday, 13 September 2024 at 22:00 To: Chockalingam, Karthikeyan (STFC,DL,HC) Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Hypre AMS usage On Fri, Sep 13, 2024 at 4:56?PM Karthikeyan Chockalingam - STFC UKRI > wrote: I would like to call HYPRE_AMSSetBetaPoissonMatrix from Hypre https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/hypre/hypre.c?ref_type=heads*L325__;Iw!!G_uCfscf7eWS!fGbqNXg5CfhwXeT3V0Drg4Pm22hSYJSfMQZxLlhNo9ax2s9XDXVSUY8t_5ly57te4gqalgJL4pn9XWZ6tJJXfxXCJLV895Ooco1g$ https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/hypre/hypre.c?ref_type=heads*L329__;Iw!!G_uCfscf7eWS!fGbqNXg5CfhwXeT3V0Drg4Pm22hSYJSfMQZxLlhNo9ax2s9XDXVSUY8t_5ly57te4gqalgJL4pn9XWZ6tJJXfxXCJLV892d8V53I$ Look like ams_beta_is_zero has to be true I don't think that is true: https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blame/main/src/ksp/pc/impls/hypre/hypre.c?ref_type=heads*L329__;Iw!!G_uCfscf7eWS!fGbqNXg5CfhwXeT3V0Drg4Pm22hSYJSfMQZxLlhNo9ax2s9XDXVSUY8t_5ly57te4gqalgJL4pn9XWZ6tJJXfxXCJLV896TlVv-V$ Thanks, Matt So would I end up calling PCHYPRESetPoissonMatrix_HYPRE with isalpha being false? https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/hypre/hypre.c?ref_type=heads*L1640__;Iw!!G_uCfscf7eWS!fGbqNXg5CfhwXeT3V0Drg4Pm22hSYJSfMQZxLlhNo9ax2s9XDXVSUY8t_5ly57te4gqalgJL4pn9XWZ6tJJXfxXCJLV8924nlFrx$ Kind regards, Karthik. From: Matthew Knepley > Date: Tuesday, 10 September 2024 at 19:22 To: Chockalingam, Karthikeyan (STFC,DL,HC) > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Hypre AMS usage On Tue, Sep 10, 2024 at 1:16?PM Karthikeyan Chockalingam - STFC UKRI > wrote: Hi Matt, I am not sure, if I understand how to read the source code, let?s take the below line PetscCall(PetscOptionsReal("-pc_hypre_ams_relax_weight", "Relaxation weight for AMS smoother", "None", jac->as_relax_weight, &jac->as_relax_weight, &flag3)) (Q1) Am I doing it right by setting as in the line below? How does the line below match up to the above? Since the above line has more arguments petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_weight", "1.0") Yes. All options take a single string argument. The other arguments to the function help present it to the user (say through a GUI). (Q2) How do I go about setting the four parameters below using PetscOptionsSetValue? It looks like you have 5 parameters, and you would call petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_amg_alpha_options", "10,21,32,43,54") Thanks, Matt PetscCall(PetscOptionsIntArray("-pc_hypre_ams_amg_alpha_options", "AMG options for vector Poisson", "None", jac->as_amg_alpha_opts, &n, &flag2)); if (flag || flag2) { PetscCallExternal(HYPRE_AMSSetAlphaAMGOptions, jac->hsolver, jac->as_amg_alpha_opts[0], /* AMG coarsen type */ jac->as_amg_alpha_opts[1], /* AMG agg_levels */ jac->as_amg_alpha_opts[2], /* AMG relax_type */ jac->as_amg_alpha_theta, jac->as_amg_alpha_opts[3], /* AMG interp_type */ jac->as_amg_alpha_opts[4]); /* AMG Pmax */ } Kind regards, Karthik. From: Matthew Knepley > Date: Monday, 9 September 2024 at 22:00 To: Chockalingam, Karthikeyan (STFC,DL,HC) > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Hypre AMS usage On Mon, Sep 9, 2024 at 4:32?PM Karthikeyan Chockalingam - STFC UKRI > wrote: Great. Thank you for letting me know. I got the reference to KSP as well from libmesh and I am not creating a new KSP. This time around, I didn?t have to set the PC type and it seems to work. Excellent. Thanks, Matt petscErr = PetscOptionsSetValue(NULL,"-ksp_type", "gmres"); petscErr = PetscOptionsSetValue(NULL,"-pc_type", "hypre"); petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_type", "ams"); petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_type", "2"); petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_times", "1"); petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_weight", "1.0"); petscErr = PetscOptionsSetValue(NULL, "-pc_hypre_ams_omega", "1.0"); petscErr = PetscOptionsSetValue(NULL, "-ksp_view", NULL); petscErr = PetscOptionsSetValue(NULL, "-ksp_monitor_true_residual", NULL); petscErr = PetscOptionsSetValue(NULL, "-ksp_converged_reason", NULL); petscErr = PetscOptionsSetValue(NULL, "-options_left", NULL); petscErr = KSPSetFromOptions(ksp); // Set discrete gradient petscErr = PCHYPRESetDiscreteGradient(pc, par_G); // Set vertex coordinates petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, par_zvec); From: Matthew Knepley > Date: Monday, 9 September 2024 at 21:11 To: Chockalingam, Karthikeyan (STFC,DL,HC) > Cc: Stefano Zampini >, petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Hypre AMS usage On Mon, Sep 9, 2024 at 4:03?PM Karthikeyan Chockalingam - STFC UKRI > wrote: I never explicitly called KSPGetPC(). I am embedding AMS in a libmesh (fem) example. So, I only got the reference to pc from libmesh and set petscErr = PCHYPRESetDiscreteGradient(pc, par_G); However, you are creating a _new_ KSP, so your PC will not match it. This is not doing what you want. Thanks, Matt From: Matthew Knepley > Date: Monday, 9 September 2024 at 20:57 To: Chockalingam, Karthikeyan (STFC,DL,HC) > Cc: Stefano Zampini >, petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Hypre AMS usage On Mon, Sep 9, 2024 at 3:38?PM Karthikeyan Chockalingam - STFC UKRI > wrote: I didn?t know how to check the pc type but adding petscErr = PCSetType(pc, "hypre"); before the two functions made it to work. But how does it work from the command line? For summary: KSPCreate(mesh.comm().get(), &ksp); petscErr = PetscOptionsSetValue(NULL,"-ksp_type", "gmres"); petscErr = PetscOptionsSetValue(NULL,"-pc_type", "hypre"); petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_type", "ams"); petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_type", "2"); petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_times", "1"); petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_ams_relax_weigh", "1.0"); petscErr = PetscOptionsSetValue(NULL, "-pc_hypre_ams_omega", "1.0"); petscErr = PetscOptionsSetValue(NULL, "-ksp_view", NULL); petscErr = PetscOptionsSetValue(NULL, "-ksp_monitor_true_residual", NULL); petscErr = PetscOptionsSetValue(NULL, "-ksp_converged_reason", NULL); petscErr = PetscOptionsSetValue(NULL, "-options_left", NULL); petscErr = KSPSetFromOptions(ksp); Where is the KSPGetPC() call? Thanks, Matt // Set pc type petscErr = PCSetType(pc, "hypre"); // Set discrete gradient petscErr = PCHYPRESetDiscreteGradient(pc, par_G); // Set vertex coordinates petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, par_zvec); From: Matthew Knepley > Date: Monday, 9 September 2024 at 19:46 To: Chockalingam, Karthikeyan (STFC,DL,HC) > Cc: Stefano Zampini >, petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Hypre AMS usage On Mon, Sep 9, 2024 at 2:18?PM Karthikeyan Chockalingam - STFC UKRI > wrote: Calling the two functions after KSPSetFromOptions() did not work either. Can you check that the PC has the correct type when you call it? Thanks, Matt Everything works from the command line. I haven?t set KSPSetOperators, not sure if that is an issue. From: Matthew Knepley > Date: Monday, 9 September 2024 at 19:09 To: Chockalingam, Karthikeyan (STFC,DL,HC) > Cc: Stefano Zampini >, petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Hypre AMS usage On Mon, Sep 9, 2024 at 1:21?PM Karthikeyan Chockalingam - STFC UKRI > wrote: Hi Stefano, Thank you. That was helpful. I tried the following: petscErr = PetscOptionsSetValue(NULL,"-ksp_type", "gmres"); petscErr = PetscOptionsSetValue(NULL,"-pc_type", "hypre"); petscErr = PetscOptionsSetValue(NULL,"-pc_hypre_type", "ams"); // Set discrete gradient petscErr = PCHYPRESetDiscreteGradient(pc, par_G); // Set vertex coordinates petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, par_zvec); petscErr = PetscOptionsSetValue(NULL, "-ksp_view", NULL); petscErr = PetscOptionsSetValue(NULL, "-ksp_monitor_true_residual", NULL); petscErr = PetscOptionsSetValue(NULL, "-ksp_converged_reason", NULL); petscErr = PetscOptionsSetValue(NULL, "-options_left", NULL); petscErr = KSPSetFromOptions(ksp); It errored though I have set PCHYPRESetEdgeConstantVectors. My guess is that the "pc" was not yet Hypre, since that type was not set until KSPSetFromOptions() was called. So you need to call those two functions after that call. Thanks, Matt But my program works, without PetscOptionsSetValue and by passing everything on the command line. [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: HYPRE AMS preconditioner needs either the coordinate vectors via PCSetCoordinates() or the edge constant vectors via PCHYPRESetEdgeConstantVectors() or the interpolation matrix via PCHYPRESetInterpolations() [0]PETS C ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc! [0]PETSC ERROR: Option left: name:-ksp_converged_reason (no value) source: code [0]PETSC ERROR: Option left: name:-options_left (no value) source: code [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!fGbqNXg5CfhwXeT3V0Drg4Pm22hSYJSfMQZxLlhNo9ax2s9XDXVSUY8t_5ly57te4gqalgJL4pn9XWZ6tJJXfxXCJLV89ydc2aM7$ for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.20.3, unknown [0]PETSC ERROR: ./example-dbg on a arch-moose named HC20210312 by karthikeyan.chockalingam Mon Sep 9 18:10:13 2024 [0]PETSC ERROR: Configure options --with-64-bit-indices --with-cxx-dialect=C++17 --with-debugging=no --with-fortran-bindings=0 --with-mpi=1 --with-openmp=1 --with-shared-libraries=1 --with-sowing=0 --download-fblaslapack=1 --download-hypre=1 --download-metis=1 --download-mumps=1 --download-ptscotch=1 --download-parmetis=1 --download-scalapack=1 --download-slepc=1 --download-strumpack=1 --download-superlu_dist=1 --download-bison=1 --download-hdf5=1 --with-hdf5-fortran-bindings=0 --download-hdf5-configure-arguments="--with-zlib" --with-make-np=4 [0]PETSC ERROR: #1 PCSetUp_HYPRE() at /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/pc/impls/hypre/hypre.c:295 [0]PETSC ERROR: #2 PCSetUp() at /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/pc/interface/precon.c:1080 [0]PETSC ERROR: #3 KSPSetUp() at /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/ksp/interface/itfunc.c:415 [0]PETSC ERROR: #4 KSPSolve_Private() at /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/ksp/interface/itfunc.c:833 [0]PETSC ERROR: #5 KSPSolve() at /Users/karthikeyan.chockalingam/moose/petsc/src/ksp/ksp/interface/itfunc.c:1080 libMesh terminating: HYPRE AMS preconditioner needs either the coordinate vectors via PCSetCoordinates() or the edge constant vectors via PCHYPRESetEdgeConstantVectors() or the interpolation matrix via PCHYPRESetInterpolations() From: Stefano Zampini > Date: Monday, 9 September 2024 at 17:02 To: Matthew Knepley > Cc: Chockalingam, Karthikeyan (STFC,DL,HC) >, petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Hypre AMS usage I would say the best way is to look at the source code https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/hypre/hypre.c?ref_type=heads*L2061__;Iw!!G_uCfscf7eWS!fGbqNXg5CfhwXeT3V0Drg4Pm22hSYJSfMQZxLlhNo9ax2s9XDXVSUY8t_5ly57te4gqalgJL4pn9XWZ6tJJXfxXCJLV895FDHJiC$ https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/hypre/hypre.c?ref_type=heads*L1239__;Iw!!G_uCfscf7eWS!fGbqNXg5CfhwXeT3V0Drg4Pm22hSYJSfMQZxLlhNo9ax2s9XDXVSUY8t_5ly57te4gqalgJL4pn9XWZ6tJJXfxXCJLV89xg-q7jy$ Il giorno lun 9 set 2024 alle ore 18:50 Matthew Knepley > ha scritto: On Mon, Sep 9, 2024 at 10:17?AM Karthikeyan Chockalingam - STFC UKRI > wrote: Hi Matt, You mentioned it doesn?t hurt to set the smoothing flags https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L965__;Iw!!G_uCfscf7eWS!fGbqNXg5CfhwXeT3V0Drg4Pm22hSYJSfMQZxLlhNo9ax2s9XDXVSUY8t_5ly57te4gqalgJL4pn9XWZ6tJJXfxXCJLV894_XFimO$ Do you know where I can look for the equivalent PETSc commands? Thank you. Yes, this is described here: https://urldefense.us/v3/__https://petsc.org/main/manualpages/PC/PCHYPRE/__;!!G_uCfscf7eWS!fGbqNXg5CfhwXeT3V0Drg4Pm22hSYJSfMQZxLlhNo9ax2s9XDXVSUY8t_5ly57te4gqalgJL4pn9XWZ6tJJXfxXCJLV893xdd5fQ$ The best way is to run with -help or look at the source. Thanks, Matt Kind regards, Karthik. From: Matthew Knepley > Date: Friday, 6 September 2024 at 17:57 To: Chockalingam, Karthikeyan (STFC,DL,HC) > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Hypre AMS usage On Fri, Sep 6, 2024 at 11:37?AM Karthikeyan Chockalingam - STFC UKRI via petsc-users > wrote: Hello, I am trying to use the Hypre AMS preconditioner for the first time. I am following the example problem from Hypre https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L954__;Iw!!G_uCfscf7eWS!fGbqNXg5CfhwXeT3V0Drg4Pm22hSYJSfMQZxLlhNo9ax2s9XDXVSUY8t_5ly57te4gqalgJL4pn9XWZ6tJJXfxXCJLV89ztdoO1D$ I have so far successfully set the discrete gradient operator and vertex co-ordinates, // Set discrete gradient petscErr = PCHYPRESetDiscreteGradient(pc, par_G); // Set vertex coordinates petscErr = PCHYPRESetEdgeConstantVectors(pc, par_xvec, par_yvec, par_zvec); Do I need to set the following smoothing options? https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L965__;Iw!!G_uCfscf7eWS!fGbqNXg5CfhwXeT3V0Drg4Pm22hSYJSfMQZxLlhNo9ax2s9XDXVSUY8t_5ly57te4gqalgJL4pn9XWZ6tJJXfxXCJLV894_XFimO$ It cannot hurt. I would set them to begin with. Also, do I need to convert from MATMPIAIJ to CSR? https://urldefense.us/v3/__https://github.com/hypre-space/hypre/blob/3caa81955eb8d1b4e35d9b450e27cf6d07b50f6e/src/examples/ex15.c*L984__;Iw!!G_uCfscf7eWS!fGbqNXg5CfhwXeT3V0Drg4Pm22hSYJSfMQZxLlhNo9ax2s9XDXVSUY8t_5ly57te4gqalgJL4pn9XWZ6tJJXfxXCJLV897Bph6m0$ No. What are the other PETSc calls to invoke AMS? Is there an example problem I can look at? I do not know. I don't think we have an example. Thanks, Matt Thank you. Karthik. -- Karthik Chockalingam, Ph.D. Senior Research Software Engineer High Performance Systems Engineering Group Hartree Centre | Science and Technology Facilities Council karthikeyan.chockalingam at stfc.ac.uk [signature_3970890138] -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fGbqNXg5CfhwXeT3V0Drg4Pm22hSYJSfMQZxLlhNo9ax2s9XDXVSUY8t_5ly57te4gqalgJL4pn9XWZ6tJJXfxXCJLV899xE8_hA$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fGbqNXg5CfhwXeT3V0Drg4Pm22hSYJSfMQZxLlhNo9ax2s9XDXVSUY8t_5ly57te4gqalgJL4pn9XWZ6tJJXfxXCJLV899xE8_hA$ -- Stefano -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fGbqNXg5CfhwXeT3V0Drg4Pm22hSYJSfMQZxLlhNo9ax2s9XDXVSUY8t_5ly57te4gqalgJL4pn9XWZ6tJJXfxXCJLV899xE8_hA$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fGbqNXg5CfhwXeT3V0Drg4Pm22hSYJSfMQZxLlhNo9ax2s9XDXVSUY8t_5ly57te4gqalgJL4pn9XWZ6tJJXfxXCJLV899xE8_hA$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fGbqNXg5CfhwXeT3V0Drg4Pm22hSYJSfMQZxLlhNo9ax2s9XDXVSUY8t_5ly57te4gqalgJL4pn9XWZ6tJJXfxXCJLV899xE8_hA$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fGbqNXg5CfhwXeT3V0Drg4Pm22hSYJSfMQZxLlhNo9ax2s9XDXVSUY8t_5ly57te4gqalgJL4pn9XWZ6tJJXfxXCJLV899xE8_hA$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fGbqNXg5CfhwXeT3V0Drg4Pm22hSYJSfMQZxLlhNo9ax2s9XDXVSUY8t_5ly57te4gqalgJL4pn9XWZ6tJJXfxXCJLV899xE8_hA$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fGbqNXg5CfhwXeT3V0Drg4Pm22hSYJSfMQZxLlhNo9ax2s9XDXVSUY8t_5ly57te4gqalgJL4pn9XWZ6tJJXfxXCJLV899xE8_hA$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fGbqNXg5CfhwXeT3V0Drg4Pm22hSYJSfMQZxLlhNo9ax2s9XDXVSUY8t_5ly57te4gqalgJL4pn9XWZ6tJJXfxXCJLV899xE8_hA$ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 18270 bytes Desc: image001.png URL: From liufield at gmail.com Sun Sep 15 05:08:23 2024 From: liufield at gmail.com (neil liu) Date: Sun, 15 Sep 2024 06:08:23 -0400 Subject: [petsc-users] Inquiry about metric-based adaptation using MMG in petsc In-Reply-To: <87o74rfm69.fsf@jedbrown.org> References: <87o74rfm69.fsf@jedbrown.org> Message-ID: Thanks a lot, Jed. I traced the code and found the following issues, 1) in *PetscErrorCode DMPlexMetricNormalize(DM dm, Vec metricIn, PetscBool restrictSizes, PetscBool restrictAnisotropy, Vec metricOut, Vec determinant),* the metricIn is zero *. And this seems not reasonable. * *Go back to,* 2) DMAdaptorAdapt_Sequence_Private(DMAdaptor adaptor, Vec inx, PetscBool doSolve, DM *adm, Vec *ax) with -sol_adapt_loc_pre_view, -adapt_gradient_view and -adapt_hessian_view, all these three vectors are zero. This is not reasonable. *Go back to, * *3) *PetscErrorCode DMAdaptorAdapt(DMAdaptor adaptor, Vec x, DMAdaptationStrategy strategy, DM *adm, Vec *ax) x is zero here. Go back to, *4) *PetscCall(SNESSolve(snes, NULL, u)); The initial guess u is zero. It seems this is the reason. A little weird. An intiial guess with zero should be resonable. It seems maybe something is not reasonable for the logic in *DMPlexMetricNormalize.* *Thanks,* *Xiaodong * On Fri, Sep 13, 2024 at 2:22?PM Jed Brown wrote: > This error message is coming from the following: > > $ rg -B4 'Global metric normalization factor must be in' > src/dm/impls/plex/plexmetric.c > 1308- PetscCall(PetscDSSetObjective(ds, 0, detMFunc)); > 1309- PetscCall(DMPlexComputeIntegralFEM(dmDet, determinant, &integral, > NULL)); > 1310- } > 1311- realIntegral = PetscRealPart(integral); > 1312: PetscCheck(realIntegral > 1.0e-30, PETSC_COMM_SELF, > PETSC_ERR_ARG_OUTOFRANGE, "Global metric normalization factor must be in > (0, inf). Is the input metric positive-definite?"); > > Perhaps you can independently check what integrand is being provided. It's > probably zero or negative. You could apply this patch so the error message > will report a value. > > > diff --git i/src/dm/impls/plex/plexmetric.c > w/src/dm/impls/plex/plexmetric.c > index 61caeed28de..906cb394027 100644 > --- i/src/dm/impls/plex/plexmetric.c > +++ w/src/dm/impls/plex/plexmetric.c > @@ -1309,7 +1309,7 @@ PetscErrorCode DMPlexMetricNormalize(DM dm, Vec > metricIn, PetscBool restrictSize > PetscCall(DMPlexComputeIntegralFEM(dmDet, determinant, &integral, > NULL)); > } > realIntegral = PetscRealPart(integral); > - PetscCheck(realIntegral > 1.0e-30, PETSC_COMM_SELF, > PETSC_ERR_ARG_OUTOFRANGE, "Global metric normalization factor must be in > (0, inf). Is the input metric positive-definite?"); > + PetscCheck(realIntegral > 1.0e-30, PETSC_COMM_SELF, > PETSC_ERR_ARG_OUTOFRANGE, "Global metric normalization factor %g must be in > (0, inf). Is the input metric positive-definite?", (double)realIntegral); > factGlob = PetscPowReal(target / realIntegral, 2.0 / dim); > > /* Apply local scaling */ > > > neil liu writes: > > > Dear Petsc developers and users, > > > > I am trying to explore adaptive mesh refinement (tetradedra) with Petsc. > > It seems case 12 ($PETSC DIR/src/snes/tutorials/ex12.c, from paper, > > > https://urldefense.us/v3/__https://arxiv.org/pdf/2201.02806__;!!G_uCfscf7eWS!ZEwaHkvYc_IrjycFr8LiNi-nZcFqu0ZpsydAAomxptQ6C0xkBs3qhn5ba31Z4vipKf4mTrqDm8A5S65DDD_xFg$ > ) is a good example. > > However when I tried, > > ./ex12 -run_type full -dm_plex_dim 3 -dm_distribute -dm_plex_box_faces > > 10,10,10 -bc_type dirichlet -petscspace_degree 1 -variable_coefficient > ball > > -snes_converged > > _reason ::ascii_info_detail -ksp_type cg -pc_type sor > -snes_adapt_sequence > > 3 -adaptor_target_num 1000 -dm_plex_metric_h_max 0.5 -dm_adaptor mmg > > L_2 Error: 1.55486 > > Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE > iterations 2 > > > > it shows the following error, > > [0]PETSC ERROR: --------------------- Error Message > > -------------------------------------------------------------- > > [0]PETSC ERROR: Argument out of range > > [0]PETSC ERROR: Global metric normalization factor must be in (0, inf). > Is > > the input metric positive-definite? > > > > Do you have any suggestions here? > > > > Thanks, > > > > Xiaodong > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sun Sep 15 08:33:28 2024 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 15 Sep 2024 09:33:28 -0400 Subject: [petsc-users] Inquiry about metric-based adaptation using MMG in petsc In-Reply-To: References: <87o74rfm69.fsf@jedbrown.org> Message-ID: On Sun, Sep 15, 2024 at 7:25?AM neil liu wrote: > Thanks a lot, Jed. > I traced the code and found the following issues, > 1) in *PetscErrorCode DMPlexMetricNormalize(DM dm, Vec metricIn, > PetscBool restrictSizes, PetscBool restrictAnisotropy, Vec metricOut, Vec > determinant),* > the metricIn is zero > *. And this seems not reasonable. * > *Go back to,* > 2) DMAdaptorAdapt_Sequence_Private(DMAdaptor adaptor, Vec inx, PetscBool > doSolve, DM *adm, Vec *ax) > with -sol_adapt_loc_pre_view, -adapt_gradient_view and > -adapt_hessian_view, all these three vectors are zero. > This is not reasonable. > > *Go back to, * > *3) *PetscErrorCode DMAdaptorAdapt(DMAdaptor adaptor, Vec x, > DMAdaptationStrategy strategy, DM *adm, Vec *ax) > x is zero here. > Go back to, > *4) *PetscCall(SNESSolve(snes, NULL, u)); > The initial guess u is zero. It seems this is the reason. A little weird. > An intiial guess with zero should be resonable. > > It seems maybe something is not reasonable for the logic in > *DMPlexMetricNormalize.* > Is the solve not doing anything? We need at least one solution in order to adapt. Did you look at SNES ex27? I adaptively refine that one. Thanks, Matt > *Thanks,* > > > *Xiaodong * > > > > > > > > On Fri, Sep 13, 2024 at 2:22?PM Jed Brown wrote: > >> This error message is coming from the following: >> >> $ rg -B4 'Global metric normalization factor must be in' >> src/dm/impls/plex/plexmetric.c >> 1308- PetscCall(PetscDSSetObjective(ds, 0, detMFunc)); >> 1309- PetscCall(DMPlexComputeIntegralFEM(dmDet, determinant, >> &integral, NULL)); >> 1310- } >> 1311- realIntegral = PetscRealPart(integral); >> 1312: PetscCheck(realIntegral > 1.0e-30, PETSC_COMM_SELF, >> PETSC_ERR_ARG_OUTOFRANGE, "Global metric normalization factor must be in >> (0, inf). Is the input metric positive-definite?"); >> >> Perhaps you can independently check what integrand is being provided. >> It's probably zero or negative. You could apply this patch so the error >> message will report a value. >> >> >> diff --git i/src/dm/impls/plex/plexmetric.c >> w/src/dm/impls/plex/plexmetric.c >> index 61caeed28de..906cb394027 100644 >> --- i/src/dm/impls/plex/plexmetric.c >> +++ w/src/dm/impls/plex/plexmetric.c >> @@ -1309,7 +1309,7 @@ PetscErrorCode DMPlexMetricNormalize(DM dm, Vec >> metricIn, PetscBool restrictSize >> PetscCall(DMPlexComputeIntegralFEM(dmDet, determinant, &integral, >> NULL)); >> } >> realIntegral = PetscRealPart(integral); >> - PetscCheck(realIntegral > 1.0e-30, PETSC_COMM_SELF, >> PETSC_ERR_ARG_OUTOFRANGE, "Global metric normalization factor must be in >> (0, inf). Is the input metric positive-definite?"); >> + PetscCheck(realIntegral > 1.0e-30, PETSC_COMM_SELF, >> PETSC_ERR_ARG_OUTOFRANGE, "Global metric normalization factor %g must be in >> (0, inf). Is the input metric positive-definite?", (double)realIntegral); >> factGlob = PetscPowReal(target / realIntegral, 2.0 / dim); >> >> /* Apply local scaling */ >> >> >> neil liu writes: >> >> > Dear Petsc developers and users, >> > >> > I am trying to explore adaptive mesh refinement (tetradedra) with >> Petsc. >> > It seems case 12 ($PETSC DIR/src/snes/tutorials/ex12.c, from paper, >> > >> https://urldefense.us/v3/__https://arxiv.org/pdf/2201.02806__;!!G_uCfscf7eWS!ZEwaHkvYc_IrjycFr8LiNi-nZcFqu0ZpsydAAomxptQ6C0xkBs3qhn5ba31Z4vipKf4mTrqDm8A5S65DDD_xFg$ >> ) is a good example. >> > However when I tried, >> > ./ex12 -run_type full -dm_plex_dim 3 -dm_distribute -dm_plex_box_faces >> > 10,10,10 -bc_type dirichlet -petscspace_degree 1 -variable_coefficient >> ball >> > -snes_converged >> > _reason ::ascii_info_detail -ksp_type cg -pc_type sor >> -snes_adapt_sequence >> > 3 -adaptor_target_num 1000 -dm_plex_metric_h_max 0.5 -dm_adaptor mmg >> > L_2 Error: 1.55486 >> > Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE >> iterations 2 >> > >> > it shows the following error, >> > [0]PETSC ERROR: --------------------- Error Message >> > -------------------------------------------------------------- >> > [0]PETSC ERROR: Argument out of range >> > [0]PETSC ERROR: Global metric normalization factor must be in (0, inf). >> Is >> > the input metric positive-definite? >> > >> > Do you have any suggestions here? >> > >> > Thanks, >> > >> > Xiaodong >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!djDLOY8Rh_OQw8vhb4B7V0-zm0LToQcf2kEcTsKIYyUymnyy49THra-UsRtW-B_YH1Fm5t5I2bSpjfhAEoLt$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Mon Sep 16 07:41:41 2024 From: jroman at dsic.upv.es (Jose E. Roman) Date: Mon, 16 Sep 2024 12:41:41 +0000 Subject: [petsc-users] How to create an array of vectors of type Vec (for use with slepc EPSGetInvariantSubspace) In-Reply-To: References: <01AFD9AF-F905-435F-AEF0-CF1E6B793A5F@dsic.upv.es> Message-ID: <1A52C7FF-379B-4C8E-9B18-C6AA5B7D8F80@dsic.upv.es> Please respond to the list. The Fortran interface is not fully tested. In particular, this function is not employed in any of the Fortran examples. The Fortran interface has undergone successive improvements in recent times. I have tried the following: Vec, pointer :: subspace_receive(:) ... PetscCall(EPSGetConverged(eps, nconv, ierr)) PetscCall(MatCreateVecs(A, PETSC_NULL_VEC, xr, ierr)) PetscCall(VecDuplicateVecsF90(xr, nconv, subspace_receive, ierr)) PetscCall(EPSGetInvariantSubspace(eps, subspace_receive, ierr)) ... PetscCall(VecDestroyVecsF90(nconv, subspace_receive, ierr)) PetscCall(VecDestroy(xr, ierr)) I have tested this in the 'main' branch and it works as expected (note that 'main' will become version 3.22 in two weeks). You can try with version 3.21 and it may work, probably not in older versions. Jose > El 16 sept 2024, a las 11:07, dazza simplythebest escribi?: > > Dear All and Jose, > I am now trying to code up the extraction of the invariant subspace, but I seem to be running into > a compilation problem. I am using the slepc example file ex1f.F as a base, and have added the following lines: > > !Declaration > Vec subspace_receive > . > . > . > ! Executed code (following EPSSolve) > call EPSGetConverged(eps,nconv,ierr); CHKERRA(ierr) > call VecDuplicateVecs(xr, nconv, subspace_receive, ierr); CHKERRA(ierr) > call EPSGetInvariantSubspace(eps, subspace_receive, ierr) ; CHKERRA(ierr). > . > . > > However, I get the following compilation error when I include the call EPSGetInvariantSubspace(eps, subspace_receive, ierr) line: > > > minitest.F(133): error #8284: If the actual argument is scalar, the dummy argument shall be scalar unless the actual argument is of type character or is an element of an array that is not assumed shape, pointer, or polymorphic. [B] > call EPSGetInvariantSubspace(eps, subspace_receive, ierr) ; if (ierr .ne. 0) then;call PetscErrorF(ierr,133,"minitest.F");call MPIU_Abort(PETSC_COMM_SELF,ierr);endif > -----------^ > compilation aborted for minitest.F (code 1) > > > If I comment out the call EPSGetInvariantSubspace(eps, subspace_receive, ierr) line the code compiles, links and runs successfully. > I presume that I must be making a silly mistake in the above lines, but I can't quite see what it is. (There is some discussion in the > manual of there being a different situation if the matrix is real but the eigenvalues are complex, but for this case the eigenvalues > are real). Can you offer any guidance ?! I can provide the whole code of course if that would help. > Many thanks and best wishes, > Dan. > > > > > > From: Jose E. Roman > Sent: Monday, September 2, 2024 7:06 AM > To: dazza simplythebest > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] How to create an array of vectors of type Vec (for use with slepc EPSGetInvariantSubspace) > You can use VecDuplicateVecs() to create an array of Vec. > Yes, you can use VecLoad() to read the data from a binary file. > > Jose > > > > El 2 sept 2024, a las 8:56, dazza simplythebest escribi?: > > > > Dear All, > > I am seeking to perform a large number of eigenvalue calculations (related to a fluid dynamics > > problem) using the slepc framework. I have written the program that calls slepc and validated its solutions against > > known results for special cases - the Jacobi-Davidson iteration seemed to perform best on this problem. > > Since in the 'production-level' calculation I will step through a parameter space which should > > mean only small changes to the eigenproblem between each calculation, it seems to make sense > > to use the previous solution as the initial guess for the next calculation. > > > > The sequence to execute this strategy would seem to be: > > To write out the current soln subspace: > > > > call EPSGetInvariantSubspace > > [ call VecView -> binary file (if a file storage is required for possible restart)] > > > > To then read in this data to use it as the initial subspace for the next run: > > > > call EPSSetInitialSpace > > > > My question is that EPSGetInvariantSubspace needs an 'array of nconv vectors', how does one create > > such an array ? My understanding is that the Vec structure is just a one-dimensional array (with contents > > possibly scattered in non-contiguous memory locations !?) , so do we just need to create a long Vec > > that stores multiple vectors ? How can we do this (the examples I have seen thus far use a MatCreateVecs to specify the size > > of the Vec)? > > > > If loading such a subspace from a binary file (created maybe by VecView), presumably VecLoad is the > > best way to load such an array of vectors from the binary file ? > > > > Best wishes and many thanks, > > Dan. From liufield at gmail.com Mon Sep 16 10:10:34 2024 From: liufield at gmail.com (neil liu) Date: Mon, 16 Sep 2024 11:10:34 -0400 Subject: [petsc-users] Inquiry about metric-based adaptation using MMG in petsc In-Reply-To: References: <87o74rfm69.fsf@jedbrown.org> Message-ID: Hi, Mathew, It seems for version 3.21.1 and 3.21.5, there is not ex27 for sne/tutorial/. Then I cloned the petsc from gitlab. ex27 was found there. For this version, ex12 could be run without the aformentioned error. mpirun -n 8 ./ex12 -run_type full -dm_plex_dim 3 -dm_distribute -dm_plex_box_faces 10,10,10 -bc_type dirichlet -p etscspace_degree 1 -variable_coefficient ball -snes_converged_reason ::ascii_info_detail -ksp_type cg -pc_type sor -snes_adapt_sequence 3 -adaptor_target_num 1000 -dm_plex_metric_h_max 0.5 -dm_ad aptor parmmg -dm_plex_metric_target_complexity 3200 But the refinement (top one) still can be improved, right? Thanks, Xiaodong [image: test.png] [image: image.png] On Sun, Sep 15, 2024 at 9:33?AM Matthew Knepley wrote: > On Sun, Sep 15, 2024 at 7:25?AM neil liu wrote: > >> Thanks a lot, Jed. >> I traced the code and found the following issues, >> 1) in *PetscErrorCode DMPlexMetricNormalize(DM dm, Vec metricIn, >> PetscBool restrictSizes, PetscBool restrictAnisotropy, Vec metricOut, Vec >> determinant),* >> the metricIn is zero >> *. And this seems not reasonable. * >> *Go back to,* >> 2) DMAdaptorAdapt_Sequence_Private(DMAdaptor adaptor, Vec inx, PetscBool >> doSolve, DM *adm, Vec *ax) >> with -sol_adapt_loc_pre_view, -adapt_gradient_view and >> -adapt_hessian_view, all these three vectors are zero. >> This is not reasonable. >> >> *Go back to, * >> *3) *PetscErrorCode DMAdaptorAdapt(DMAdaptor adaptor, Vec x, >> DMAdaptationStrategy strategy, DM *adm, Vec *ax) >> x is zero here. >> Go back to, >> *4) *PetscCall(SNESSolve(snes, NULL, u)); >> The initial guess u is zero. It seems this is the reason. A little weird. >> An intiial guess with zero should be resonable. >> >> It seems maybe something is not reasonable for the logic in >> *DMPlexMetricNormalize.* >> > > Is the solve not doing anything? We need at least one solution in order to > adapt. Did you look at SNES ex27? I adaptively refine that one. > > Thanks, > > Matt > > >> *Thanks,* >> >> >> *Xiaodong * >> >> >> >> >> >> >> >> On Fri, Sep 13, 2024 at 2:22?PM Jed Brown wrote: >> >>> This error message is coming from the following: >>> >>> $ rg -B4 'Global metric normalization factor must be in' >>> src/dm/impls/plex/plexmetric.c >>> 1308- PetscCall(PetscDSSetObjective(ds, 0, detMFunc)); >>> 1309- PetscCall(DMPlexComputeIntegralFEM(dmDet, determinant, >>> &integral, NULL)); >>> 1310- } >>> 1311- realIntegral = PetscRealPart(integral); >>> 1312: PetscCheck(realIntegral > 1.0e-30, PETSC_COMM_SELF, >>> PETSC_ERR_ARG_OUTOFRANGE, "Global metric normalization factor must be in >>> (0, inf). Is the input metric positive-definite?"); >>> >>> Perhaps you can independently check what integrand is being provided. >>> It's probably zero or negative. You could apply this patch so the error >>> message will report a value. >>> >>> >>> diff --git i/src/dm/impls/plex/plexmetric.c >>> w/src/dm/impls/plex/plexmetric.c >>> index 61caeed28de..906cb394027 100644 >>> --- i/src/dm/impls/plex/plexmetric.c >>> +++ w/src/dm/impls/plex/plexmetric.c >>> @@ -1309,7 +1309,7 @@ PetscErrorCode DMPlexMetricNormalize(DM dm, Vec >>> metricIn, PetscBool restrictSize >>> PetscCall(DMPlexComputeIntegralFEM(dmDet, determinant, &integral, >>> NULL)); >>> } >>> realIntegral = PetscRealPart(integral); >>> - PetscCheck(realIntegral > 1.0e-30, PETSC_COMM_SELF, >>> PETSC_ERR_ARG_OUTOFRANGE, "Global metric normalization factor must be in >>> (0, inf). Is the input metric positive-definite?"); >>> + PetscCheck(realIntegral > 1.0e-30, PETSC_COMM_SELF, >>> PETSC_ERR_ARG_OUTOFRANGE, "Global metric normalization factor %g must be in >>> (0, inf). Is the input metric positive-definite?", (double)realIntegral); >>> factGlob = PetscPowReal(target / realIntegral, 2.0 / dim); >>> >>> /* Apply local scaling */ >>> >>> >>> neil liu writes: >>> >>> > Dear Petsc developers and users, >>> > >>> > I am trying to explore adaptive mesh refinement (tetradedra) with >>> Petsc. >>> > It seems case 12 ($PETSC DIR/src/snes/tutorials/ex12.c, from paper, >>> > >>> https://urldefense.us/v3/__https://arxiv.org/pdf/2201.02806__;!!G_uCfscf7eWS!ZEwaHkvYc_IrjycFr8LiNi-nZcFqu0ZpsydAAomxptQ6C0xkBs3qhn5ba31Z4vipKf4mTrqDm8A5S65DDD_xFg$ >>> ) is a good example. >>> > However when I tried, >>> > ./ex12 -run_type full -dm_plex_dim 3 -dm_distribute -dm_plex_box_faces >>> > 10,10,10 -bc_type dirichlet -petscspace_degree 1 -variable_coefficient >>> ball >>> > -snes_converged >>> > _reason ::ascii_info_detail -ksp_type cg -pc_type sor >>> -snes_adapt_sequence >>> > 3 -adaptor_target_num 1000 -dm_plex_metric_h_max 0.5 -dm_adaptor mmg >>> > L_2 Error: 1.55486 >>> > Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE >>> iterations 2 >>> > >>> > it shows the following error, >>> > [0]PETSC ERROR: --------------------- Error Message >>> > -------------------------------------------------------------- >>> > [0]PETSC ERROR: Argument out of range >>> > [0]PETSC ERROR: Global metric normalization factor must be in (0, >>> inf). Is >>> > the input metric positive-definite? >>> > >>> > Do you have any suggestions here? >>> > >>> > Thanks, >>> > >>> > Xiaodong >>> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!dmsOhOb8Z6LziJbOMbGSNL4APyr2KlT905erGKhckRCcL5hAeaopEC78FlFcKjqla_qzqTAyiHK9lZki1ilMFA$ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: test.png Type: image/png Size: 485255 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 1689600 bytes Desc: not available URL: From Pierre.LEDAC at cea.fr Mon Sep 16 10:51:23 2024 From: Pierre.LEDAC at cea.fr (LEDAC Pierre) Date: Mon, 16 Sep 2024 15:51:23 +0000 Subject: [petsc-users] [MPI GPU Aware] KSP_DIVERGED Message-ID: Hi all, We are using PETSc 3.20 in our code and running succesfully several solvers on Nvidia GPU with OpenMPI library which are not GPU aware (so I need to add the flag -use_gpu_aware_mpi 0). But now, when using OpenMPI GPU Aware library (OpenMPI 4.0.5 ou 4.1.5 from NVHPC), some parallel calculations failed with KSP_DIVERGED_ITS or KSP_DIVERGED_DTOL with several configurations. It may run wells on a small test case with (matrix is symmetric): -ksp_type cg -pc_type gamg -pc_gamg_type classical But suddenly with a number of devices for instance bigger than 4 or 8, it may fail. If I switch to another solver (BiCGstab), it may converge: -ksp_type bcgs -pc_type gamg -pc_gamg_type classical The more sensitive cases where it diverges are the following: -ksp_type cg -pc_type hypre -pc_hypre_type boomeramg -ksp_type cg -pc_type gamg -pc_gamg_type classical And the bcgs turnaroud doesn't work each time... It seems to work without problem with aggregation (at least 128 GPUs on my simulation): -ksp_type cg -pc_type gamg -pc_gamg_type agg So I guess there is a weird thing happening in my code during the solve in PETSc with MPI GPU Aware, as all the previous configurations works with non GPU aware MPI. Here is the -ksp_view log during one fail with the first configuration: KSP Object: () 8 MPI processes type: cg maximum iterations=10000, nonzero initial guess tolerances: relative=0., absolute=0.0001, divergence=10000. left preconditioning using UNPRECONDITIONED norm type for convergence test PC Object: () 8 MPI processes type: hypre HYPRE BoomerAMG preconditioning Cycle type V Maximum number of levels 25 Maximum number of iterations PER hypre call 1 Convergence tolerance PER hypre call 0. Threshold for strong coupling 0.7 Interpolation truncation factor 0. Interpolation: max elements per row 0 Number of levels of aggressive coarsening 0 Number of paths for aggressive coarsening 1 Maximum row sums 0.9 Sweeps down 1 Sweeps up 1 Sweeps on coarse 1 Relax down l1scaled-Jacobi Relax up l1scaled-Jacobi Relax on coarse Gaussian-elimination Relax weight (all) 1. Outer relax weight (all) 1. Maximum size of coarsest grid 9 Minimum size of coarsest grid 1 Not using CF-relaxation Not using more complex smoothers. Measure type local Coarsen type PMIS Interpolation type ext+i SpGEMM type cusparse linear system matrix = precond matrix: Mat Object: () 8 MPI processes type: mpiaijcusparse rows=64000, cols=64000 total: nonzeros=311040, allocated nonzeros=311040 total number of mallocs used during MatSetValues calls=0 not using I-node (on process 0) routines I didn't succeed for the moment creating a reproducer with ex.c examples... Did you see this kind of behaviour before? Should I update my PETSc version ? Thanks for any advice, Pierre LEDAC Commissariat ? l??nergie atomique et aux ?nergies alternatives Centre de SACLAY DES/ISAS/DM2S/SGLS/LCAN B?timent 451 ? point courrier n?43 F-91191 Gif-sur-Yvette +33 1 69 08 04 03 +33 6 83 42 05 79 -------------- next part -------------- An HTML attachment was scrubbed... URL: From junchao.zhang at gmail.com Mon Sep 16 11:14:21 2024 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Mon, 16 Sep 2024 11:14:21 -0500 Subject: [petsc-users] [MPI GPU Aware] KSP_DIVERGED In-Reply-To: References: Message-ID: Could you try petsc/main to see if the problem persists? --Junchao Zhang On Mon, Sep 16, 2024 at 10:51?AM LEDAC Pierre wrote: > Hi all, > > > We are using PETSc 3.20 in our code and running succesfully several > solvers on Nvidia GPU with OpenMPI library which are not GPU aware (so I > need to add the flag -use_gpu_aware_mpi 0). > > > But now, when using OpenMPI GPU Aware library (OpenMPI 4.0.5 ou 4.1.5 from > NVHPC), some parallel calculations failed with *KSP_DIVERGED_ITS* or > *KSP_DIVERGED_DTOL* > > with several configurations. It may run wells on a small test case with > (matrix is symmetric): > > > *-ksp_type cg -pc_type gamg -pc_gamg_type classical* > > > But suddenly with a number of devices for instance bigger than 4 or 8, it > may fail. > > > If I switch to another solver (BiCGstab), it may converge: > > > *-ksp_type bcgs -pc_type gamg -pc_gamg_type classical* > > > The more sensitive cases where it diverges are the following: > > > *-ksp_type cg -pc_type hypre -pc_hypre_type boomeramg * > > *-ksp_type cg -pc_type gamg -pc_gamg_type classical* > > > And the *bcgs* turnaroud doesn't work each time... > > > It seems to work without problem with aggregation (at least 128 GPUs on my > simulation): > > *-ksp_type cg -pc_type gamg -pc_gamg_type agg* > > > So I guess there is a weird thing happening in my code during the solve in > PETSc with MPI GPU Aware, as all the previous configurations works with non > GPU aware MPI. > > > Here is the -ksp_view log during one fail with the first configuration: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *KSP Object: () 8 MPI processes type: cg maximum iterations=10000, > nonzero initial guess tolerances: relative=0., absolute=0.0001, > divergence=10000. left preconditioning using UNPRECONDITIONED norm type > for convergence test PC Object: () 8 MPI processes type: hypre HYPRE > BoomerAMG preconditioning Cycle type V Maximum number of levels > 25 Maximum number of iterations PER hypre call 1 Convergence > tolerance PER hypre call 0. Threshold for strong coupling 0.7 > Interpolation truncation factor 0. Interpolation: max elements per > row 0 Number of levels of aggressive coarsening 0 Number of > paths for aggressive coarsening 1 Maximum row sums 0.9 Sweeps > down 1 Sweeps up 1 Sweeps on coarse 1 > Relax down l1scaled-Jacobi Relax up > l1scaled-Jacobi Relax on coarse Gaussian-elimination Relax > weight (all) 1. Outer relax weight (all) 1. Maximum size > of coarsest grid 9 Minimum size of coarsest grid 1 Not using > CF-relaxation Not using more complex smoothers. Measure > type local Coarsen type PMIS Interpolation type > ext+i SpGEMM type cusparse linear system matrix = precond > matrix: Mat Object: () 8 MPI processes type: mpiaijcusparse > rows=64000, cols=64000 total: nonzeros=311040, allocated > nonzeros=311040 total number of mallocs used during MatSetValues > calls=0 not using I-node (on process 0) routines* > > > I didn't succeed for the moment creating a reproducer with ex.c examples... > > > Did you see this kind of behaviour before? > > Should I update my PETSc version ? > > > Thanks for any advice, > > > Pierre LEDAC > Commissariat ? l??nergie atomique et aux ?nergies alternatives > Centre de SACLAY > DES/ISAS/DM2S/SGLS/LCAN > B?timent 451 ? point courrier n?43 > F-91191 Gif-sur-Yvette > +33 1 69 08 04 03 > +33 6 83 42 05 79 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay.anl at fastmail.org Mon Sep 16 11:57:02 2024 From: balay.anl at fastmail.org (Satish Balay) Date: Mon, 16 Sep 2024 11:57:02 -0500 (CDT) Subject: [petsc-users] [MPI GPU Aware] KSP_DIVERGED In-Reply-To: References: Message-ID: <99843d6c-206c-07bd-33a6-dce8e311643a@fastmail.org> And/Or - try latest OpenMPI [or MPICH] and see if that makes a difference. --download-mpich or --download-openmpi with latest petsc should build gpu-aware-mpi Satish On Mon, 16 Sep 2024, Junchao Zhang wrote: > Could you try petsc/main to see if the problem persists? > > --Junchao Zhang > > > On Mon, Sep 16, 2024 at 10:51?AM LEDAC Pierre wrote: > > > Hi all, > > > > > > We are using PETSc 3.20 in our code and running succesfully several > > solvers on Nvidia GPU with OpenMPI library which are not GPU aware (so I > > need to add the flag -use_gpu_aware_mpi 0). > > > > > > But now, when using OpenMPI GPU Aware library (OpenMPI 4.0.5 ou 4.1.5 from > > NVHPC), some parallel calculations failed with *KSP_DIVERGED_ITS* or > > *KSP_DIVERGED_DTOL* > > > > with several configurations. It may run wells on a small test case with > > (matrix is symmetric): > > > > > > *-ksp_type cg -pc_type gamg -pc_gamg_type classical* > > > > > > But suddenly with a number of devices for instance bigger than 4 or 8, it > > may fail. > > > > > > If I switch to another solver (BiCGstab), it may converge: > > > > > > *-ksp_type bcgs -pc_type gamg -pc_gamg_type classical* > > > > > > The more sensitive cases where it diverges are the following: > > > > > > *-ksp_type cg -pc_type hypre -pc_hypre_type boomeramg * > > > > *-ksp_type cg -pc_type gamg -pc_gamg_type classical* > > > > > > And the *bcgs* turnaroud doesn't work each time... > > > > > > It seems to work without problem with aggregation (at least 128 GPUs on my > > simulation): > > > > *-ksp_type cg -pc_type gamg -pc_gamg_type agg* > > > > > > So I guess there is a weird thing happening in my code during the solve in > > PETSc with MPI GPU Aware, as all the previous configurations works with non > > GPU aware MPI. > > > > > > Here is the -ksp_view log during one fail with the first configuration: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *KSP Object: () 8 MPI processes type: cg maximum iterations=10000, > > nonzero initial guess tolerances: relative=0., absolute=0.0001, > > divergence=10000. left preconditioning using UNPRECONDITIONED norm type > > for convergence test PC Object: () 8 MPI processes type: hypre HYPRE > > BoomerAMG preconditioning Cycle type V Maximum number of levels > > 25 Maximum number of iterations PER hypre call 1 Convergence > > tolerance PER hypre call 0. Threshold for strong coupling 0.7 > > Interpolation truncation factor 0. Interpolation: max elements per > > row 0 Number of levels of aggressive coarsening 0 Number of > > paths for aggressive coarsening 1 Maximum row sums 0.9 Sweeps > > down 1 Sweeps up 1 Sweeps on coarse 1 > > Relax down l1scaled-Jacobi Relax up > > l1scaled-Jacobi Relax on coarse Gaussian-elimination Relax > > weight (all) 1. Outer relax weight (all) 1. Maximum size > > of coarsest grid 9 Minimum size of coarsest grid 1 Not using > > CF-relaxation Not using more complex smoothers. Measure > > type local Coarsen type PMIS Interpolation type > > ext+i SpGEMM type cusparse linear system matrix = precond > > matrix: Mat Object: () 8 MPI processes type: mpiaijcusparse > > rows=64000, cols=64000 total: nonzeros=311040, allocated > > nonzeros=311040 total number of mallocs used during MatSetValues > > calls=0 not using I-node (on process 0) routines* > > > > > > I didn't succeed for the moment creating a reproducer with ex.c examples... > > > > > > Did you see this kind of behaviour before? > > > > Should I update my PETSc version ? > > > > > > Thanks for any advice, > > > > > > Pierre LEDAC > > Commissariat ? l??nergie atomique et aux ?nergies alternatives > > Centre de SACLAY > > DES/ISAS/DM2S/SGLS/LCAN > > B?timent 451 ? point courrier n?43 > > F-91191 Gif-sur-Yvette > > +33 1 69 08 04 03 > > +33 6 83 42 05 79 > > > From miguel.salazar at corintis.com Mon Sep 16 12:05:23 2024 From: miguel.salazar at corintis.com (miguel.salazar) Date: Mon, 16 Sep 2024 17:05:23 +0000 Subject: [petsc-users] Residual at the fieldsplit level References: <0e58dc19-e943-4d42-9a05-4cabbcbc8185.d0d10b7c-e93e-4ca5-8394-d4339a327c55.8303ace2-c862-49ea-97f8-20db7341a75a@emailsignatures365.codetwo.com> Message-ID: Hello, I have this simple example on Firedrake to illustrate my point. I am solving for a two-component poisson equation (uncoupled). Only the first component has a non-zero residual. ``` import firedrake as fd from firedrake import inner, grad, dx, sin, pi N = 10 mesh = fd.UnitSquareMesh(N, N) V = fd.FunctionSpace(mesh, "CG", 1) W = V * V u = fd.Function(W) v = fd.TestFunction(W) a = inner(grad(u[0]), grad(v[0])) * dx + inner(grad(u[1]), grad(v[1])) * dx x = fd.SpatialCoordinate(mesh) f = fd.Function(V) f.interpolate(fd.Constant(1e4) * sin(x[0] * pi) * sin(2 * x[1] * pi)) L = f * v[0] * dx F = a - L bcs = [fd.DirichletBC(W.sub(0), fd.Constant(2.0), (1,))] def snes_firedrake_residual(F, u, bcs): for bcs_ in bcs: bcs_.apply(u) residual = fd.assemble(F, bcs=bcs, zero_bc_nodes=True) with residual.dat.vec_ro as r: print("Initial residual:", r.norm()) snes_firedrake_residual(F, u, bcs) problem = fd.NonlinearVariationalProblem(F, u, bcs=bcs) solver_mumps_assembled = { "ksp_type": "preonly", "ksp_monitor": None, "pc_type": "python", "pc_python_type": "firedrake.AssembledPC", "assembled_pc_type": "lu", "assembled_pc_factor_mat_solver_type": "mumps", "assembled_mat_mumps_icntl_14": 200, "assembled_mat_mumps_icntl_24": 1, } solver_fieldsplit = { "mat_type": "matfree", "snes_type": "newtonls", "ksp_type": "fgmres", "ksp_rtol": 1e-1, "ksp_monitor": None, "pc_type": "fieldsplit", "pc_fieldsplit_type": "additive", "fieldsplit_0": solver_mumps_assembled, "fieldsplit_1": solver_mumps_assembled, } solver = fd.NonlinearVariationalSolver(problem, solver_parameters=solver_fieldsplit) solver.solve() ``` The PETSc output is as follow ``` Initial residual: 462.13689530272404 0 SNES Function norm 4.621368953027e+02 Residual norms for firedrake_0_ solve. 0 KSP Residual norm 4.621368953027e+02 Residual norms for firedrake_0_fieldsplit_0_ solve. 0 KSP Residual norm 1.000000000000e+00 1 KSP Residual norm 3.501082228626e-15 Residual norms for firedrake_0_fieldsplit_1_ solve. 0 KSP Residual norm 0.000000000000e+00 1 KSP Residual norm 0.000000000000e+00 1 KSP Residual norm 1.612167203819e-12 1 SNES Function norm 1.599350481360e-12 ``` Using the fieldsplit additive preconditioner, the problem converges in a single KSP iteration, as expected. However, I do not understand why the residual of fieldsplit_0 (1e+0) does not coincide with the outer residual (462.13689530272404). It should be the case given that only fieldsplit_0 has a non-zero residual contribution. The fact that it is just 1 is suspicious. Is there something about how the fieldsplit works that I am missing? Thanks, Miguel MIGUEL ANGEL SALAZAR DE TROYA Head of Software Engineering miguel.salazar at corintis.com Corintis SA EPFL Innovation Park Building C 1015 Lausanne [cid:2024-08-1609_02_11-re_mailmigrationfromgoogletooffice365-sebastien.gobel at corintis.com-co_4456a1bf-dd3e-46b7-85f9-21f278e66a79.png] Here at Corintis we care for your privacy. That is why we have taken appropriate measures to ensure that the data you have provided to us is always secure. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 2024-08-1609_02_11-re_mailmigrationfromgoogletooffice365-sebastien.gobel at corintis.com-co_4456a1bf-dd3e-46b7-85f9-21f278e66a79.png Type: image/png Size: 38219 bytes Desc: 2024-08-1609_02_11-re_mailmigrationfromgoogletooffice365-sebastien.gobel at corintis.com-co_4456a1bf-dd3e-46b7-85f9-21f278e66a79.png URL: From bsmith at petsc.dev Mon Sep 16 12:34:11 2024 From: bsmith at petsc.dev (Barry Smith) Date: Mon, 16 Sep 2024 13:34:11 -0400 Subject: [petsc-users] Residual at the fieldsplit level In-Reply-To: References: <0e58dc19-e943-4d42-9a05-4cabbcbc8185.d0d10b7c-e93e-4ca5-8394-d4339a327c55.8303ace2-c862-49ea-97f8-20db7341a75a@emailsignatures365.codetwo.com> Message-ID: <29C71683-CADA-4F66-8A0A-40459076323C@petsc.dev> Try ksp_monitor_true_residual. Let's see if it is an issue of preconditioned vs unpreconditioned residual. > On Sep 16, 2024, at 1:05?PM, miguel.salazar wrote: > > Hello, > > I have this simple example on Firedrake to illustrate my point. I am solving for a two-component poisson equation (uncoupled). Only the first component has a non-zero residual. > > ``` > import firedrake as fd > from firedrake import inner, grad, dx, sin, pi > > N = 10 > mesh = fd.UnitSquareMesh(N, N) > V = fd.FunctionSpace(mesh, "CG", 1) > W = V * V > u = fd.Function(W) > v = fd.TestFunction(W) > a = inner(grad(u[0]), grad(v[0])) * dx + inner(grad(u[1]), grad(v[1])) * dx > x = fd.SpatialCoordinate(mesh) > f = fd.Function(V) > f.interpolate(fd.Constant(1e4) * sin(x[0] * pi) * sin(2 * x[1] * pi)) > L = f * v[0] * dx > F = a - L > bcs = [fd.DirichletBC(W.sub(0), fd.Constant(2.0), (1,))] > > > def snes_firedrake_residual(F, u, bcs): > for bcs_ in bcs: > bcs_.apply(u) > residual = fd.assemble(F, bcs=bcs, zero_bc_nodes=True) > with residual.dat.vec_ro as r: > print("Initial residual:", r.norm()) > > > snes_firedrake_residual(F, u, bcs) > > > problem = fd.NonlinearVariationalProblem(F, u, bcs=bcs) > solver_mumps_assembled = { > "ksp_type": "preonly", > "ksp_monitor": None, > "pc_type": "python", > "pc_python_type": "firedrake.AssembledPC", > "assembled_pc_type": "lu", > "assembled_pc_factor_mat_solver_type": "mumps", > "assembled_mat_mumps_icntl_14": 200, > "assembled_mat_mumps_icntl_24": 1, > } > solver_fieldsplit = { > "mat_type": "matfree", > "snes_type": "newtonls", > "ksp_type": "fgmres", > "ksp_rtol": 1e-1, > "ksp_monitor": None, > "pc_type": "fieldsplit", > "pc_fieldsplit_type": "additive", > "fieldsplit_0": solver_mumps_assembled, > "fieldsplit_1": solver_mumps_assembled, > } > > solver = fd.NonlinearVariationalSolver(problem, solver_parameters=solver_fieldsplit) > solver.solve() > ``` > > The PETSc output is as follow > > ``` > Initial residual: 462.13689530272404 > 0 SNES Function norm 4.621368953027e+02 > Residual norms for firedrake_0_ solve. > 0 KSP Residual norm 4.621368953027e+02 > Residual norms for firedrake_0_fieldsplit_0_ solve. > 0 KSP Residual norm 1.000000000000e+00 > 1 KSP Residual norm 3.501082228626e-15 > Residual norms for firedrake_0_fieldsplit_1_ solve. > 0 KSP Residual norm 0.000000000000e+00 > 1 KSP Residual norm 0.000000000000e+00 > 1 KSP Residual norm 1.612167203819e-12 > 1 SNES Function norm 1.599350481360e-12 > ``` > > Using the fieldsplit additive preconditioner, the problem converges in a single KSP iteration, as expected. However, I do not understand why the residual of fieldsplit_0 (1e+0) does not coincide with the outer residual (462.13689530272404). It should be the case given that only fieldsplit_0 has a non-zero residual contribution. The fact that it is just 1 is suspicious. Is there something about how the fieldsplit works that I am missing? > > Thanks, > Miguel > > > > MIGUEL ANGEL SALAZAR DE TROYA > Head of Software Engineering > miguel.salazar at corintis.com > Corintis SA > EPFL Innovation Park Building C > 1015 Lausanne > <2024-08-1609_02_11-re_mailmigrationfromgoogletooffice365-sebastien.gobel at corintis.com-co_4456a1bf-dd3e-46b7-85f9-21f278e66a79.png> > Here at Corintis we care for your privacy. That is why we have taken appropriate measures to ensure that the data you have provided to us is always secure. -------------- next part -------------- An HTML attachment was scrubbed... URL: From miguel.salazar at corintis.com Tue Sep 17 00:04:01 2024 From: miguel.salazar at corintis.com (miguel.salazar) Date: Tue, 17 Sep 2024 05:04:01 +0000 Subject: [petsc-users] Residual at the fieldsplit level In-Reply-To: <29C71683-CADA-4F66-8A0A-40459076323C@petsc.dev> References: <0e58dc19-e943-4d42-9a05-4cabbcbc8185.d0d10b7c-e93e-4ca5-8394-d4339a327c55.8303ace2-c862-49ea-97f8-20db7341a75a@emailsignatures365.codetwo.com> <29C71683-CADA-4F66-8A0A-40459076323C@petsc.dev> Message-ID: These are the new options (with ksp_monitor_true_residual) solver_mumps_assembled = { "ksp_type": "preonly", "ksp_monitor": None, "ksp_monitor_true_residual": None, "pc_type": "python", "pc_python_type": "firedrake.AssembledPC", "assembled_pc_type": "lu", "assembled_pc_factor_mat_solver_type": "mumps", "assembled_mat_mumps_icntl_14": 200, "assembled_mat_mumps_icntl_24": 1, } solver_fieldsplit = { "mat_type": "matfree", "snes_monitor": None, "snes_type": "newtonls", "ksp_type": "fgmres", "ksp_monitor_true_residual": None, "ksp_rtol": 1e-1, "ksp_monitor": None, "pc_type": "fieldsplit", "pc_fieldsplit_type": "additive", "fieldsplit_0": solver_mumps_assembled, "fieldsplit_1": solver_mumps_assembled, } And this is the output Initial residual: 462.13689530272404 0 SNES Function norm 4.621368953027e+02 Residual norms for firedrake_0_ solve. 0 KSP Residual norm 4.621368953027e+02 Residual norms for firedrake_0_ solve. 0 KSP unpreconditioned resid norm 4.621368953027e+02 true resid norm 4.621368953027e+02 ||r(i)||/||b|| 1.000000000000e+00 Residual norms for firedrake_0_fieldsplit_0_ solve. 0 KSP Residual norm 1.000000000000e+00 Residual norms for firedrake_0_fieldsplit_0_ solve. 0 KSP none resid norm 1.000000000000e+00 true resid norm 3.501082228626e-15 ||r(i)||/||b|| 3.501082228626e-15 1 KSP Residual norm 3.501082228626e-15 1 KSP none resid norm 3.501082228626e-15 true resid norm 3.501082228626e-15 ||r(i)||/||b|| 3.501082228626e-15 Residual norms for firedrake_0_fieldsplit_1_ solve. 0 KSP Residual norm 0.000000000000e+00 Residual norms for firedrake_0_fieldsplit_1_ solve. 0 KSP none resid norm 0.000000000000e+00 true resid norm 0.000000000000e+00 ||r(i)||/||b|| inf 1 KSP Residual norm 0.000000000000e+00 1 KSP none resid norm 0.000000000000e+00 true resid norm 0.000000000000e+00 ||r(i)||/||b|| inf 1 KSP Residual norm 1.612167203819e-12 1 KSP unpreconditioned resid norm 1.612167203819e-12 true resid norm 1.589286585800e-12 ||r(i)||/||b|| 3.438995245681e-15 1 SNES Function norm 1.599350481360e-12 The true residual is very low. (3.501082228626e-15) MIGUEL ANGEL SALAZAR DE TROYA Head of Software Engineering miguel.salazar at corintis.com Corintis SA EPFL Innovation Park Building C 1015 Lausanne [cid:2024-08-1609_02_11-re_mailmigrationfromgoogletooffice365-sebastien.gobel at corintis.com-co_4456a1bf-dd3e-46b7-85f9-21f278e66a79.png] Here at Corintis we care for your privacy. That is why we have taken appropriate measures to ensure that the data you have provided to us is always secure. From: Barry Smith Date: Monday, 16 September 2024 at 19:34 To: miguel.salazar Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Residual at the fieldsplit level Try ksp_monitor_true_residual. Let's see if it is an issue of preconditioned vs unpreconditioned residual. On Sep 16, 2024, at 1:05?PM, miguel.salazar wrote: Hello, I have this simple example on Firedrake to illustrate my point. I am solving for a two-component poisson equation (uncoupled). Only the first component has a non-zero residual. ``` import firedrake as fd from firedrake import inner, grad, dx, sin, pi N = 10 mesh = fd.UnitSquareMesh(N, N) V = fd.FunctionSpace(mesh, "CG", 1) W = V * V u = fd.Function(W) v = fd.TestFunction(W) a = inner(grad(u[0]), grad(v[0])) * dx + inner(grad(u[1]), grad(v[1])) * dx x = fd.SpatialCoordinate(mesh) f = fd.Function(V) f.interpolate(fd.Constant(1e4) * sin(x[0] * pi) * sin(2 * x[1] * pi)) L = f * v[0] * dx F = a - L bcs = [fd.DirichletBC(W.sub(0), fd.Constant(2.0), (1,))] def snes_firedrake_residual(F, u, bcs): for bcs_ in bcs: bcs_.apply(u) residual = fd.assemble(F, bcs=bcs, zero_bc_nodes=True) with residual.dat.vec_ro as r: print("Initial residual:", r.norm()) snes_firedrake_residual(F, u, bcs) problem = fd.NonlinearVariationalProblem(F, u, bcs=bcs) solver_mumps_assembled = { "ksp_type": "preonly", "ksp_monitor": None, "pc_type": "python", "pc_python_type": "firedrake.AssembledPC", "assembled_pc_type": "lu", "assembled_pc_factor_mat_solver_type": "mumps", "assembled_mat_mumps_icntl_14": 200, "assembled_mat_mumps_icntl_24": 1, } solver_fieldsplit = { "mat_type": "matfree", "snes_type": "newtonls", "ksp_type": "fgmres", "ksp_rtol": 1e-1, "ksp_monitor": None, "pc_type": "fieldsplit", "pc_fieldsplit_type": "additive", "fieldsplit_0": solver_mumps_assembled, "fieldsplit_1": solver_mumps_assembled, } solver = fd.NonlinearVariationalSolver(problem, solver_parameters=solver_fieldsplit) solver.solve() ``` The PETSc output is as follow ``` Initial residual: 462.13689530272404 0 SNES Function norm 4.621368953027e+02 Residual norms for firedrake_0_ solve. 0 KSP Residual norm 4.621368953027e+02 Residual norms for firedrake_0_fieldsplit_0_ solve. 0 KSP Residual norm 1.000000000000e+00 1 KSP Residual norm 3.501082228626e-15 Residual norms for firedrake_0_fieldsplit_1_ solve. 0 KSP Residual norm 0.000000000000e+00 1 KSP Residual norm 0.000000000000e+00 1 KSP Residual norm 1.612167203819e-12 1 SNES Function norm 1.599350481360e-12 ``` Using the fieldsplit additive preconditioner, the problem converges in a single KSP iteration, as expected. However, I do not understand why the residual of fieldsplit_0 (1e+0) does not coincide with the outer residual (462.13689530272404). It should be the case given that only fieldsplit_0 has a non-zero residual contribution. The fact that it is just 1 is suspicious. Is there something about how the fieldsplit works that I am missing? Thanks, Miguel MIGUEL ANGEL SALAZAR DE TROYA Head of Software Engineering miguel.salazar at corintis.com Corintis SA EPFL Innovation Park Building C 1015 Lausanne <2024-08-1609_02_11-re_mailmigrationfromgoogletooffice365-sebastien.gobel at corintis.com-co_4456a1bf-dd3e-46b7-85f9-21f278e66a79.png> Here at Corintis we care for your privacy. That is why we have taken appropriate measures to ensure that the data you have provided to us is always secure. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 2024-08-1609_02_11-re_mailmigrationfromgoogletooffice365-sebastien.gobel at corintis.com-co_4456a1bf-dd3e-46b7-85f9-21f278e66a79.png Type: image/png Size: 38219 bytes Desc: 2024-08-1609_02_11-re_mailmigrationfromgoogletooffice365-sebastien.gobel at corintis.com-co_4456a1bf-dd3e-46b7-85f9-21f278e66a79.png URL: From Pierre.LEDAC at cea.fr Tue Sep 17 01:05:00 2024 From: Pierre.LEDAC at cea.fr (LEDAC Pierre) Date: Tue, 17 Sep 2024 06:05:00 +0000 Subject: [petsc-users] [MPI GPU Aware] KSP_DIVERGED In-Reply-To: <99843d6c-206c-07bd-33a6-dce8e311643a@fastmail.org> References: , <99843d6c-206c-07bd-33a6-dce8e311643a@fastmail.org> Message-ID: <7cb0513bb7a14f1eaca0a314b6e55749@cea.fr> Thanks all, I will try and report. Last question, if I use "-use_gpu_aware_mpi 0" flag with a MPI GPU Aware library, do PETSc disable GPU intra/inter communications and send MPI buffers as usual (with extra Device<->Host copies) ? Thanks, Pierre LEDAC Commissariat ? l??nergie atomique et aux ?nergies alternatives Centre de SACLAY DES/ISAS/DM2S/SGLS/LCAN B?timent 451 ? point courrier n?43 F-91191 Gif-sur-Yvette +33 1 69 08 04 03 +33 6 83 42 05 79 ________________________________ De : Satish Balay Envoy? : lundi 16 septembre 2024 18:57:02 ? : Junchao Zhang Cc : LEDAC Pierre; petsc-users at mcs.anl.gov; ROUMET Elie Objet : Re: [petsc-users] [MPI GPU Aware] KSP_DIVERGED And/Or - try latest OpenMPI [or MPICH] and see if that makes a difference. --download-mpich or --download-openmpi with latest petsc should build gpu-aware-mpi Satish On Mon, 16 Sep 2024, Junchao Zhang wrote: > Could you try petsc/main to see if the problem persists? > > --Junchao Zhang > > > On Mon, Sep 16, 2024 at 10:51?AM LEDAC Pierre wrote: > > > Hi all, > > > > > > We are using PETSc 3.20 in our code and running succesfully several > > solvers on Nvidia GPU with OpenMPI library which are not GPU aware (so I > > need to add the flag -use_gpu_aware_mpi 0). > > > > > > But now, when using OpenMPI GPU Aware library (OpenMPI 4.0.5 ou 4.1.5 from > > NVHPC), some parallel calculations failed with *KSP_DIVERGED_ITS* or > > *KSP_DIVERGED_DTOL* > > > > with several configurations. It may run wells on a small test case with > > (matrix is symmetric): > > > > > > *-ksp_type cg -pc_type gamg -pc_gamg_type classical* > > > > > > But suddenly with a number of devices for instance bigger than 4 or 8, it > > may fail. > > > > > > If I switch to another solver (BiCGstab), it may converge: > > > > > > *-ksp_type bcgs -pc_type gamg -pc_gamg_type classical* > > > > > > The more sensitive cases where it diverges are the following: > > > > > > *-ksp_type cg -pc_type hypre -pc_hypre_type boomeramg * > > > > *-ksp_type cg -pc_type gamg -pc_gamg_type classical* > > > > > > And the *bcgs* turnaroud doesn't work each time... > > > > > > It seems to work without problem with aggregation (at least 128 GPUs on my > > simulation): > > > > *-ksp_type cg -pc_type gamg -pc_gamg_type agg* > > > > > > So I guess there is a weird thing happening in my code during the solve in > > PETSc with MPI GPU Aware, as all the previous configurations works with non > > GPU aware MPI. > > > > > > Here is the -ksp_view log during one fail with the first configuration: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *KSP Object: () 8 MPI processes type: cg maximum iterations=10000, > > nonzero initial guess tolerances: relative=0., absolute=0.0001, > > divergence=10000. left preconditioning using UNPRECONDITIONED norm type > > for convergence test PC Object: () 8 MPI processes type: hypre HYPRE > > BoomerAMG preconditioning Cycle type V Maximum number of levels > > 25 Maximum number of iterations PER hypre call 1 Convergence > > tolerance PER hypre call 0. Threshold for strong coupling 0.7 > > Interpolation truncation factor 0. Interpolation: max elements per > > row 0 Number of levels of aggressive coarsening 0 Number of > > paths for aggressive coarsening 1 Maximum row sums 0.9 Sweeps > > down 1 Sweeps up 1 Sweeps on coarse 1 > > Relax down l1scaled-Jacobi Relax up > > l1scaled-Jacobi Relax on coarse Gaussian-elimination Relax > > weight (all) 1. Outer relax weight (all) 1. Maximum size > > of coarsest grid 9 Minimum size of coarsest grid 1 Not using > > CF-relaxation Not using more complex smoothers. Measure > > type local Coarsen type PMIS Interpolation type > > ext+i SpGEMM type cusparse linear system matrix = precond > > matrix: Mat Object: () 8 MPI processes type: mpiaijcusparse > > rows=64000, cols=64000 total: nonzeros=311040, allocated > > nonzeros=311040 total number of mallocs used during MatSetValues > > calls=0 not using I-node (on process 0) routines* > > > > > > I didn't succeed for the moment creating a reproducer with ex.c examples... > > > > > > Did you see this kind of behaviour before? > > > > Should I update my PETSc version ? > > > > > > Thanks for any advice, > > > > > > Pierre LEDAC > > Commissariat ? l??nergie atomique et aux ?nergies alternatives > > Centre de SACLAY > > DES/ISAS/DM2S/SGLS/LCAN > > B?timent 451 ? point courrier n?43 > > F-91191 Gif-sur-Yvette > > +33 1 69 08 04 03 > > +33 6 83 42 05 79 > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay.anl at fastmail.org Tue Sep 17 08:39:22 2024 From: balay.anl at fastmail.org (Satish Balay) Date: Tue, 17 Sep 2024 08:39:22 -0500 (CDT) Subject: [petsc-users] [MPI GPU Aware] KSP_DIVERGED In-Reply-To: <7cb0513bb7a14f1eaca0a314b6e55749@cea.fr> References: , <99843d6c-206c-07bd-33a6-dce8e311643a@fastmail.org> <7cb0513bb7a14f1eaca0a314b6e55749@cea.fr> Message-ID: <4544e89c-7611-dfc4-7295-15d57e2d9395@fastmail.org> On Tue, 17 Sep 2024, LEDAC Pierre wrote: > Thanks all, I will try and report. > > > Last question, if I use "-use_gpu_aware_mpi 0" flag with a MPI GPU Aware library, do PETSc > > disable GPU intra/inter communications and send MPI buffers as usual (with extra Device<->Host copies) ? Yes. Not: Wrt using MPI that is not GPU-aware - we are changing the default behavior - to not require "-use_gpu_aware_mpi 0" flag. https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/merge_requests/7813__;!!G_uCfscf7eWS!eioDrjl7DzCjQpMICHlaO-fyowQjitm2uYdZt51lAQidiOUu7zZDRKnRoN22zrcs7Dbi-Q3KQ8xrY9dwcOHaoulu9Wg$ Satish > > > Thanks, > > > Pierre LEDAC > Commissariat ? l??nergie atomique et aux ?nergies alternatives > Centre de SACLAY > DES/ISAS/DM2S/SGLS/LCAN > B?timent 451 ? point courrier n?43 > F-91191 Gif-sur-Yvette > +33 1 69 08 04 03 > +33 6 83 42 05 79 > ________________________________ > De : Satish Balay > Envoy? : lundi 16 septembre 2024 18:57:02 > ? : Junchao Zhang > Cc : LEDAC Pierre; petsc-users at mcs.anl.gov; ROUMET Elie > Objet : Re: [petsc-users] [MPI GPU Aware] KSP_DIVERGED > > And/Or - try latest OpenMPI [or MPICH] and see if that makes a difference. > > --download-mpich or --download-openmpi with latest petsc should build gpu-aware-mpi > > Satish > > On Mon, 16 Sep 2024, Junchao Zhang wrote: > > > Could you try petsc/main to see if the problem persists? > > > > --Junchao Zhang > > > > > > On Mon, Sep 16, 2024 at 10:51?AM LEDAC Pierre wrote: > > > > > Hi all, > > > > > > > > > We are using PETSc 3.20 in our code and running succesfully several > > > solvers on Nvidia GPU with OpenMPI library which are not GPU aware (so I > > > need to add the flag -use_gpu_aware_mpi 0). > > > > > > > > > But now, when using OpenMPI GPU Aware library (OpenMPI 4.0.5 ou 4.1.5 from > > > NVHPC), some parallel calculations failed with *KSP_DIVERGED_ITS* or > > > *KSP_DIVERGED_DTOL* > > > > > > with several configurations. It may run wells on a small test case with > > > (matrix is symmetric): > > > > > > > > > *-ksp_type cg -pc_type gamg -pc_gamg_type classical* > > > > > > > > > But suddenly with a number of devices for instance bigger than 4 or 8, it > > > may fail. > > > > > > > > > If I switch to another solver (BiCGstab), it may converge: > > > > > > > > > *-ksp_type bcgs -pc_type gamg -pc_gamg_type classical* > > > > > > > > > The more sensitive cases where it diverges are the following: > > > > > > > > > *-ksp_type cg -pc_type hypre -pc_hypre_type boomeramg * > > > > > > *-ksp_type cg -pc_type gamg -pc_gamg_type classical* > > > > > > > > > And the *bcgs* turnaroud doesn't work each time... > > > > > > > > > It seems to work without problem with aggregation (at least 128 GPUs on my > > > simulation): > > > > > > *-ksp_type cg -pc_type gamg -pc_gamg_type agg* > > > > > > > > > So I guess there is a weird thing happening in my code during the solve in > > > PETSc with MPI GPU Aware, as all the previous configurations works with non > > > GPU aware MPI. > > > > > > > > > Here is the -ksp_view log during one fail with the first configuration: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *KSP Object: () 8 MPI processes type: cg maximum iterations=10000, > > > nonzero initial guess tolerances: relative=0., absolute=0.0001, > > > divergence=10000. left preconditioning using UNPRECONDITIONED norm type > > > for convergence test PC Object: () 8 MPI processes type: hypre HYPRE > > > BoomerAMG preconditioning Cycle type V Maximum number of levels > > > 25 Maximum number of iterations PER hypre call 1 Convergence > > > tolerance PER hypre call 0. Threshold for strong coupling 0.7 > > > Interpolation truncation factor 0. Interpolation: max elements per > > > row 0 Number of levels of aggressive coarsening 0 Number of > > > paths for aggressive coarsening 1 Maximum row sums 0.9 Sweeps > > > down 1 Sweeps up 1 Sweeps on coarse 1 > > > Relax down l1scaled-Jacobi Relax up > > > l1scaled-Jacobi Relax on coarse Gaussian-elimination Relax > > > weight (all) 1. Outer relax weight (all) 1. Maximum size > > > of coarsest grid 9 Minimum size of coarsest grid 1 Not using > > > CF-relaxation Not using more complex smoothers. Measure > > > type local Coarsen type PMIS Interpolation type > > > ext+i SpGEMM type cusparse linear system matrix = precond > > > matrix: Mat Object: () 8 MPI processes type: mpiaijcusparse > > > rows=64000, cols=64000 total: nonzeros=311040, allocated > > > nonzeros=311040 total number of mallocs used during MatSetValues > > > calls=0 not using I-node (on process 0) routines* > > > > > > > > > I didn't succeed for the moment creating a reproducer with ex.c examples... > > > > > > > > > Did you see this kind of behaviour before? > > > > > > Should I update my PETSc version ? > > > > > > > > > Thanks for any advice, > > > > > > > > > Pierre LEDAC > > > Commissariat ? l??nergie atomique et aux ?nergies alternatives > > > Centre de SACLAY > > > DES/ISAS/DM2S/SGLS/LCAN > > > B?timent 451 ? point courrier n?43 > > > F-91191 Gif-sur-Yvette > > > +33 1 69 08 04 03 > > > +33 6 83 42 05 79 > > > > > > From bsmith at petsc.dev Tue Sep 17 09:24:38 2024 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 17 Sep 2024 10:24:38 -0400 Subject: [petsc-users] Residual at the fieldsplit level In-Reply-To: References: <0e58dc19-e943-4d42-9a05-4cabbcbc8185.d0d10b7c-e93e-4ca5-8394-d4339a327c55.8303ace2-c862-49ea-97f8-20db7341a75a@emailsignatures365.codetwo.com> <29C71683-CADA-4F66-8A0A-40459076323C@petsc.dev> Message-ID: <4E5EDC3B-0D39-4B7F-AEB7-7854B3BC9A24@petsc.dev> > Initial residual: 462.13689530272404 > 0 SNES Function norm 4.621368953027e+02 > Residual norms for firedrake_0_ solve. > 0 KSP Residual norm 4.621368953027e+02 > Residual norms for firedrake_0_fieldsplit_0_ solve. > 0 KSP Residual norm 1.000000000000e+00 > 1 KSP Residual norm 3.501082228626e-15 > Residual norms for firedrake_0_fieldsplit_1_ solve. > 0 KSP Residual norm 0.000000000000e+00 > 1 KSP Residual norm 0.000000000000e+00 > 1 KSP Residual norm 1.612167203819e-12 > 1 SNES Function norm 1.599350481360e-12 > ``` > > Using the fieldsplit additive preconditioner, the problem converges in a single KSP iteration, as expected. However, I do not understand why the residual of fieldsplit_0 (1e+0) does not coincide with the outer residual (462.13689530272404). It should be the case given that only fieldsplit_0 has a non-zero residual contribution. The fact that it is just 1 is suspicious. Is there something about how the fieldsplit works that I am missing? In my reading, it says > Residual norms for firedrake_0_ solve. > 0 KSP Residual norm 4.621368953027e+02 so the residual in the first (0th) split does correspond to the outer-most KSP residual norm (not printed but equal to the SNES at the first iteration of Newton. It is the residual norm of the first field of the first field that is 1 > Residual norms for firedrake_0_fieldsplit_0_ solve. > 0 KSP Residual norm 1.000000000000e+00 Or am I missing something Barry > Initial residual: 462.13689530272404 > 0 SNES Function norm 4.621368953027e+02 > Residual norms for firedrake_0_ solve. > 0 KSP Residual norm 4.621368953027e+02 > Residual norms for firedrake_0_ solve. > 0 KSP unpreconditioned resid norm 4.621368953027e+02 true resid norm 4.621368953027e+02 ||r(i)||/||b|| 1.000000000000e+00 > Residual norms for firedrake_0_fieldsplit_0_ solve. > 0 KSP Residual norm 1.000000000000e+00 > Residual norms for firedrake_0_fieldsplit_0_ solve. > 0 KSP none resid norm 1.000000000000e+00 true resid norm 3.501082228626e-15 ||r(i)||/||b|| 3.501082228626e-15 > 1 KSP Residual norm 3.501082228626e-15 > 1 KSP none resid norm 3.501082228626e-15 true resid norm 3.501082228626e-15 ||r(i)||/||b|| 3.501082228626e-15 > Residual norms for firedrake_0_fieldsplit_1_ solve. > 0 KSP Residual norm 0.000000000000e+00 > Residual norms for firedrake_0_fieldsplit_1_ solve. > 0 KSP none resid norm 0.000000000000e+00 true resid norm 0.000000000000e+00 ||r(i)||/||b|| inf > 1 KSP Residual norm 0.000000000000e+00 > 1 KSP none resid norm 0.000000000000e+00 true resid norm 0.000000000000e+00 ||r(i)||/||b|| inf > 1 KSP Residual norm 1.612167203819e-12 > 1 KSP unpreconditioned resid norm 1.612167203819e-12 true resid norm 1.589286585800e-12 ||r(i)||/||b|| 3.438995245681e-15 > 1 SNES Function norm 1.599350481360e-12 > On Sep 17, 2024, at 1:04?AM, miguel.salazar wrote: > > These are the new options (with ksp_monitor_true_residual) > > solver_mumps_assembled = { > "ksp_type": "preonly", > "ksp_monitor": None, > "ksp_monitor_true_residual": None, > "pc_type": "python", > "pc_python_type": "firedrake.AssembledPC", > "assembled_pc_type": "lu", > "assembled_pc_factor_mat_solver_type": "mumps", > "assembled_mat_mumps_icntl_14": 200, > "assembled_mat_mumps_icntl_24": 1, > } > solver_fieldsplit = { > "mat_type": "matfree", > "snes_monitor": None, > "snes_type": "newtonls", > "ksp_type": "fgmres", > "ksp_monitor_true_residual": None, > "ksp_rtol": 1e-1, > "ksp_monitor": None, > "pc_type": "fieldsplit", > "pc_fieldsplit_type": "additive", > "fieldsplit_0": solver_mumps_assembled, > "fieldsplit_1": solver_mumps_assembled, > } > > And this is the output > > Initial residual: 462.13689530272404 > 0 SNES Function norm 4.621368953027e+02 > Residual norms for firedrake_0_ solve. > 0 KSP Residual norm 4.621368953027e+02 > Residual norms for firedrake_0_ solve. > 0 KSP unpreconditioned resid norm 4.621368953027e+02 true resid norm 4.621368953027e+02 ||r(i)||/||b|| 1.000000000000e+00 > Residual norms for firedrake_0_fieldsplit_0_ solve. > 0 KSP Residual norm 1.000000000000e+00 > Residual norms for firedrake_0_fieldsplit_0_ solve. > 0 KSP none resid norm 1.000000000000e+00 true resid norm 3.501082228626e-15 ||r(i)||/||b|| 3.501082228626e-15 > 1 KSP Residual norm 3.501082228626e-15 > 1 KSP none resid norm 3.501082228626e-15 true resid norm 3.501082228626e-15 ||r(i)||/||b|| 3.501082228626e-15 > Residual norms for firedrake_0_fieldsplit_1_ solve. > 0 KSP Residual norm 0.000000000000e+00 > Residual norms for firedrake_0_fieldsplit_1_ solve. > 0 KSP none resid norm 0.000000000000e+00 true resid norm 0.000000000000e+00 ||r(i)||/||b|| inf > 1 KSP Residual norm 0.000000000000e+00 > 1 KSP none resid norm 0.000000000000e+00 true resid norm 0.000000000000e+00 ||r(i)||/||b|| inf > 1 KSP Residual norm 1.612167203819e-12 > 1 KSP unpreconditioned resid norm 1.612167203819e-12 true resid norm 1.589286585800e-12 ||r(i)||/||b|| 3.438995245681e-15 > 1 SNES Function norm 1.599350481360e-12 > > The true residual is very low. (3.501082228626e-15) > > > > > MIGUEL ANGEL SALAZAR DE TROYA > Head of Software Engineering > miguel.salazar at corintis.com > Corintis SA > EPFL Innovation Park Building C > 1015 Lausanne > <2024-08-1609_02_11-re_mailmigrationfromgoogletooffice365-sebastien.gobel at corintis.com-co_4456a1bf-dd3e-46b7-85f9-21f278e66a79.png> > Here at Corintis we care for your privacy. That is why we have taken appropriate measures to ensure that the data you have provided to us is always secure. > From: Barry Smith > > Date: Monday, 16 September 2024 at 19:34 > To: miguel.salazar > > Cc: petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] Residual at the fieldsplit level > > > Try ksp_monitor_true_residual. Let's see if it is an issue of preconditioned vs unpreconditioned residual. > > > > > On Sep 16, 2024, at 1:05?PM, miguel.salazar > wrote: > > Hello, > > I have this simple example on Firedrake to illustrate my point. I am solving for a two-component poisson equation (uncoupled). Only the first component has a non-zero residual. > > ``` > import firedrake as fd > from firedrake import inner, grad, dx, sin, pi > > N = 10 > mesh = fd.UnitSquareMesh(N, N) > V = fd.FunctionSpace(mesh, "CG", 1) > W = V * V > u = fd.Function(W) > v = fd.TestFunction(W) > a = inner(grad(u[0]), grad(v[0])) * dx + inner(grad(u[1]), grad(v[1])) * dx > x = fd.SpatialCoordinate(mesh) > f = fd.Function(V) > f.interpolate(fd.Constant(1e4) * sin(x[0] * pi) * sin(2 * x[1] * pi)) > L = f * v[0] * dx > F = a - L > bcs = [fd.DirichletBC(W.sub(0), fd.Constant(2.0), (1,))] > > > def snes_firedrake_residual(F, u, bcs): > for bcs_ in bcs: > bcs_.apply(u) > residual = fd.assemble(F, bcs=bcs, zero_bc_nodes=True) > with residual.dat.vec_ro as r: > print("Initial residual:", r.norm()) > > > snes_firedrake_residual(F, u, bcs) > > > problem = fd.NonlinearVariationalProblem(F, u, bcs=bcs) > solver_mumps_assembled = { > "ksp_type": "preonly", > "ksp_monitor": None, > "pc_type": "python", > "pc_python_type": "firedrake.AssembledPC", > "assembled_pc_type": "lu", > "assembled_pc_factor_mat_solver_type": "mumps", > "assembled_mat_mumps_icntl_14": 200, > "assembled_mat_mumps_icntl_24": 1, > } > solver_fieldsplit = { > "mat_type": "matfree", > "snes_type": "newtonls", > "ksp_type": "fgmres", > "ksp_rtol": 1e-1, > "ksp_monitor": None, > "pc_type": "fieldsplit", > "pc_fieldsplit_type": "additive", > "fieldsplit_0": solver_mumps_assembled, > "fieldsplit_1": solver_mumps_assembled, > } > > solver = fd.NonlinearVariationalSolver(problem, solver_parameters=solver_fieldsplit) > solver.solve() > ``` > > The PETSc output is as follow > > ``` > Initial residual: 462.13689530272404 > 0 SNES Function norm 4.621368953027e+02 > Residual norms for firedrake_0_ solve. > 0 KSP Residual norm 4.621368953027e+02 > Residual norms for firedrake_0_fieldsplit_0_ solve. > 0 KSP Residual norm 1.000000000000e+00 > 1 KSP Residual norm 3.501082228626e-15 > Residual norms for firedrake_0_fieldsplit_1_ solve. > 0 KSP Residual norm 0.000000000000e+00 > 1 KSP Residual norm 0.000000000000e+00 > 1 KSP Residual norm 1.612167203819e-12 > 1 SNES Function norm 1.599350481360e-12 > ``` > > Using the fieldsplit additive preconditioner, the problem converges in a single KSP iteration, as expected. However, I do not understand why the residual of fieldsplit_0 (1e+0) does not coincide with the outer residual (462.13689530272404). It should be the case given that only fieldsplit_0 has a non-zero residual contribution. The fact that it is just 1 is suspicious. Is there something about how the fieldsplit works that I am missing? > > Thanks, > Miguel > > MIGUEL ANGEL SALAZAR DE TROYA > Head of Software Engineering > miguel.salazar at corintis.com > Corintis SA > EPFL Innovation Park Building C > 1015 Lausanne > > <2024-08-1609_02_11-re_mailmigrationfromgoogletooffice365-sebastien.gobel at corintis.com-co_4456a1bf-dd3e-46b7-85f9-21f278e66a79.png> > Here at Corintis we care for your privacy. That is why we have taken appropriate measures to ensure that the data you have provided to us is always secure. -------------- next part -------------- An HTML attachment was scrubbed... URL: From miguel.salazar at corintis.com Tue Sep 17 09:37:48 2024 From: miguel.salazar at corintis.com (miguel.salazar) Date: Tue, 17 Sep 2024 14:37:48 +0000 Subject: [petsc-users] Residual at the fieldsplit level In-Reply-To: <4E5EDC3B-0D39-4B7F-AEB7-7854B3BC9A24@petsc.dev> References: <0e58dc19-e943-4d42-9a05-4cabbcbc8185.d0d10b7c-e93e-4ca5-8394-d4339a327c55.8303ace2-c862-49ea-97f8-20db7341a75a@emailsignatures365.codetwo.com> <29C71683-CADA-4F66-8A0A-40459076323C@petsc.dev> <4E5EDC3B-0D39-4B7F-AEB7-7854B3BC9A24@petsc.dev> Message-ID: >In my reading, it says >Residual norms for firedrake_0_ solve. > 0 KSP Residual norm 4.621368953027e+02 > so the residual in the first (0th) split does correspond to the outer-most KSP residual norm (not printed but equal to the SNES at the first iteration of Newton. > It is the residual norm of the first field of the first field that is 1 >Residual norms for firedrake_0_fieldsplit_0_ solve. > 0 KSP Residual norm 1.000000000000e+00 Residual norms for firedrake_0_ solve. 0 KSP Residual norm 4.621368953027e+02 Is the residual of the outer solve (it is being printed). Residual norms for firedrake_0_fieldsplit_0_ solve. 0 KSP Residual norm 1.000000000000e+00 Is the residual of the first field. I am not sure what you are referring to with ?the residual norm of the first field of the first field? Thanks, Miguel MIGUEL ANGEL SALAZAR DE TROYA Head of Software Engineering miguel.salazar at corintis.com Corintis SA EPFL Innovation Park Building C 1015 Lausanne [cid:2024-08-1609_02_11-re_mailmigrationfromgoogletooffice365-sebastien.gobel at corintis.com-co_4456a1bf-dd3e-46b7-85f9-21f278e66a79.png] Here at Corintis we care for your privacy. That is why we have taken appropriate measures to ensure that the data you have provided to us is always secure. From: Barry Smith Date: Tuesday, 17 September 2024 at 16:24 To: miguel.salazar Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Residual at the fieldsplit level Initial residual: 462.13689530272404 0 SNES Function norm 4.621368953027e+02 Residual norms for firedrake_0_ solve. 0 KSP Residual norm 4.621368953027e+02 Residual norms for firedrake_0_fieldsplit_0_ solve. 0 KSP Residual norm 1.000000000000e+00 1 KSP Residual norm 3.501082228626e-15 Residual norms for firedrake_0_fieldsplit_1_ solve. 0 KSP Residual norm 0.000000000000e+00 1 KSP Residual norm 0.000000000000e+00 1 KSP Residual norm 1.612167203819e-12 1 SNES Function norm 1.599350481360e-12 ``` Using the fieldsplit additive preconditioner, the problem converges in a single KSP iteration, as expected. However, I do not understand why the residual of fieldsplit_0 (1e+0) does not coincide with the outer residual (462.13689530272404). It should be the case given that only fieldsplit_0 has a non-zero residual contribution. The fact that it is just 1 is suspicious. Is there something about how the fieldsplit works that I am missing? In my reading, it says Residual norms for firedrake_0_ solve. 0 KSP Residual norm 4.621368953027e+02 so the residual in the first (0th) split does correspond to the outer-most KSP residual norm (not printed but equal to the SNES at the first iteration of Newton. It is the residual norm of the first field of the first field that is 1 Residual norms for firedrake_0_fieldsplit_0_ solve. 0 KSP Residual norm 1.000000000000e+00 Or am I missing something Barry Initial residual: 462.13689530272404 0 SNES Function norm 4.621368953027e+02 Residual norms for firedrake_0_ solve. 0 KSP Residual norm 4.621368953027e+02 Residual norms for firedrake_0_ solve. 0 KSP unpreconditioned resid norm 4.621368953027e+02 true resid norm 4.621368953027e+02 ||r(i)||/||b|| 1.000000000000e+00 Residual norms for firedrake_0_fieldsplit_0_ solve. 0 KSP Residual norm 1.000000000000e+00 Residual norms for firedrake_0_fieldsplit_0_ solve. 0 KSP none resid norm 1.000000000000e+00 true resid norm 3.501082228626e-15 ||r(i)||/||b|| 3.501082228626e-15 1 KSP Residual norm 3.501082228626e-15 1 KSP none resid norm 3.501082228626e-15 true resid norm 3.501082228626e-15 ||r(i)||/||b|| 3.501082228626e-15 Residual norms for firedrake_0_fieldsplit_1_ solve. 0 KSP Residual norm 0.000000000000e+00 Residual norms for firedrake_0_fieldsplit_1_ solve. 0 KSP none resid norm 0.000000000000e+00 true resid norm 0.000000000000e+00 ||r(i)||/||b|| inf 1 KSP Residual norm 0.000000000000e+00 1 KSP none resid norm 0.000000000000e+00 true resid norm 0.000000000000e+00 ||r(i)||/||b|| inf 1 KSP Residual norm 1.612167203819e-12 1 KSP unpreconditioned resid norm 1.612167203819e-12 true resid norm 1.589286585800e-12 ||r(i)||/||b|| 3.438995245681e-15 1 SNES Function norm 1.599350481360e-12 On Sep 17, 2024, at 1:04?AM, miguel.salazar wrote: These are the new options (with ksp_monitor_true_residual) solver_mumps_assembled = { "ksp_type": "preonly", "ksp_monitor": None, "ksp_monitor_true_residual": None, "pc_type": "python", "pc_python_type": "firedrake.AssembledPC", "assembled_pc_type": "lu", "assembled_pc_factor_mat_solver_type": "mumps", "assembled_mat_mumps_icntl_14": 200, "assembled_mat_mumps_icntl_24": 1, } solver_fieldsplit = { "mat_type": "matfree", "snes_monitor": None, "snes_type": "newtonls", "ksp_type": "fgmres", "ksp_monitor_true_residual": None, "ksp_rtol": 1e-1, "ksp_monitor": None, "pc_type": "fieldsplit", "pc_fieldsplit_type": "additive", "fieldsplit_0": solver_mumps_assembled, "fieldsplit_1": solver_mumps_assembled, } And this is the output Initial residual: 462.13689530272404 0 SNES Function norm 4.621368953027e+02 Residual norms for firedrake_0_ solve. 0 KSP Residual norm 4.621368953027e+02 Residual norms for firedrake_0_ solve. 0 KSP unpreconditioned resid norm 4.621368953027e+02 true resid norm 4.621368953027e+02 ||r(i)||/||b|| 1.000000000000e+00 Residual norms for firedrake_0_fieldsplit_0_ solve. 0 KSP Residual norm 1.000000000000e+00 Residual norms for firedrake_0_fieldsplit_0_ solve. 0 KSP none resid norm 1.000000000000e+00 true resid norm 3.501082228626e-15 ||r(i)||/||b|| 3.501082228626e-15 1 KSP Residual norm 3.501082228626e-15 1 KSP none resid norm 3.501082228626e-15 true resid norm 3.501082228626e-15 ||r(i)||/||b|| 3.501082228626e-15 Residual norms for firedrake_0_fieldsplit_1_ solve. 0 KSP Residual norm 0.000000000000e+00 Residual norms for firedrake_0_fieldsplit_1_ solve. 0 KSP none resid norm 0.000000000000e+00 true resid norm 0.000000000000e+00 ||r(i)||/||b|| inf 1 KSP Residual norm 0.000000000000e+00 1 KSP none resid norm 0.000000000000e+00 true resid norm 0.000000000000e+00 ||r(i)||/||b|| inf 1 KSP Residual norm 1.612167203819e-12 1 KSP unpreconditioned resid norm 1.612167203819e-12 true resid norm 1.589286585800e-12 ||r(i)||/||b|| 3.438995245681e-15 1 SNES Function norm 1.599350481360e-12 The true residual is very low. (3.501082228626e-15) MIGUEL ANGEL SALAZAR DE TROYA Head of Software Engineering miguel.salazar at corintis.com Corintis SA EPFL Innovation Park Building C 1015 Lausanne <2024-08-1609_02_11-re_mailmigrationfromgoogletooffice365-sebastien.gobel at corintis.com-co_4456a1bf-dd3e-46b7-85f9-21f278e66a79.png> Here at Corintis we care for your privacy. That is why we have taken appropriate measures to ensure that the data you have provided to us is always secure. From: Barry Smith > Date: Monday, 16 September 2024 at 19:34 To: miguel.salazar > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Residual at the fieldsplit level Try ksp_monitor_true_residual. Let's see if it is an issue of preconditioned vs unpreconditioned residual. On Sep 16, 2024, at 1:05?PM, miguel.salazar > wrote: Hello, I have this simple example on Firedrake to illustrate my point. I am solving for a two-component poisson equation (uncoupled). Only the first component has a non-zero residual. ``` import firedrake as fd from firedrake import inner, grad, dx, sin, pi N = 10 mesh = fd.UnitSquareMesh(N, N) V = fd.FunctionSpace(mesh, "CG", 1) W = V * V u = fd.Function(W) v = fd.TestFunction(W) a = inner(grad(u[0]), grad(v[0])) * dx + inner(grad(u[1]), grad(v[1])) * dx x = fd.SpatialCoordinate(mesh) f = fd.Function(V) f.interpolate(fd.Constant(1e4) * sin(x[0] * pi) * sin(2 * x[1] * pi)) L = f * v[0] * dx F = a - L bcs = [fd.DirichletBC(W.sub(0), fd.Constant(2.0), (1,))] def snes_firedrake_residual(F, u, bcs): for bcs_ in bcs: bcs_.apply(u) residual = fd.assemble(F, bcs=bcs, zero_bc_nodes=True) with residual.dat.vec_ro as r: print("Initial residual:", r.norm()) snes_firedrake_residual(F, u, bcs) problem = fd.NonlinearVariationalProblem(F, u, bcs=bcs) solver_mumps_assembled = { "ksp_type": "preonly", "ksp_monitor": None, "pc_type": "python", "pc_python_type": "firedrake.AssembledPC", "assembled_pc_type": "lu", "assembled_pc_factor_mat_solver_type": "mumps", "assembled_mat_mumps_icntl_14": 200, "assembled_mat_mumps_icntl_24": 1, } solver_fieldsplit = { "mat_type": "matfree", "snes_type": "newtonls", "ksp_type": "fgmres", "ksp_rtol": 1e-1, "ksp_monitor": None, "pc_type": "fieldsplit", "pc_fieldsplit_type": "additive", "fieldsplit_0": solver_mumps_assembled, "fieldsplit_1": solver_mumps_assembled, } solver = fd.NonlinearVariationalSolver(problem, solver_parameters=solver_fieldsplit) solver.solve() ``` The PETSc output is as follow ``` Initial residual: 462.13689530272404 0 SNES Function norm 4.621368953027e+02 Residual norms for firedrake_0_ solve. 0 KSP Residual norm 4.621368953027e+02 Residual norms for firedrake_0_fieldsplit_0_ solve. 0 KSP Residual norm 1.000000000000e+00 1 KSP Residual norm 3.501082228626e-15 Residual norms for firedrake_0_fieldsplit_1_ solve. 0 KSP Residual norm 0.000000000000e+00 1 KSP Residual norm 0.000000000000e+00 1 KSP Residual norm 1.612167203819e-12 1 SNES Function norm 1.599350481360e-12 ``` Using the fieldsplit additive preconditioner, the problem converges in a single KSP iteration, as expected. However, I do not understand why the residual of fieldsplit_0 (1e+0) does not coincide with the outer residual (462.13689530272404). It should be the case given that only fieldsplit_0 has a non-zero residual contribution. The fact that it is just 1 is suspicious. Is there something about how the fieldsplit works that I am missing? Thanks, Miguel MIGUEL ANGEL SALAZAR DE TROYA Head of Software Engineering miguel.salazar at corintis.com Corintis SA EPFL Innovation Park Building C 1015 Lausanne <2024-08-1609_02_11-re_mailmigrationfromgoogletooffice365-sebastien.gobel at corintis.com-co_4456a1bf-dd3e-46b7-85f9-21f278e66a79.png> Here at Corintis we care for your privacy. That is why we have taken appropriate measures to ensure that the data you have provided to us is always secure. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 2024-08-1609_02_11-re_mailmigrationfromgoogletooffice365-sebastien.gobel at corintis.com-co_4456a1bf-dd3e-46b7-85f9-21f278e66a79.png Type: image/png Size: 38219 bytes Desc: 2024-08-1609_02_11-re_mailmigrationfromgoogletooffice365-sebastien.gobel at corintis.com-co_4456a1bf-dd3e-46b7-85f9-21f278e66a79.png URL: From Pierre.LEDAC at cea.fr Tue Sep 17 09:47:48 2024 From: Pierre.LEDAC at cea.fr (LEDAC Pierre) Date: Tue, 17 Sep 2024 14:47:48 +0000 Subject: [petsc-users] [MPI GPU Aware] KSP_DIVERGED In-Reply-To: <4544e89c-7611-dfc4-7295-15d57e2d9395@fastmail.org> References: , <99843d6c-206c-07bd-33a6-dce8e311643a@fastmail.org> <7cb0513bb7a14f1eaca0a314b6e55749@cea.fr>, <4544e89c-7611-dfc4-7295-15d57e2d9395@fastmail.org> Message-ID: Thanks Satish, and nice guess for OpenMPI 5 ! It seems it solves the issue (at least on my GPU box where I reproduced the issue with 8 MPI ranks with OpenMPI 4.x). Unhappily, all the clusters we currently use have no module with OpenMPI 5.x. Seems I need to build it to really confirm. Probably we will prevent users from configuring our code with OpenMPI-cuda 4.x cause it is really a weird bug. Pierre LEDAC Commissariat ? l??nergie atomique et aux ?nergies alternatives Centre de SACLAY DES/ISAS/DM2S/SGLS/LCAN B?timent 451 ? point courrier n?43 F-91191 Gif-sur-Yvette +33 1 69 08 04 03 +33 6 83 42 05 79 ________________________________ De : Satish Balay Envoy? : mardi 17 septembre 2024 15:39:22 ? : LEDAC Pierre Cc : Junchao Zhang; petsc-users; ROUMET Elie Objet : Re: [petsc-users] [MPI GPU Aware] KSP_DIVERGED On Tue, 17 Sep 2024, LEDAC Pierre wrote: > Thanks all, I will try and report. > > > Last question, if I use "-use_gpu_aware_mpi 0" flag with a MPI GPU Aware library, do PETSc > > disable GPU intra/inter communications and send MPI buffers as usual (with extra Device<->Host copies) ? Yes. Not: Wrt using MPI that is not GPU-aware - we are changing the default behavior - to not require "-use_gpu_aware_mpi 0" flag. https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/merge_requests/7813__;!!G_uCfscf7eWS!Y40I3F5rvv3KWuJGoo1EdEDq62cnqn2R1Do7WFeicCnNbTaWDHj9SieJOPp2DkxXCkl3iHo8XwantIfdGcpZA8lvcoZH$ Satish > > > Thanks, > > > Pierre LEDAC > Commissariat ? l??nergie atomique et aux ?nergies alternatives > Centre de SACLAY > DES/ISAS/DM2S/SGLS/LCAN > B?timent 451 ? point courrier n?43 > F-91191 Gif-sur-Yvette > +33 1 69 08 04 03 > +33 6 83 42 05 79 > ________________________________ > De : Satish Balay > Envoy? : lundi 16 septembre 2024 18:57:02 > ? : Junchao Zhang > Cc : LEDAC Pierre; petsc-users at mcs.anl.gov; ROUMET Elie > Objet : Re: [petsc-users] [MPI GPU Aware] KSP_DIVERGED > > And/Or - try latest OpenMPI [or MPICH] and see if that makes a difference. > > --download-mpich or --download-openmpi with latest petsc should build gpu-aware-mpi > > Satish > > On Mon, 16 Sep 2024, Junchao Zhang wrote: > > > Could you try petsc/main to see if the problem persists? > > > > --Junchao Zhang > > > > > > On Mon, Sep 16, 2024 at 10:51?AM LEDAC Pierre wrote: > > > > > Hi all, > > > > > > > > > We are using PETSc 3.20 in our code and running succesfully several > > > solvers on Nvidia GPU with OpenMPI library which are not GPU aware (so I > > > need to add the flag -use_gpu_aware_mpi 0). > > > > > > > > > But now, when using OpenMPI GPU Aware library (OpenMPI 4.0.5 ou 4.1.5 from > > > NVHPC), some parallel calculations failed with *KSP_DIVERGED_ITS* or > > > *KSP_DIVERGED_DTOL* > > > > > > with several configurations. It may run wells on a small test case with > > > (matrix is symmetric): > > > > > > > > > *-ksp_type cg -pc_type gamg -pc_gamg_type classical* > > > > > > > > > But suddenly with a number of devices for instance bigger than 4 or 8, it > > > may fail. > > > > > > > > > If I switch to another solver (BiCGstab), it may converge: > > > > > > > > > *-ksp_type bcgs -pc_type gamg -pc_gamg_type classical* > > > > > > > > > The more sensitive cases where it diverges are the following: > > > > > > > > > *-ksp_type cg -pc_type hypre -pc_hypre_type boomeramg * > > > > > > *-ksp_type cg -pc_type gamg -pc_gamg_type classical* > > > > > > > > > And the *bcgs* turnaroud doesn't work each time... > > > > > > > > > It seems to work without problem with aggregation (at least 128 GPUs on my > > > simulation): > > > > > > *-ksp_type cg -pc_type gamg -pc_gamg_type agg* > > > > > > > > > So I guess there is a weird thing happening in my code during the solve in > > > PETSc with MPI GPU Aware, as all the previous configurations works with non > > > GPU aware MPI. > > > > > > > > > Here is the -ksp_view log during one fail with the first configuration: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *KSP Object: () 8 MPI processes type: cg maximum iterations=10000, > > > nonzero initial guess tolerances: relative=0., absolute=0.0001, > > > divergence=10000. left preconditioning using UNPRECONDITIONED norm type > > > for convergence test PC Object: () 8 MPI processes type: hypre HYPRE > > > BoomerAMG preconditioning Cycle type V Maximum number of levels > > > 25 Maximum number of iterations PER hypre call 1 Convergence > > > tolerance PER hypre call 0. Threshold for strong coupling 0.7 > > > Interpolation truncation factor 0. Interpolation: max elements per > > > row 0 Number of levels of aggressive coarsening 0 Number of > > > paths for aggressive coarsening 1 Maximum row sums 0.9 Sweeps > > > down 1 Sweeps up 1 Sweeps on coarse 1 > > > Relax down l1scaled-Jacobi Relax up > > > l1scaled-Jacobi Relax on coarse Gaussian-elimination Relax > > > weight (all) 1. Outer relax weight (all) 1. Maximum size > > > of coarsest grid 9 Minimum size of coarsest grid 1 Not using > > > CF-relaxation Not using more complex smoothers. Measure > > > type local Coarsen type PMIS Interpolation type > > > ext+i SpGEMM type cusparse linear system matrix = precond > > > matrix: Mat Object: () 8 MPI processes type: mpiaijcusparse > > > rows=64000, cols=64000 total: nonzeros=311040, allocated > > > nonzeros=311040 total number of mallocs used during MatSetValues > > > calls=0 not using I-node (on process 0) routines* > > > > > > > > > I didn't succeed for the moment creating a reproducer with ex.c examples... > > > > > > > > > Did you see this kind of behaviour before? > > > > > > Should I update my PETSc version ? > > > > > > > > > Thanks for any advice, > > > > > > > > > Pierre LEDAC > > > Commissariat ? l??nergie atomique et aux ?nergies alternatives > > > Centre de SACLAY > > > DES/ISAS/DM2S/SGLS/LCAN > > > B?timent 451 ? point courrier n?43 > > > F-91191 Gif-sur-Yvette > > > +33 1 69 08 04 03 > > > +33 6 83 42 05 79 > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Tue Sep 17 09:49:25 2024 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 17 Sep 2024 10:49:25 -0400 Subject: [petsc-users] Residual at the fieldsplit level In-Reply-To: References: <0e58dc19-e943-4d42-9a05-4cabbcbc8185.d0d10b7c-e93e-4ca5-8394-d4339a327c55.8303ace2-c862-49ea-97f8-20db7341a75a@emailsignatures365.codetwo.com> <29C71683-CADA-4F66-8A0A-40459076323C@petsc.dev> <4E5EDC3B-0D39-4B7F-AEB7-7854B3BC9A24@petsc.dev> Message-ID: <4131CBF5-CDF8-44E8-A994-0BFE57C1DB40@petsc.dev> So firedrake_0 is the object prefix of your SNES object? > On Sep 17, 2024, at 10:37?AM, miguel.salazar wrote: > > >In my reading, it says > >Residual norms for firedrake_0_ solve. > > 0 KSP Residual norm 4.621368953027e+02 > > > so the residual in the first (0th) split does correspond to the outer-most KSP residual norm (not printed but equal to the SNES at the first iteration of Newton. > > > It is the residual norm of the first field of the first field that is 1 > > >Residual norms for firedrake_0_fieldsplit_0_ solve. > > 0 KSP Residual norm 1.000000000000e+00 > > Residual norms for firedrake_0_ solve. > 0 KSP Residual norm 4.621368953027e+02 > > Is the residual of the outer solve (it is being printed). > > Residual norms for firedrake_0_fieldsplit_0_ solve. > 0 KSP Residual norm 1.000000000000e+00 > > Is the residual of the first field. I am not sure what you are referring to with ?the residual norm of the first field of the first field? > > Thanks, > Miguel > > > > > > > MIGUEL ANGEL SALAZAR DE TROYA > Head of Software Engineering > miguel.salazar at corintis.com > Corintis SA > EPFL Innovation Park Building C > 1015 Lausanne > <2024-08-1609_02_11-re_mailmigrationfromgoogletooffice365-sebastien.gobel at corintis.com-co_4456a1bf-dd3e-46b7-85f9-21f278e66a79.png> > Here at Corintis we care for your privacy. That is why we have taken appropriate measures to ensure that the data you have provided to us is always secure. > From: Barry Smith > > Date: Tuesday, 17 September 2024 at 16:24 > To: miguel.salazar > > Cc: petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] Residual at the fieldsplit level > > > Initial residual: 462.13689530272404 > 0 SNES Function norm 4.621368953027e+02 > Residual norms for firedrake_0_ solve. > 0 KSP Residual norm 4.621368953027e+02 > Residual norms for firedrake_0_fieldsplit_0_ solve. > 0 KSP Residual norm 1.000000000000e+00 > 1 KSP Residual norm 3.501082228626e-15 > Residual norms for firedrake_0_fieldsplit_1_ solve. > 0 KSP Residual norm 0.000000000000e+00 > 1 KSP Residual norm 0.000000000000e+00 > 1 KSP Residual norm 1.612167203819e-12 > 1 SNES Function norm 1.599350481360e-12 > ``` > > Using the fieldsplit additive preconditioner, the problem converges in a single KSP iteration, as expected. However, I do not understand why the residual of fieldsplit_0 (1e+0) does not coincide with the outer residual (462.13689530272404). It should be the case given that only fieldsplit_0 has a non-zero residual contribution. The fact that it is just 1 is suspicious. Is there something about how the fieldsplit works that I am missing? > > In my reading, it says > Residual norms for firedrake_0_ solve. > 0 KSP Residual norm 4.621368953027e+02 > > so the residual in the first (0th) split does correspond to the outer-most KSP residual norm (not printed but equal to the SNES at the first iteration of Newton. > > It is the residual norm of the first field of the first field that is 1 > > Residual norms for firedrake_0_fieldsplit_0_ solve. > 0 KSP Residual norm 1.000000000000e+00 > > Or am I missing something > > Barry > > > > > > > > Initial residual: 462.13689530272404 > 0 SNES Function norm 4.621368953027e+02 > Residual norms for firedrake_0_ solve. > 0 KSP Residual norm 4.621368953027e+02 > Residual norms for firedrake_0_ solve. > 0 KSP unpreconditioned resid norm 4.621368953027e+02 true resid norm 4.621368953027e+02 ||r(i)||/||b|| 1.000000000000e+00 > Residual norms for firedrake_0_fieldsplit_0_ solve. > 0 KSP Residual norm 1.000000000000e+00 > Residual norms for firedrake_0_fieldsplit_0_ solve. > 0 KSP none resid norm 1.000000000000e+00 true resid norm 3.501082228626e-15 ||r(i)||/||b|| 3.501082228626e-15 > 1 KSP Residual norm 3.501082228626e-15 > 1 KSP none resid norm 3.501082228626e-15 true resid norm 3.501082228626e-15 ||r(i)||/||b|| 3.501082228626e-15 > Residual norms for firedrake_0_fieldsplit_1_ solve. > 0 KSP Residual norm 0.000000000000e+00 > Residual norms for firedrake_0_fieldsplit_1_ solve. > 0 KSP none resid norm 0.000000000000e+00 true resid norm 0.000000000000e+00 ||r(i)||/||b|| inf > 1 KSP Residual norm 0.000000000000e+00 > 1 KSP none resid norm 0.000000000000e+00 true resid norm 0.000000000000e+00 ||r(i)||/||b|| inf > 1 KSP Residual norm 1.612167203819e-12 > 1 KSP unpreconditioned resid norm 1.612167203819e-12 true resid norm 1.589286585800e-12 ||r(i)||/||b|| 3.438995245681e-15 > 1 SNES Function norm 1.599350481360e-12 > > > > On Sep 17, 2024, at 1:04?AM, miguel.salazar > wrote: > > These are the new options (with ksp_monitor_true_residual) > > solver_mumps_assembled = { > "ksp_type": "preonly", > "ksp_monitor": None, > "ksp_monitor_true_residual": None, > "pc_type": "python", > "pc_python_type": "firedrake.AssembledPC", > "assembled_pc_type": "lu", > "assembled_pc_factor_mat_solver_type": "mumps", > "assembled_mat_mumps_icntl_14": 200, > "assembled_mat_mumps_icntl_24": 1, > } > solver_fieldsplit = { > "mat_type": "matfree", > "snes_monitor": None, > "snes_type": "newtonls", > "ksp_type": "fgmres", > "ksp_monitor_true_residual": None, > "ksp_rtol": 1e-1, > "ksp_monitor": None, > "pc_type": "fieldsplit", > "pc_fieldsplit_type": "additive", > "fieldsplit_0": solver_mumps_assembled, > "fieldsplit_1": solver_mumps_assembled, > } > > And this is the output > > Initial residual: 462.13689530272404 > 0 SNES Function norm 4.621368953027e+02 > Residual norms for firedrake_0_ solve. > 0 KSP Residual norm 4.621368953027e+02 > Residual norms for firedrake_0_ solve. > 0 KSP unpreconditioned resid norm 4.621368953027e+02 true resid norm 4.621368953027e+02 ||r(i)||/||b|| 1.000000000000e+00 > Residual norms for firedrake_0_fieldsplit_0_ solve. > 0 KSP Residual norm 1.000000000000e+00 > Residual norms for firedrake_0_fieldsplit_0_ solve. > 0 KSP none resid norm 1.000000000000e+00 true resid norm 3.501082228626e-15 ||r(i)||/||b|| 3.501082228626e-15 > 1 KSP Residual norm 3.501082228626e-15 > 1 KSP none resid norm 3.501082228626e-15 true resid norm 3.501082228626e-15 ||r(i)||/||b|| 3.501082228626e-15 > Residual norms for firedrake_0_fieldsplit_1_ solve. > 0 KSP Residual norm 0.000000000000e+00 > Residual norms for firedrake_0_fieldsplit_1_ solve. > 0 KSP none resid norm 0.000000000000e+00 true resid norm 0.000000000000e+00 ||r(i)||/||b|| inf > 1 KSP Residual norm 0.000000000000e+00 > 1 KSP none resid norm 0.000000000000e+00 true resid norm 0.000000000000e+00 ||r(i)||/||b|| inf > 1 KSP Residual norm 1.612167203819e-12 > 1 KSP unpreconditioned resid norm 1.612167203819e-12 true resid norm 1.589286585800e-12 ||r(i)||/||b|| 3.438995245681e-15 > 1 SNES Function norm 1.599350481360e-12 > > The true residual is very low. (3.501082228626e-15) > > > MIGUEL ANGEL SALAZAR DE TROYA > Head of Software Engineering > miguel.salazar at corintis.com > Corintis SA > EPFL Innovation Park Building C > 1015 Lausanne > > <2024-08-1609_02_11-re_mailmigrationfromgoogletooffice365-sebastien.gobel at corintis.com-co_4456a1bf-dd3e-46b7-85f9-21f278e66a79.png> > Here at Corintis we care for your privacy. That is why we have taken appropriate measures to ensure that the data you have provided to us is always secure. > From: Barry Smith > > Date: Monday, 16 September 2024 at 19:34 > To: miguel.salazar > > Cc: petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] Residual at the fieldsplit level > > > Try ksp_monitor_true_residual. Let's see if it is an issue of preconditioned vs unpreconditioned residual. > > > > > On Sep 16, 2024, at 1:05?PM, miguel.salazar > wrote: > > Hello, > > I have this simple example on Firedrake to illustrate my point. I am solving for a two-component poisson equation (uncoupled). Only the first component has a non-zero residual. > > ``` > import firedrake as fd > from firedrake import inner, grad, dx, sin, pi > > N = 10 > mesh = fd.UnitSquareMesh(N, N) > V = fd.FunctionSpace(mesh, "CG", 1) > W = V * V > u = fd.Function(W) > v = fd.TestFunction(W) > a = inner(grad(u[0]), grad(v[0])) * dx + inner(grad(u[1]), grad(v[1])) * dx > x = fd.SpatialCoordinate(mesh) > f = fd.Function(V) > f.interpolate(fd.Constant(1e4) * sin(x[0] * pi) * sin(2 * x[1] * pi)) > L = f * v[0] * dx > F = a - L > bcs = [fd.DirichletBC(W.sub(0), fd.Constant(2.0), (1,))] > > > def snes_firedrake_residual(F, u, bcs): > for bcs_ in bcs: > bcs_.apply(u) > residual = fd.assemble(F, bcs=bcs, zero_bc_nodes=True) > with residual.dat.vec_ro as r: > print("Initial residual:", r.norm()) > > > snes_firedrake_residual(F, u, bcs) > > > problem = fd.NonlinearVariationalProblem(F, u, bcs=bcs) > solver_mumps_assembled = { > "ksp_type": "preonly", > "ksp_monitor": None, > "pc_type": "python", > "pc_python_type": "firedrake.AssembledPC", > "assembled_pc_type": "lu", > "assembled_pc_factor_mat_solver_type": "mumps", > "assembled_mat_mumps_icntl_14": 200, > "assembled_mat_mumps_icntl_24": 1, > } > solver_fieldsplit = { > "mat_type": "matfree", > "snes_type": "newtonls", > "ksp_type": "fgmres", > "ksp_rtol": 1e-1, > "ksp_monitor": None, > "pc_type": "fieldsplit", > "pc_fieldsplit_type": "additive", > "fieldsplit_0": solver_mumps_assembled, > "fieldsplit_1": solver_mumps_assembled, > } > > solver = fd.NonlinearVariationalSolver(problem, solver_parameters=solver_fieldsplit) > solver.solve() > ``` > > The PETSc output is as follow > > ``` > Initial residual: 462.13689530272404 > 0 SNES Function norm 4.621368953027e+02 > Residual norms for firedrake_0_ solve. > 0 KSP Residual norm 4.621368953027e+02 > Residual norms for firedrake_0_fieldsplit_0_ solve. > 0 KSP Residual norm 1.000000000000e+00 > 1 KSP Residual norm 3.501082228626e-15 > Residual norms for firedrake_0_fieldsplit_1_ solve. > 0 KSP Residual norm 0.000000000000e+00 > 1 KSP Residual norm 0.000000000000e+00 > 1 KSP Residual norm 1.612167203819e-12 > 1 SNES Function norm 1.599350481360e-12 > ``` > > Using the fieldsplit additive preconditioner, the problem converges in a single KSP iteration, as expected. However, I do not understand why the residual of fieldsplit_0 (1e+0) does not coincide with the outer residual (462.13689530272404). It should be the case given that only fieldsplit_0 has a non-zero residual contribution. The fact that it is just 1 is suspicious. Is there something about how the fieldsplit works that I am missing? > > Thanks, > Miguel > > MIGUEL ANGEL SALAZAR DE TROYA > Head of Software Engineering > miguel.salazar at corintis.com > Corintis SA > EPFL Innovation Park Building C > 1015 Lausanne > > <2024-08-1609_02_11-re_mailmigrationfromgoogletooffice365-sebastien.gobel at corintis.com-co_4456a1bf-dd3e-46b7-85f9-21f278e66a79.png> > Here at Corintis we care for your privacy. That is why we have taken appropriate measures to ensure that the data you have provided to us is always secure. -------------- next part -------------- An HTML attachment was scrubbed... URL: From miguel.salazar at corintis.com Tue Sep 17 09:52:19 2024 From: miguel.salazar at corintis.com (miguel.salazar) Date: Tue, 17 Sep 2024 14:52:19 +0000 Subject: [petsc-users] Residual at the fieldsplit level In-Reply-To: <4131CBF5-CDF8-44E8-A994-0BFE57C1DB40@petsc.dev> References: <0e58dc19-e943-4d42-9a05-4cabbcbc8185.d0d10b7c-e93e-4ca5-8394-d4339a327c55.8303ace2-c862-49ea-97f8-20db7341a75a@emailsignatures365.codetwo.com> <29C71683-CADA-4F66-8A0A-40459076323C@petsc.dev> <4E5EDC3B-0D39-4B7F-AEB7-7854B3BC9A24@petsc.dev> <4131CBF5-CDF8-44E8-A994-0BFE57C1DB40@petsc.dev> Message-ID: That is correct MIGUEL ANGEL SALAZAR DE TROYA Head of Software Engineering miguel.salazar at corintis.com Corintis SA EPFL Innovation Park Building C 1015 Lausanne [cid:2024-08-1609_02_11-re_mailmigrationfromgoogletooffice365-sebastien.gobel at corintis.com-co_4456a1bf-dd3e-46b7-85f9-21f278e66a79.png] Here at Corintis we care for your privacy. That is why we have taken appropriate measures to ensure that the data you have provided to us is always secure. From: Barry Smith Date: Tuesday, 17 September 2024 at 16:49 To: miguel.salazar Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Residual at the fieldsplit level So firedrake_0 is the object prefix of your SNES object? On Sep 17, 2024, at 10:37?AM, miguel.salazar wrote: >In my reading, it says >Residual norms for firedrake_0_ solve. > 0 KSP Residual norm 4.621368953027e+02 > so the residual in the first (0th) split does correspond to the outer-most KSP residual norm (not printed but equal to the SNES at the first iteration of Newton. > It is the residual norm of the first field of the first field that is 1 >Residual norms for firedrake_0_fieldsplit_0_ solve. > 0 KSP Residual norm 1.000000000000e+00 Residual norms for firedrake_0_ solve. 0 KSP Residual norm 4.621368953027e+02 Is the residual of the outer solve (it is being printed). Residual norms for firedrake_0_fieldsplit_0_ solve. 0 KSP Residual norm 1.000000000000e+00 Is the residual of the first field. I am not sure what you are referring to with ?the residual norm of the first field of the first field? Thanks, Miguel MIGUEL ANGEL SALAZAR DE TROYA Head of Software Engineering miguel.salazar at corintis.com Corintis SA EPFL Innovation Park Building C 1015 Lausanne <2024-08-1609_02_11-re_mailmigrationfromgoogletooffice365-sebastien.gobel at corintis.com-co_4456a1bf-dd3e-46b7-85f9-21f278e66a79.png> Here at Corintis we care for your privacy. That is why we have taken appropriate measures to ensure that the data you have provided to us is always secure. From: Barry Smith > Date: Tuesday, 17 September 2024 at 16:24 To: miguel.salazar > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Residual at the fieldsplit level Initial residual: 462.13689530272404 0 SNES Function norm 4.621368953027e+02 Residual norms for firedrake_0_ solve. 0 KSP Residual norm 4.621368953027e+02 Residual norms for firedrake_0_fieldsplit_0_ solve. 0 KSP Residual norm 1.000000000000e+00 1 KSP Residual norm 3.501082228626e-15 Residual norms for firedrake_0_fieldsplit_1_ solve. 0 KSP Residual norm 0.000000000000e+00 1 KSP Residual norm 0.000000000000e+00 1 KSP Residual norm 1.612167203819e-12 1 SNES Function norm 1.599350481360e-12 ``` Using the fieldsplit additive preconditioner, the problem converges in a single KSP iteration, as expected. However, I do not understand why the residual of fieldsplit_0 (1e+0) does not coincide with the outer residual (462.13689530272404). It should be the case given that only fieldsplit_0 has a non-zero residual contribution. The fact that it is just 1 is suspicious. Is there something about how the fieldsplit works that I am missing? In my reading, it says Residual norms for firedrake_0_ solve. 0 KSP Residual norm 4.621368953027e+02 so the residual in the first (0th) split does correspond to the outer-most KSP residual norm (not printed but equal to the SNES at the first iteration of Newton. It is the residual norm of the first field of the first field that is 1 Residual norms for firedrake_0_fieldsplit_0_ solve. 0 KSP Residual norm 1.000000000000e+00 Or am I missing something Barry Initial residual: 462.13689530272404 0 SNES Function norm 4.621368953027e+02 Residual norms for firedrake_0_ solve. 0 KSP Residual norm 4.621368953027e+02 Residual norms for firedrake_0_ solve. 0 KSP unpreconditioned resid norm 4.621368953027e+02 true resid norm 4.621368953027e+02 ||r(i)||/||b|| 1.000000000000e+00 Residual norms for firedrake_0_fieldsplit_0_ solve. 0 KSP Residual norm 1.000000000000e+00 Residual norms for firedrake_0_fieldsplit_0_ solve. 0 KSP none resid norm 1.000000000000e+00 true resid norm 3.501082228626e-15 ||r(i)||/||b|| 3.501082228626e-15 1 KSP Residual norm 3.501082228626e-15 1 KSP none resid norm 3.501082228626e-15 true resid norm 3.501082228626e-15 ||r(i)||/||b|| 3.501082228626e-15 Residual norms for firedrake_0_fieldsplit_1_ solve. 0 KSP Residual norm 0.000000000000e+00 Residual norms for firedrake_0_fieldsplit_1_ solve. 0 KSP none resid norm 0.000000000000e+00 true resid norm 0.000000000000e+00 ||r(i)||/||b|| inf 1 KSP Residual norm 0.000000000000e+00 1 KSP none resid norm 0.000000000000e+00 true resid norm 0.000000000000e+00 ||r(i)||/||b|| inf 1 KSP Residual norm 1.612167203819e-12 1 KSP unpreconditioned resid norm 1.612167203819e-12 true resid norm 1.589286585800e-12 ||r(i)||/||b|| 3.438995245681e-15 1 SNES Function norm 1.599350481360e-12 On Sep 17, 2024, at 1:04?AM, miguel.salazar > wrote: These are the new options (with ksp_monitor_true_residual) solver_mumps_assembled = { "ksp_type": "preonly", "ksp_monitor": None, "ksp_monitor_true_residual": None, "pc_type": "python", "pc_python_type": "firedrake.AssembledPC", "assembled_pc_type": "lu", "assembled_pc_factor_mat_solver_type": "mumps", "assembled_mat_mumps_icntl_14": 200, "assembled_mat_mumps_icntl_24": 1, } solver_fieldsplit = { "mat_type": "matfree", "snes_monitor": None, "snes_type": "newtonls", "ksp_type": "fgmres", "ksp_monitor_true_residual": None, "ksp_rtol": 1e-1, "ksp_monitor": None, "pc_type": "fieldsplit", "pc_fieldsplit_type": "additive", "fieldsplit_0": solver_mumps_assembled, "fieldsplit_1": solver_mumps_assembled, } And this is the output Initial residual: 462.13689530272404 0 SNES Function norm 4.621368953027e+02 Residual norms for firedrake_0_ solve. 0 KSP Residual norm 4.621368953027e+02 Residual norms for firedrake_0_ solve. 0 KSP unpreconditioned resid norm 4.621368953027e+02 true resid norm 4.621368953027e+02 ||r(i)||/||b|| 1.000000000000e+00 Residual norms for firedrake_0_fieldsplit_0_ solve. 0 KSP Residual norm 1.000000000000e+00 Residual norms for firedrake_0_fieldsplit_0_ solve. 0 KSP none resid norm 1.000000000000e+00 true resid norm 3.501082228626e-15 ||r(i)||/||b|| 3.501082228626e-15 1 KSP Residual norm 3.501082228626e-15 1 KSP none resid norm 3.501082228626e-15 true resid norm 3.501082228626e-15 ||r(i)||/||b|| 3.501082228626e-15 Residual norms for firedrake_0_fieldsplit_1_ solve. 0 KSP Residual norm 0.000000000000e+00 Residual norms for firedrake_0_fieldsplit_1_ solve. 0 KSP none resid norm 0.000000000000e+00 true resid norm 0.000000000000e+00 ||r(i)||/||b|| inf 1 KSP Residual norm 0.000000000000e+00 1 KSP none resid norm 0.000000000000e+00 true resid norm 0.000000000000e+00 ||r(i)||/||b|| inf 1 KSP Residual norm 1.612167203819e-12 1 KSP unpreconditioned resid norm 1.612167203819e-12 true resid norm 1.589286585800e-12 ||r(i)||/||b|| 3.438995245681e-15 1 SNES Function norm 1.599350481360e-12 The true residual is very low. (3.501082228626e-15) MIGUEL ANGEL SALAZAR DE TROYA Head of Software Engineering miguel.salazar at corintis.com Corintis SA EPFL Innovation Park Building C 1015 Lausanne <2024-08-1609_02_11-re_mailmigrationfromgoogletooffice365-sebastien.gobel at corintis.com-co_4456a1bf-dd3e-46b7-85f9-21f278e66a79.png> Here at Corintis we care for your privacy. That is why we have taken appropriate measures to ensure that the data you have provided to us is always secure. From: Barry Smith > Date: Monday, 16 September 2024 at 19:34 To: miguel.salazar > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Residual at the fieldsplit level Try ksp_monitor_true_residual. Let's see if it is an issue of preconditioned vs unpreconditioned residual. On Sep 16, 2024, at 1:05?PM, miguel.salazar > wrote: Hello, I have this simple example on Firedrake to illustrate my point. I am solving for a two-component poisson equation (uncoupled). Only the first component has a non-zero residual. ``` import firedrake as fd from firedrake import inner, grad, dx, sin, pi N = 10 mesh = fd.UnitSquareMesh(N, N) V = fd.FunctionSpace(mesh, "CG", 1) W = V * V u = fd.Function(W) v = fd.TestFunction(W) a = inner(grad(u[0]), grad(v[0])) * dx + inner(grad(u[1]), grad(v[1])) * dx x = fd.SpatialCoordinate(mesh) f = fd.Function(V) f.interpolate(fd.Constant(1e4) * sin(x[0] * pi) * sin(2 * x[1] * pi)) L = f * v[0] * dx F = a - L bcs = [fd.DirichletBC(W.sub(0), fd.Constant(2.0), (1,))] def snes_firedrake_residual(F, u, bcs): for bcs_ in bcs: bcs_.apply(u) residual = fd.assemble(F, bcs=bcs, zero_bc_nodes=True) with residual.dat.vec_ro as r: print("Initial residual:", r.norm()) snes_firedrake_residual(F, u, bcs) problem = fd.NonlinearVariationalProblem(F, u, bcs=bcs) solver_mumps_assembled = { "ksp_type": "preonly", "ksp_monitor": None, "pc_type": "python", "pc_python_type": "firedrake.AssembledPC", "assembled_pc_type": "lu", "assembled_pc_factor_mat_solver_type": "mumps", "assembled_mat_mumps_icntl_14": 200, "assembled_mat_mumps_icntl_24": 1, } solver_fieldsplit = { "mat_type": "matfree", "snes_type": "newtonls", "ksp_type": "fgmres", "ksp_rtol": 1e-1, "ksp_monitor": None, "pc_type": "fieldsplit", "pc_fieldsplit_type": "additive", "fieldsplit_0": solver_mumps_assembled, "fieldsplit_1": solver_mumps_assembled, } solver = fd.NonlinearVariationalSolver(problem, solver_parameters=solver_fieldsplit) solver.solve() ``` The PETSc output is as follow ``` Initial residual: 462.13689530272404 0 SNES Function norm 4.621368953027e+02 Residual norms for firedrake_0_ solve. 0 KSP Residual norm 4.621368953027e+02 Residual norms for firedrake_0_fieldsplit_0_ solve. 0 KSP Residual norm 1.000000000000e+00 1 KSP Residual norm 3.501082228626e-15 Residual norms for firedrake_0_fieldsplit_1_ solve. 0 KSP Residual norm 0.000000000000e+00 1 KSP Residual norm 0.000000000000e+00 1 KSP Residual norm 1.612167203819e-12 1 SNES Function norm 1.599350481360e-12 ``` Using the fieldsplit additive preconditioner, the problem converges in a single KSP iteration, as expected. However, I do not understand why the residual of fieldsplit_0 (1e+0) does not coincide with the outer residual (462.13689530272404). It should be the case given that only fieldsplit_0 has a non-zero residual contribution. The fact that it is just 1 is suspicious. Is there something about how the fieldsplit works that I am missing? Thanks, Miguel MIGUEL ANGEL SALAZAR DE TROYA Head of Software Engineering miguel.salazar at corintis.com Corintis SA EPFL Innovation Park Building C 1015 Lausanne <2024-08-1609_02_11-re_mailmigrationfromgoogletooffice365-sebastien.gobel at corintis.com-co_4456a1bf-dd3e-46b7-85f9-21f278e66a79.png> Here at Corintis we care for your privacy. That is why we have taken appropriate measures to ensure that the data you have provided to us is always secure. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 2024-08-1609_02_11-re_mailmigrationfromgoogletooffice365-sebastien.gobel at corintis.com-co_4456a1bf-dd3e-46b7-85f9-21f278e66a79.png Type: image/png Size: 38219 bytes Desc: 2024-08-1609_02_11-re_mailmigrationfromgoogletooffice365-sebastien.gobel at corintis.com-co_4456a1bf-dd3e-46b7-85f9-21f278e66a79.png URL: From bsmith at petsc.dev Tue Sep 17 09:57:54 2024 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 17 Sep 2024 10:57:54 -0400 Subject: [petsc-users] Residual at the fieldsplit level In-Reply-To: References: <0e58dc19-e943-4d42-9a05-4cabbcbc8185.d0d10b7c-e93e-4ca5-8394-d4339a327c55.8303ace2-c862-49ea-97f8-20db7341a75a@emailsignatures365.codetwo.com> <29C71683-CADA-4F66-8A0A-40459076323C@petsc.dev> <4E5EDC3B-0D39-4B7F-AEB7-7854B3BC9A24@petsc.dev> <4131CBF5-CDF8-44E8-A994-0BFE57C1DB40@petsc.dev> Message-ID: <5E923642-A231-42A8-8958-470352157CDD@petsc.dev> Ok. Please ignore my last message about fieldsplit_0_fieldsplit_0 I misunderstood > On Sep 17, 2024, at 10:52?AM, miguel.salazar wrote: > > That is correct > > > > > MIGUEL ANGEL SALAZAR DE TROYA > Head of Software Engineering > miguel.salazar at corintis.com > Corintis SA > EPFL Innovation Park Building C > 1015 Lausanne > <2024-08-1609_02_11-re_mailmigrationfromgoogletooffice365-sebastien.gobel at corintis.com-co_4456a1bf-dd3e-46b7-85f9-21f278e66a79.png> > Here at Corintis we care for your privacy. That is why we have taken appropriate measures to ensure that the data you have provided to us is always secure. > From: Barry Smith > > Date: Tuesday, 17 September 2024 at 16:49 > To: miguel.salazar > > Cc: petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] Residual at the fieldsplit level > > > So firedrake_0 is the object prefix of your SNES object? > > > > On Sep 17, 2024, at 10:37?AM, miguel.salazar > wrote: > > >In my reading, it says > >Residual norms for firedrake_0_ solve. > > 0 KSP Residual norm 4.621368953027e+02 > > > so the residual in the first (0th) split does correspond to the outer-most KSP residual norm (not printed but equal to the SNES at the first iteration of Newton. > > > It is the residual norm of the first field of the first field that is 1 > > >Residual norms for firedrake_0_fieldsplit_0_ solve. > > 0 KSP Residual norm 1.000000000000e+00 > > Residual norms for firedrake_0_ solve. > 0 KSP Residual norm 4.621368953027e+02 > > Is the residual of the outer solve (it is being printed). > > Residual norms for firedrake_0_fieldsplit_0_ solve. > 0 KSP Residual norm 1.000000000000e+00 > > Is the residual of the first field. I am not sure what you are referring to with ?the residual norm of the first field of the first field? > > Thanks, > Miguel > > > > > MIGUEL ANGEL SALAZAR DE TROYA > Head of Software Engineering > miguel.salazar at corintis.com > Corintis SA > EPFL Innovation Park Building C > 1015 Lausanne > > <2024-08-1609_02_11-re_mailmigrationfromgoogletooffice365-sebastien.gobel at corintis.com-co_4456a1bf-dd3e-46b7-85f9-21f278e66a79.png> > Here at Corintis we care for your privacy. That is why we have taken appropriate measures to ensure that the data you have provided to us is always secure. > From: Barry Smith > > Date: Tuesday, 17 September 2024 at 16:24 > To: miguel.salazar > > Cc: petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] Residual at the fieldsplit level > > > Initial residual: 462.13689530272404 > 0 SNES Function norm 4.621368953027e+02 > Residual norms for firedrake_0_ solve. > 0 KSP Residual norm 4.621368953027e+02 > Residual norms for firedrake_0_fieldsplit_0_ solve. > 0 KSP Residual norm 1.000000000000e+00 > 1 KSP Residual norm 3.501082228626e-15 > Residual norms for firedrake_0_fieldsplit_1_ solve. > 0 KSP Residual norm 0.000000000000e+00 > 1 KSP Residual norm 0.000000000000e+00 > 1 KSP Residual norm 1.612167203819e-12 > 1 SNES Function norm 1.599350481360e-12 > ``` > > Using the fieldsplit additive preconditioner, the problem converges in a single KSP iteration, as expected. However, I do not understand why the residual of fieldsplit_0 (1e+0) does not coincide with the outer residual (462.13689530272404). It should be the case given that only fieldsplit_0 has a non-zero residual contribution. The fact that it is just 1 is suspicious. Is there something about how the fieldsplit works that I am missing? > > In my reading, it says > Residual norms for firedrake_0_ solve. > 0 KSP Residual norm 4.621368953027e+02 > > so the residual in the first (0th) split does correspond to the outer-most KSP residual norm (not printed but equal to the SNES at the first iteration of Newton. > > It is the residual norm of the first field of the first field that is 1 > > Residual norms for firedrake_0_fieldsplit_0_ solve. > 0 KSP Residual norm 1.000000000000e+00 > > Or am I missing something > > Barry > > > > > > > > Initial residual: 462.13689530272404 > 0 SNES Function norm 4.621368953027e+02 > Residual norms for firedrake_0_ solve. > 0 KSP Residual norm 4.621368953027e+02 > Residual norms for firedrake_0_ solve. > 0 KSP unpreconditioned resid norm 4.621368953027e+02 true resid norm 4.621368953027e+02 ||r(i)||/||b|| 1.000000000000e+00 > Residual norms for firedrake_0_fieldsplit_0_ solve. > 0 KSP Residual norm 1.000000000000e+00 > Residual norms for firedrake_0_fieldsplit_0_ solve. > 0 KSP none resid norm 1.000000000000e+00 true resid norm 3.501082228626e-15 ||r(i)||/||b|| 3.501082228626e-15 > 1 KSP Residual norm 3.501082228626e-15 > 1 KSP none resid norm 3.501082228626e-15 true resid norm 3.501082228626e-15 ||r(i)||/||b|| 3.501082228626e-15 > Residual norms for firedrake_0_fieldsplit_1_ solve. > 0 KSP Residual norm 0.000000000000e+00 > Residual norms for firedrake_0_fieldsplit_1_ solve. > 0 KSP none resid norm 0.000000000000e+00 true resid norm 0.000000000000e+00 ||r(i)||/||b|| inf > 1 KSP Residual norm 0.000000000000e+00 > 1 KSP none resid norm 0.000000000000e+00 true resid norm 0.000000000000e+00 ||r(i)||/||b|| inf > 1 KSP Residual norm 1.612167203819e-12 > 1 KSP unpreconditioned resid norm 1.612167203819e-12 true resid norm 1.589286585800e-12 ||r(i)||/||b|| 3.438995245681e-15 > 1 SNES Function norm 1.599350481360e-12 > > > > On Sep 17, 2024, at 1:04?AM, miguel.salazar > wrote: > > These are the new options (with ksp_monitor_true_residual) > > solver_mumps_assembled = { > "ksp_type": "preonly", > "ksp_monitor": None, > "ksp_monitor_true_residual": None, > "pc_type": "python", > "pc_python_type": "firedrake.AssembledPC", > "assembled_pc_type": "lu", > "assembled_pc_factor_mat_solver_type": "mumps", > "assembled_mat_mumps_icntl_14": 200, > "assembled_mat_mumps_icntl_24": 1, > } > solver_fieldsplit = { > "mat_type": "matfree", > "snes_monitor": None, > "snes_type": "newtonls", > "ksp_type": "fgmres", > "ksp_monitor_true_residual": None, > "ksp_rtol": 1e-1, > "ksp_monitor": None, > "pc_type": "fieldsplit", > "pc_fieldsplit_type": "additive", > "fieldsplit_0": solver_mumps_assembled, > "fieldsplit_1": solver_mumps_assembled, > } > > And this is the output > > Initial residual: 462.13689530272404 > 0 SNES Function norm 4.621368953027e+02 > Residual norms for firedrake_0_ solve. > 0 KSP Residual norm 4.621368953027e+02 > Residual norms for firedrake_0_ solve. > 0 KSP unpreconditioned resid norm 4.621368953027e+02 true resid norm 4.621368953027e+02 ||r(i)||/||b|| 1.000000000000e+00 > Residual norms for firedrake_0_fieldsplit_0_ solve. > 0 KSP Residual norm 1.000000000000e+00 > Residual norms for firedrake_0_fieldsplit_0_ solve. > 0 KSP none resid norm 1.000000000000e+00 true resid norm 3.501082228626e-15 ||r(i)||/||b|| 3.501082228626e-15 > 1 KSP Residual norm 3.501082228626e-15 > 1 KSP none resid norm 3.501082228626e-15 true resid norm 3.501082228626e-15 ||r(i)||/||b|| 3.501082228626e-15 > Residual norms for firedrake_0_fieldsplit_1_ solve. > 0 KSP Residual norm 0.000000000000e+00 > Residual norms for firedrake_0_fieldsplit_1_ solve. > 0 KSP none resid norm 0.000000000000e+00 true resid norm 0.000000000000e+00 ||r(i)||/||b|| inf > 1 KSP Residual norm 0.000000000000e+00 > 1 KSP none resid norm 0.000000000000e+00 true resid norm 0.000000000000e+00 ||r(i)||/||b|| inf > 1 KSP Residual norm 1.612167203819e-12 > 1 KSP unpreconditioned resid norm 1.612167203819e-12 true resid norm 1.589286585800e-12 ||r(i)||/||b|| 3.438995245681e-15 > 1 SNES Function norm 1.599350481360e-12 > > The true residual is very low. (3.501082228626e-15) > > > MIGUEL ANGEL SALAZAR DE TROYA > Head of Software Engineering > miguel.salazar at corintis.com > Corintis SA > EPFL Innovation Park Building C > 1015 Lausanne > > <2024-08-1609_02_11-re_mailmigrationfromgoogletooffice365-sebastien.gobel at corintis.com-co_4456a1bf-dd3e-46b7-85f9-21f278e66a79.png> > Here at Corintis we care for your privacy. That is why we have taken appropriate measures to ensure that the data you have provided to us is always secure. > From: Barry Smith > > Date: Monday, 16 September 2024 at 19:34 > To: miguel.salazar > > Cc: petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] Residual at the fieldsplit level > > > Try ksp_monitor_true_residual. Let's see if it is an issue of preconditioned vs unpreconditioned residual. > > > > > On Sep 16, 2024, at 1:05?PM, miguel.salazar > wrote: > > Hello, > > I have this simple example on Firedrake to illustrate my point. I am solving for a two-component poisson equation (uncoupled). Only the first component has a non-zero residual. > > ``` > import firedrake as fd > from firedrake import inner, grad, dx, sin, pi > > N = 10 > mesh = fd.UnitSquareMesh(N, N) > V = fd.FunctionSpace(mesh, "CG", 1) > W = V * V > u = fd.Function(W) > v = fd.TestFunction(W) > a = inner(grad(u[0]), grad(v[0])) * dx + inner(grad(u[1]), grad(v[1])) * dx > x = fd.SpatialCoordinate(mesh) > f = fd.Function(V) > f.interpolate(fd.Constant(1e4) * sin(x[0] * pi) * sin(2 * x[1] * pi)) > L = f * v[0] * dx > F = a - L > bcs = [fd.DirichletBC(W.sub(0), fd.Constant(2.0), (1,))] > > > def snes_firedrake_residual(F, u, bcs): > for bcs_ in bcs: > bcs_.apply(u) > residual = fd.assemble(F, bcs=bcs, zero_bc_nodes=True) > with residual.dat.vec_ro as r: > print("Initial residual:", r.norm()) > > > snes_firedrake_residual(F, u, bcs) > > > problem = fd.NonlinearVariationalProblem(F, u, bcs=bcs) > solver_mumps_assembled = { > "ksp_type": "preonly", > "ksp_monitor": None, > "pc_type": "python", > "pc_python_type": "firedrake.AssembledPC", > "assembled_pc_type": "lu", > "assembled_pc_factor_mat_solver_type": "mumps", > "assembled_mat_mumps_icntl_14": 200, > "assembled_mat_mumps_icntl_24": 1, > } > solver_fieldsplit = { > "mat_type": "matfree", > "snes_type": "newtonls", > "ksp_type": "fgmres", > "ksp_rtol": 1e-1, > "ksp_monitor": None, > "pc_type": "fieldsplit", > "pc_fieldsplit_type": "additive", > "fieldsplit_0": solver_mumps_assembled, > "fieldsplit_1": solver_mumps_assembled, > } > > solver = fd.NonlinearVariationalSolver(problem, solver_parameters=solver_fieldsplit) > solver.solve() > ``` > > The PETSc output is as follow > > ``` > Initial residual: 462.13689530272404 > 0 SNES Function norm 4.621368953027e+02 > Residual norms for firedrake_0_ solve. > 0 KSP Residual norm 4.621368953027e+02 > Residual norms for firedrake_0_fieldsplit_0_ solve. > 0 KSP Residual norm 1.000000000000e+00 > 1 KSP Residual norm 3.501082228626e-15 > Residual norms for firedrake_0_fieldsplit_1_ solve. > 0 KSP Residual norm 0.000000000000e+00 > 1 KSP Residual norm 0.000000000000e+00 > 1 KSP Residual norm 1.612167203819e-12 > 1 SNES Function norm 1.599350481360e-12 > ``` > > Using the fieldsplit additive preconditioner, the problem converges in a single KSP iteration, as expected. However, I do not understand why the residual of fieldsplit_0 (1e+0) does not coincide with the outer residual (462.13689530272404). It should be the case given that only fieldsplit_0 has a non-zero residual contribution. The fact that it is just 1 is suspicious. Is there something about how the fieldsplit works that I am missing? > > Thanks, > Miguel > > MIGUEL ANGEL SALAZAR DE TROYA > Head of Software Engineering > miguel.salazar at corintis.com > Corintis SA > EPFL Innovation Park Building C > 1015 Lausanne > > <2024-08-1609_02_11-re_mailmigrationfromgoogletooffice365-sebastien.gobel at corintis.com-co_4456a1bf-dd3e-46b7-85f9-21f278e66a79.png> > Here at Corintis we care for your privacy. That is why we have taken appropriate measures to ensure that the data you have provided to us is always secure. -------------- next part -------------- An HTML attachment was scrubbed... URL: From miguel.salazar at corintis.com Tue Sep 17 10:48:39 2024 From: miguel.salazar at corintis.com (miguel.salazar) Date: Tue, 17 Sep 2024 15:48:39 +0000 Subject: [petsc-users] Residual at the fieldsplit level In-Reply-To: <5E923642-A231-42A8-8958-470352157CDD@petsc.dev> References: <0e58dc19-e943-4d42-9a05-4cabbcbc8185.d0d10b7c-e93e-4ca5-8394-d4339a327c55.8303ace2-c862-49ea-97f8-20db7341a75a@emailsignatures365.codetwo.com> <29C71683-CADA-4F66-8A0A-40459076323C@petsc.dev> <4E5EDC3B-0D39-4B7F-AEB7-7854B3BC9A24@petsc.dev> <4131CBF5-CDF8-44E8-A994-0BFE57C1DB40@petsc.dev> <5E923642-A231-42A8-8958-470352157CDD@petsc.dev> Message-ID: Is this the expected behavior though? I am not sure if it might be an issue with the Firedrake implementation. Getting a minimum example on PETSc like this one is not trivial Miguel MIGUEL ANGEL SALAZAR DE TROYA Head of Software Engineering miguel.salazar at corintis.com Corintis SA EPFL Innovation Park Building C 1015 Lausanne [cid:2024-08-1609_02_11-re_mailmigrationfromgoogletooffice365-sebastien.gobel at corintis.com-co_4456a1bf-dd3e-46b7-85f9-21f278e66a79.png] Here at Corintis we care for your privacy. That is why we have taken appropriate measures to ensure that the data you have provided to us is always secure. From: Barry Smith Date: Tuesday, 17 September 2024 at 16:58 To: miguel.salazar Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Residual at the fieldsplit level Ok. Please ignore my last message about fieldsplit_0_fieldsplit_0 I misunderstood On Sep 17, 2024, at 10:52?AM, miguel.salazar wrote: That is correct MIGUEL ANGEL SALAZAR DE TROYA Head of Software Engineering miguel.salazar at corintis.com Corintis SA EPFL Innovation Park Building C 1015 Lausanne <2024-08-1609_02_11-re_mailmigrationfromgoogletooffice365-sebastien.gobel at corintis.com-co_4456a1bf-dd3e-46b7-85f9-21f278e66a79.png> Here at Corintis we care for your privacy. That is why we have taken appropriate measures to ensure that the data you have provided to us is always secure. From: Barry Smith > Date: Tuesday, 17 September 2024 at 16:49 To: miguel.salazar > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Residual at the fieldsplit level So firedrake_0 is the object prefix of your SNES object? On Sep 17, 2024, at 10:37?AM, miguel.salazar > wrote: >In my reading, it says >Residual norms for firedrake_0_ solve. > 0 KSP Residual norm 4.621368953027e+02 > so the residual in the first (0th) split does correspond to the outer-most KSP residual norm (not printed but equal to the SNES at the first iteration of Newton. > It is the residual norm of the first field of the first field that is 1 >Residual norms for firedrake_0_fieldsplit_0_ solve. > 0 KSP Residual norm 1.000000000000e+00 Residual norms for firedrake_0_ solve. 0 KSP Residual norm 4.621368953027e+02 Is the residual of the outer solve (it is being printed). Residual norms for firedrake_0_fieldsplit_0_ solve. 0 KSP Residual norm 1.000000000000e+00 Is the residual of the first field. I am not sure what you are referring to with ?the residual norm of the first field of the first field? Thanks, Miguel MIGUEL ANGEL SALAZAR DE TROYA Head of Software Engineering miguel.salazar at corintis.com Corintis SA EPFL Innovation Park Building C 1015 Lausanne <2024-08-1609_02_11-re_mailmigrationfromgoogletooffice365-sebastien.gobel at corintis.com-co_4456a1bf-dd3e-46b7-85f9-21f278e66a79.png> Here at Corintis we care for your privacy. That is why we have taken appropriate measures to ensure that the data you have provided to us is always secure. From: Barry Smith > Date: Tuesday, 17 September 2024 at 16:24 To: miguel.salazar > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Residual at the fieldsplit level Initial residual: 462.13689530272404 0 SNES Function norm 4.621368953027e+02 Residual norms for firedrake_0_ solve. 0 KSP Residual norm 4.621368953027e+02 Residual norms for firedrake_0_fieldsplit_0_ solve. 0 KSP Residual norm 1.000000000000e+00 1 KSP Residual norm 3.501082228626e-15 Residual norms for firedrake_0_fieldsplit_1_ solve. 0 KSP Residual norm 0.000000000000e+00 1 KSP Residual norm 0.000000000000e+00 1 KSP Residual norm 1.612167203819e-12 1 SNES Function norm 1.599350481360e-12 ``` Using the fieldsplit additive preconditioner, the problem converges in a single KSP iteration, as expected. However, I do not understand why the residual of fieldsplit_0 (1e+0) does not coincide with the outer residual (462.13689530272404). It should be the case given that only fieldsplit_0 has a non-zero residual contribution. The fact that it is just 1 is suspicious. Is there something about how the fieldsplit works that I am missing? In my reading, it says Residual norms for firedrake_0_ solve. 0 KSP Residual norm 4.621368953027e+02 so the residual in the first (0th) split does correspond to the outer-most KSP residual norm (not printed but equal to the SNES at the first iteration of Newton. It is the residual norm of the first field of the first field that is 1 Residual norms for firedrake_0_fieldsplit_0_ solve. 0 KSP Residual norm 1.000000000000e+00 Or am I missing something Barry Initial residual: 462.13689530272404 0 SNES Function norm 4.621368953027e+02 Residual norms for firedrake_0_ solve. 0 KSP Residual norm 4.621368953027e+02 Residual norms for firedrake_0_ solve. 0 KSP unpreconditioned resid norm 4.621368953027e+02 true resid norm 4.621368953027e+02 ||r(i)||/||b|| 1.000000000000e+00 Residual norms for firedrake_0_fieldsplit_0_ solve. 0 KSP Residual norm 1.000000000000e+00 Residual norms for firedrake_0_fieldsplit_0_ solve. 0 KSP none resid norm 1.000000000000e+00 true resid norm 3.501082228626e-15 ||r(i)||/||b|| 3.501082228626e-15 1 KSP Residual norm 3.501082228626e-15 1 KSP none resid norm 3.501082228626e-15 true resid norm 3.501082228626e-15 ||r(i)||/||b|| 3.501082228626e-15 Residual norms for firedrake_0_fieldsplit_1_ solve. 0 KSP Residual norm 0.000000000000e+00 Residual norms for firedrake_0_fieldsplit_1_ solve. 0 KSP none resid norm 0.000000000000e+00 true resid norm 0.000000000000e+00 ||r(i)||/||b|| inf 1 KSP Residual norm 0.000000000000e+00 1 KSP none resid norm 0.000000000000e+00 true resid norm 0.000000000000e+00 ||r(i)||/||b|| inf 1 KSP Residual norm 1.612167203819e-12 1 KSP unpreconditioned resid norm 1.612167203819e-12 true resid norm 1.589286585800e-12 ||r(i)||/||b|| 3.438995245681e-15 1 SNES Function norm 1.599350481360e-12 On Sep 17, 2024, at 1:04?AM, miguel.salazar > wrote: These are the new options (with ksp_monitor_true_residual) solver_mumps_assembled = { "ksp_type": "preonly", "ksp_monitor": None, "ksp_monitor_true_residual": None, "pc_type": "python", "pc_python_type": "firedrake.AssembledPC", "assembled_pc_type": "lu", "assembled_pc_factor_mat_solver_type": "mumps", "assembled_mat_mumps_icntl_14": 200, "assembled_mat_mumps_icntl_24": 1, } solver_fieldsplit = { "mat_type": "matfree", "snes_monitor": None, "snes_type": "newtonls", "ksp_type": "fgmres", "ksp_monitor_true_residual": None, "ksp_rtol": 1e-1, "ksp_monitor": None, "pc_type": "fieldsplit", "pc_fieldsplit_type": "additive", "fieldsplit_0": solver_mumps_assembled, "fieldsplit_1": solver_mumps_assembled, } And this is the output Initial residual: 462.13689530272404 0 SNES Function norm 4.621368953027e+02 Residual norms for firedrake_0_ solve. 0 KSP Residual norm 4.621368953027e+02 Residual norms for firedrake_0_ solve. 0 KSP unpreconditioned resid norm 4.621368953027e+02 true resid norm 4.621368953027e+02 ||r(i)||/||b|| 1.000000000000e+00 Residual norms for firedrake_0_fieldsplit_0_ solve. 0 KSP Residual norm 1.000000000000e+00 Residual norms for firedrake_0_fieldsplit_0_ solve. 0 KSP none resid norm 1.000000000000e+00 true resid norm 3.501082228626e-15 ||r(i)||/||b|| 3.501082228626e-15 1 KSP Residual norm 3.501082228626e-15 1 KSP none resid norm 3.501082228626e-15 true resid norm 3.501082228626e-15 ||r(i)||/||b|| 3.501082228626e-15 Residual norms for firedrake_0_fieldsplit_1_ solve. 0 KSP Residual norm 0.000000000000e+00 Residual norms for firedrake_0_fieldsplit_1_ solve. 0 KSP none resid norm 0.000000000000e+00 true resid norm 0.000000000000e+00 ||r(i)||/||b|| inf 1 KSP Residual norm 0.000000000000e+00 1 KSP none resid norm 0.000000000000e+00 true resid norm 0.000000000000e+00 ||r(i)||/||b|| inf 1 KSP Residual norm 1.612167203819e-12 1 KSP unpreconditioned resid norm 1.612167203819e-12 true resid norm 1.589286585800e-12 ||r(i)||/||b|| 3.438995245681e-15 1 SNES Function norm 1.599350481360e-12 The true residual is very low. (3.501082228626e-15) MIGUEL ANGEL SALAZAR DE TROYA Head of Software Engineering miguel.salazar at corintis.com Corintis SA EPFL Innovation Park Building C 1015 Lausanne <2024-08-1609_02_11-re_mailmigrationfromgoogletooffice365-sebastien.gobel at corintis.com-co_4456a1bf-dd3e-46b7-85f9-21f278e66a79.png> Here at Corintis we care for your privacy. That is why we have taken appropriate measures to ensure that the data you have provided to us is always secure. From: Barry Smith > Date: Monday, 16 September 2024 at 19:34 To: miguel.salazar > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Residual at the fieldsplit level Try ksp_monitor_true_residual. Let's see if it is an issue of preconditioned vs unpreconditioned residual. On Sep 16, 2024, at 1:05?PM, miguel.salazar > wrote: Hello, I have this simple example on Firedrake to illustrate my point. I am solving for a two-component poisson equation (uncoupled). Only the first component has a non-zero residual. ``` import firedrake as fd from firedrake import inner, grad, dx, sin, pi N = 10 mesh = fd.UnitSquareMesh(N, N) V = fd.FunctionSpace(mesh, "CG", 1) W = V * V u = fd.Function(W) v = fd.TestFunction(W) a = inner(grad(u[0]), grad(v[0])) * dx + inner(grad(u[1]), grad(v[1])) * dx x = fd.SpatialCoordinate(mesh) f = fd.Function(V) f.interpolate(fd.Constant(1e4) * sin(x[0] * pi) * sin(2 * x[1] * pi)) L = f * v[0] * dx F = a - L bcs = [fd.DirichletBC(W.sub(0), fd.Constant(2.0), (1,))] def snes_firedrake_residual(F, u, bcs): for bcs_ in bcs: bcs_.apply(u) residual = fd.assemble(F, bcs=bcs, zero_bc_nodes=True) with residual.dat.vec_ro as r: print("Initial residual:", r.norm()) snes_firedrake_residual(F, u, bcs) problem = fd.NonlinearVariationalProblem(F, u, bcs=bcs) solver_mumps_assembled = { "ksp_type": "preonly", "ksp_monitor": None, "pc_type": "python", "pc_python_type": "firedrake.AssembledPC", "assembled_pc_type": "lu", "assembled_pc_factor_mat_solver_type": "mumps", "assembled_mat_mumps_icntl_14": 200, "assembled_mat_mumps_icntl_24": 1, } solver_fieldsplit = { "mat_type": "matfree", "snes_type": "newtonls", "ksp_type": "fgmres", "ksp_rtol": 1e-1, "ksp_monitor": None, "pc_type": "fieldsplit", "pc_fieldsplit_type": "additive", "fieldsplit_0": solver_mumps_assembled, "fieldsplit_1": solver_mumps_assembled, } solver = fd.NonlinearVariationalSolver(problem, solver_parameters=solver_fieldsplit) solver.solve() ``` The PETSc output is as follow ``` Initial residual: 462.13689530272404 0 SNES Function norm 4.621368953027e+02 Residual norms for firedrake_0_ solve. 0 KSP Residual norm 4.621368953027e+02 Residual norms for firedrake_0_fieldsplit_0_ solve. 0 KSP Residual norm 1.000000000000e+00 1 KSP Residual norm 3.501082228626e-15 Residual norms for firedrake_0_fieldsplit_1_ solve. 0 KSP Residual norm 0.000000000000e+00 1 KSP Residual norm 0.000000000000e+00 1 KSP Residual norm 1.612167203819e-12 1 SNES Function norm 1.599350481360e-12 ``` Using the fieldsplit additive preconditioner, the problem converges in a single KSP iteration, as expected. However, I do not understand why the residual of fieldsplit_0 (1e+0) does not coincide with the outer residual (462.13689530272404). It should be the case given that only fieldsplit_0 has a non-zero residual contribution. The fact that it is just 1 is suspicious. Is there something about how the fieldsplit works that I am missing? Thanks, Miguel MIGUEL ANGEL SALAZAR DE TROYA Head of Software Engineering miguel.salazar at corintis.com Corintis SA EPFL Innovation Park Building C 1015 Lausanne <2024-08-1609_02_11-re_mailmigrationfromgoogletooffice365-sebastien.gobel at corintis.com-co_4456a1bf-dd3e-46b7-85f9-21f278e66a79.png> Here at Corintis we care for your privacy. That is why we have taken appropriate measures to ensure that the data you have provided to us is always secure. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 2024-08-1609_02_11-re_mailmigrationfromgoogletooffice365-sebastien.gobel at corintis.com-co_4456a1bf-dd3e-46b7-85f9-21f278e66a79.png Type: image/png Size: 38219 bytes Desc: 2024-08-1609_02_11-re_mailmigrationfromgoogletooffice365-sebastien.gobel at corintis.com-co_4456a1bf-dd3e-46b7-85f9-21f278e66a79.png URL: From junchao.zhang at gmail.com Tue Sep 17 11:09:44 2024 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Tue, 17 Sep 2024 11:09:44 -0500 Subject: [petsc-users] [MPI GPU Aware] KSP_DIVERGED In-Reply-To: References: <99843d6c-206c-07bd-33a6-dce8e311643a@fastmail.org> <7cb0513bb7a14f1eaca0a314b6e55749@cea.fr> <4544e89c-7611-dfc4-7295-15d57e2d9395@fastmail.org> Message-ID: Did you "fix" the problem with OpenMPI 5, but keep petsc unchanged (ie., still 3.20)? --Junchao Zhang On Tue, Sep 17, 2024 at 9:47?AM LEDAC Pierre wrote: > Thanks Satish, and nice guess for OpenMPI 5 ! > > > It seems it solves the issue (at least on my GPU box where I reproduced > the issue with 8 MPI ranks with OpenMPI 4.x). > > > Unhappily, all the clusters we currently use have no module with OpenMPI > 5.x. Seems I need to build it to really confirm. > > > Probably we will prevent users from configuring our code with OpenMPI-cuda > 4.x cause it is really a weird bug. > > > Pierre LEDAC > Commissariat ? l??nergie atomique et aux ?nergies alternatives > Centre de SACLAY > DES/ISAS/DM2S/SGLS/LCAN > B?timent 451 ? point courrier n?43 > F-91191 Gif-sur-Yvette > +33 1 69 08 04 03 > +33 6 83 42 05 79 > ------------------------------ > *De :* Satish Balay > *Envoy? :* mardi 17 septembre 2024 15:39:22 > *? :* LEDAC Pierre > *Cc :* Junchao Zhang; petsc-users; ROUMET Elie > *Objet :* Re: [petsc-users] [MPI GPU Aware] KSP_DIVERGED > > On Tue, 17 Sep 2024, LEDAC Pierre wrote: > > > Thanks all, I will try and report. > > > > > > Last question, if I use "-use_gpu_aware_mpi 0" flag with a MPI GPU Aware > library, do PETSc > > > > disable GPU intra/inter communications and send MPI buffers as usual > (with extra Device<->Host copies) ? > > Yes. > > Not: Wrt using MPI that is not GPU-aware - we are changing the default > behavior - to not require "-use_gpu_aware_mpi 0" flag. > > https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/merge_requests/7813__;!!G_uCfscf7eWS!ai2MqJBzpM6efcbqDSWP3wTdMXpLjCabfiqf3nJWv7JMm3lBkWsWTyi5UaKoGoNH9DCZwkWaLljMmNPNXQCBk3rS5-fw$ > > > Satish > > > > > > > Thanks, > > > > > > Pierre LEDAC > > Commissariat ? l??nergie atomique et aux ?nergies alternatives > > Centre de SACLAY > > DES/ISAS/DM2S/SGLS/LCAN > > B?timent 451 ? point courrier n?43 > > F-91191 Gif-sur-Yvette > > +33 1 69 08 04 03 > > +33 6 83 42 05 79 > > ________________________________ > > De : Satish Balay > > Envoy? : lundi 16 septembre 2024 18:57:02 > > ? : Junchao Zhang > > Cc : LEDAC Pierre; petsc-users at mcs.anl.gov; ROUMET Elie > > Objet : Re: [petsc-users] [MPI GPU Aware] KSP_DIVERGED > > > > And/Or - try latest OpenMPI [or MPICH] and see if that makes a > difference. > > > > --download-mpich or --download-openmpi with latest petsc should build > gpu-aware-mpi > > > > Satish > > > > On Mon, 16 Sep 2024, Junchao Zhang wrote: > > > > > Could you try petsc/main to see if the problem persists? > > > > > > --Junchao Zhang > > > > > > > > > On Mon, Sep 16, 2024 at 10:51?AM LEDAC Pierre > wrote: > > > > > > > Hi all, > > > > > > > > > > > > We are using PETSc 3.20 in our code and running succesfully several > > > > solvers on Nvidia GPU with OpenMPI library which are not GPU aware > (so I > > > > need to add the flag -use_gpu_aware_mpi 0). > > > > > > > > > > > > But now, when using OpenMPI GPU Aware library (OpenMPI 4.0.5 ou > 4.1.5 from > > > > NVHPC), some parallel calculations failed with *KSP_DIVERGED_ITS* or > > > > *KSP_DIVERGED_DTOL* > > > > > > > > with several configurations. It may run wells on a small test case > with > > > > (matrix is symmetric): > > > > > > > > > > > > *-ksp_type cg -pc_type gamg -pc_gamg_type classical* > > > > > > > > > > > > But suddenly with a number of devices for instance bigger than 4 or > 8, it > > > > may fail. > > > > > > > > > > > > If I switch to another solver (BiCGstab), it may converge: > > > > > > > > > > > > *-ksp_type bcgs -pc_type gamg -pc_gamg_type classical* > > > > > > > > > > > > The more sensitive cases where it diverges are the following: > > > > > > > > > > > > *-ksp_type cg -pc_type hypre -pc_hypre_type boomeramg * > > > > > > > > *-ksp_type cg -pc_type gamg -pc_gamg_type classical* > > > > > > > > > > > > And the *bcgs* turnaroud doesn't work each time... > > > > > > > > > > > > It seems to work without problem with aggregation (at least 128 GPUs > on my > > > > simulation): > > > > > > > > *-ksp_type cg -pc_type gamg -pc_gamg_type agg* > > > > > > > > > > > > So I guess there is a weird thing happening in my code during the > solve in > > > > PETSc with MPI GPU Aware, as all the previous configurations works > with non > > > > GPU aware MPI. > > > > > > > > > > > > Here is the -ksp_view log during one fail with the first > configuration: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *KSP Object: () 8 MPI processes type: cg maximum > iterations=10000, > > > > nonzero initial guess tolerances: relative=0., absolute=0.0001, > > > > divergence=10000. left preconditioning using UNPRECONDITIONED > norm type > > > > for convergence test PC Object: () 8 MPI processes type: hypre > HYPRE > > > > BoomerAMG preconditioning Cycle type V Maximum number of > levels > > > > 25 Maximum number of iterations PER hypre call 1 > Convergence > > > > tolerance PER hypre call 0. Threshold for strong coupling 0.7 > > > > Interpolation truncation factor 0. Interpolation: max elements > per > > > > row 0 Number of levels of aggressive coarsening 0 Number > of > > > > paths for aggressive coarsening 1 Maximum row sums 0.9 > Sweeps > > > > down 1 Sweeps up 1 Sweeps on coarse > 1 > > > > Relax down l1scaled-Jacobi Relax up > > > > l1scaled-Jacobi Relax on coarse Gaussian-elimination > Relax > > > > weight (all) 1. Outer relax weight (all) 1. > Maximum size > > > > of coarsest grid 9 Minimum size of coarsest grid 1 Not > using > > > > CF-relaxation Not using more complex smoothers. Measure > > > > type local Coarsen type PMIS Interpolation > type > > > > ext+i SpGEMM type cusparse linear system matrix = > precond > > > > matrix: Mat Object: () 8 MPI processes type: mpiaijcusparse > > > > rows=64000, cols=64000 total: nonzeros=311040, allocated > > > > nonzeros=311040 total number of mallocs used during MatSetValues > > > > calls=0 not using I-node (on process 0) routines* > > > > > > > > > > > > I didn't succeed for the moment creating a reproducer with ex.c > examples... > > > > > > > > > > > > Did you see this kind of behaviour before? > > > > > > > > Should I update my PETSc version ? > > > > > > > > > > > > Thanks for any advice, > > > > > > > > > > > > Pierre LEDAC > > > > Commissariat ? l??nergie atomique et aux ?nergies alternatives > > > > Centre de SACLAY > > > > DES/ISAS/DM2S/SGLS/LCAN > > > > B?timent 451 ? point courrier n?43 > > > > F-91191 Gif-sur-Yvette > > > > +33 1 69 08 04 03 > > > > +33 6 83 42 05 79 > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Tue Sep 17 12:03:54 2024 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 17 Sep 2024 13:03:54 -0400 Subject: [petsc-users] Residual at the fieldsplit level In-Reply-To: References: <0e58dc19-e943-4d42-9a05-4cabbcbc8185.d0d10b7c-e93e-4ca5-8394-d4339a327c55.8303ace2-c862-49ea-97f8-20db7341a75a@emailsignatures365.codetwo.com> <29C71683-CADA-4F66-8A0A-40459076323C@petsc.dev> <4E5EDC3B-0D39-4B7F-AEB7-7854B3BC9A24@petsc.dev> <4131CBF5-CDF8-44E8-A994-0BFE57C1DB40@petsc.dev> <5E923642-A231-42A8-8958-470352157CDD@petsc.dev> Message-ID: <24668AB2-95CA-4F5E-9854-167FBDF1829F@petsc.dev> static PetscErrorCode KSPFGMRESCycle(PetscInt *itcount, KSP ksp) { ... PetscCall(VecNorm(VEC_VV(0), NORM_2, &res_norm)); /* The first entry in the right-hand side of the Hessenberg system is just the initial residual norm */ *RS(0) = res_norm; ksp->rnorm = res_norm; ... /* scale VEC_VV (the initial residual) */ PetscCall(VecScale(VEC_VV(0), 1.0 / res_norm)); So, it is expected behavior. GMRES scales the initial residual so it has norm 1, chugs along doing its thing and then when it builds the solution at the end, it knows it has solved the scaled problem so unscales it in KSPFGMRESBuildSoln() with PetscCall(VecMAXPBY(VEC_TEMP, it + 1, nrs, 0, &PREVEC(0))); the first entry of nrs[0] contains what was set in *RS(0) = res_norm; The KSPMonitor on the outer GMRES knows that the GMRES is solving a scaled right-hand side, so it still prints the correct original unscaled residual at each iteration. It is a peculiar feature of the GMES family, the subproblems that are solved are scaled from the original problem. Presumably, this is also true for block Jacobi etc. > On Sep 17, 2024, at 11:48?AM, miguel.salazar wrote: > > Is this the expected behavior though? I am not sure if it might be an issue with the Firedrake implementation. Getting a minimum example on PETSc like this one is not trivial > > Miguel > > > > > MIGUEL ANGEL SALAZAR DE TROYA > Head of Software Engineering > miguel.salazar at corintis.com > Corintis SA > EPFL Innovation Park Building C > 1015 Lausanne > <2024-08-1609_02_11-re_mailmigrationfromgoogletooffice365-sebastien.gobel at corintis.com-co_4456a1bf-dd3e-46b7-85f9-21f278e66a79.png> > Here at Corintis we care for your privacy. That is why we have taken appropriate measures to ensure that the data you have provided to us is always secure. > From: Barry Smith > > Date: Tuesday, 17 September 2024 at 16:58 > To: miguel.salazar > > Cc: petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] Residual at the fieldsplit level > > > Ok. Please ignore my last message about fieldsplit_0_fieldsplit_0 I misunderstood > > > > On Sep 17, 2024, at 10:52?AM, miguel.salazar > wrote: > > That is correct > > > MIGUEL ANGEL SALAZAR DE TROYA > Head of Software Engineering > miguel.salazar at corintis.com > Corintis SA > EPFL Innovation Park Building C > 1015 Lausanne > > <2024-08-1609_02_11-re_mailmigrationfromgoogletooffice365-sebastien.gobel at corintis.com-co_4456a1bf-dd3e-46b7-85f9-21f278e66a79.png> > Here at Corintis we care for your privacy. That is why we have taken appropriate measures to ensure that the data you have provided to us is always secure. > From: Barry Smith > > Date: Tuesday, 17 September 2024 at 16:49 > To: miguel.salazar > > Cc: petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] Residual at the fieldsplit level > > > So firedrake_0 is the object prefix of your SNES object? > > > > On Sep 17, 2024, at 10:37?AM, miguel.salazar > wrote: > > >In my reading, it says > >Residual norms for firedrake_0_ solve. > > 0 KSP Residual norm 4.621368953027e+02 > > > so the residual in the first (0th) split does correspond to the outer-most KSP residual norm (not printed but equal to the SNES at the first iteration of Newton. > > > It is the residual norm of the first field of the first field that is 1 > > >Residual norms for firedrake_0_fieldsplit_0_ solve. > > 0 KSP Residual norm 1.000000000000e+00 > > Residual norms for firedrake_0_ solve. > 0 KSP Residual norm 4.621368953027e+02 > > Is the residual of the outer solve (it is being printed). > > Residual norms for firedrake_0_fieldsplit_0_ solve. > 0 KSP Residual norm 1.000000000000e+00 > > Is the residual of the first field. I am not sure what you are referring to with ?the residual norm of the first field of the first field? > > Thanks, > Miguel > > > > > MIGUEL ANGEL SALAZAR DE TROYA > Head of Software Engineering > miguel.salazar at corintis.com > Corintis SA > EPFL Innovation Park Building C > 1015 Lausanne > > <2024-08-1609_02_11-re_mailmigrationfromgoogletooffice365-sebastien.gobel at corintis.com-co_4456a1bf-dd3e-46b7-85f9-21f278e66a79.png> > Here at Corintis we care for your privacy. That is why we have taken appropriate measures to ensure that the data you have provided to us is always secure. > From: Barry Smith > > Date: Tuesday, 17 September 2024 at 16:24 > To: miguel.salazar > > Cc: petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] Residual at the fieldsplit level > > > Initial residual: 462.13689530272404 > 0 SNES Function norm 4.621368953027e+02 > Residual norms for firedrake_0_ solve. > 0 KSP Residual norm 4.621368953027e+02 > Residual norms for firedrake_0_fieldsplit_0_ solve. > 0 KSP Residual norm 1.000000000000e+00 > 1 KSP Residual norm 3.501082228626e-15 > Residual norms for firedrake_0_fieldsplit_1_ solve. > 0 KSP Residual norm 0.000000000000e+00 > 1 KSP Residual norm 0.000000000000e+00 > 1 KSP Residual norm 1.612167203819e-12 > 1 SNES Function norm 1.599350481360e-12 > ``` > > Using the fieldsplit additive preconditioner, the problem converges in a single KSP iteration, as expected. However, I do not understand why the residual of fieldsplit_0 (1e+0) does not coincide with the outer residual (462.13689530272404). It should be the case given that only fieldsplit_0 has a non-zero residual contribution. The fact that it is just 1 is suspicious. Is there something about how the fieldsplit works that I am missing? > > In my reading, it says > Residual norms for firedrake_0_ solve. > 0 KSP Residual norm 4.621368953027e+02 > > so the residual in the first (0th) split does correspond to the outer-most KSP residual norm (not printed but equal to the SNES at the first iteration of Newton. > > It is the residual norm of the first field of the first field that is 1 > > Residual norms for firedrake_0_fieldsplit_0_ solve. > 0 KSP Residual norm 1.000000000000e+00 > > Or am I missing something > > Barry > > > > > > > > Initial residual: 462.13689530272404 > 0 SNES Function norm 4.621368953027e+02 > Residual norms for firedrake_0_ solve. > 0 KSP Residual norm 4.621368953027e+02 > Residual norms for firedrake_0_ solve. > 0 KSP unpreconditioned resid norm 4.621368953027e+02 true resid norm 4.621368953027e+02 ||r(i)||/||b|| 1.000000000000e+00 > Residual norms for firedrake_0_fieldsplit_0_ solve. > 0 KSP Residual norm 1.000000000000e+00 > Residual norms for firedrake_0_fieldsplit_0_ solve. > 0 KSP none resid norm 1.000000000000e+00 true resid norm 3.501082228626e-15 ||r(i)||/||b|| 3.501082228626e-15 > 1 KSP Residual norm 3.501082228626e-15 > 1 KSP none resid norm 3.501082228626e-15 true resid norm 3.501082228626e-15 ||r(i)||/||b|| 3.501082228626e-15 > Residual norms for firedrake_0_fieldsplit_1_ solve. > 0 KSP Residual norm 0.000000000000e+00 > Residual norms for firedrake_0_fieldsplit_1_ solve. > 0 KSP none resid norm 0.000000000000e+00 true resid norm 0.000000000000e+00 ||r(i)||/||b|| inf > 1 KSP Residual norm 0.000000000000e+00 > 1 KSP none resid norm 0.000000000000e+00 true resid norm 0.000000000000e+00 ||r(i)||/||b|| inf > 1 KSP Residual norm 1.612167203819e-12 > 1 KSP unpreconditioned resid norm 1.612167203819e-12 true resid norm 1.589286585800e-12 ||r(i)||/||b|| 3.438995245681e-15 > 1 SNES Function norm 1.599350481360e-12 > > > > On Sep 17, 2024, at 1:04?AM, miguel.salazar > wrote: > > These are the new options (with ksp_monitor_true_residual) > > solver_mumps_assembled = { > "ksp_type": "preonly", > "ksp_monitor": None, > "ksp_monitor_true_residual": None, > "pc_type": "python", > "pc_python_type": "firedrake.AssembledPC", > "assembled_pc_type": "lu", > "assembled_pc_factor_mat_solver_type": "mumps", > "assembled_mat_mumps_icntl_14": 200, > "assembled_mat_mumps_icntl_24": 1, > } > solver_fieldsplit = { > "mat_type": "matfree", > "snes_monitor": None, > "snes_type": "newtonls", > "ksp_type": "fgmres", > "ksp_monitor_true_residual": None, > "ksp_rtol": 1e-1, > "ksp_monitor": None, > "pc_type": "fieldsplit", > "pc_fieldsplit_type": "additive", > "fieldsplit_0": solver_mumps_assembled, > "fieldsplit_1": solver_mumps_assembled, > } > > And this is the output > > Initial residual: 462.13689530272404 > 0 SNES Function norm 4.621368953027e+02 > Residual norms for firedrake_0_ solve. > 0 KSP Residual norm 4.621368953027e+02 > Residual norms for firedrake_0_ solve. > 0 KSP unpreconditioned resid norm 4.621368953027e+02 true resid norm 4.621368953027e+02 ||r(i)||/||b|| 1.000000000000e+00 > Residual norms for firedrake_0_fieldsplit_0_ solve. > 0 KSP Residual norm 1.000000000000e+00 > Residual norms for firedrake_0_fieldsplit_0_ solve. > 0 KSP none resid norm 1.000000000000e+00 true resid norm 3.501082228626e-15 ||r(i)||/||b|| 3.501082228626e-15 > 1 KSP Residual norm 3.501082228626e-15 > 1 KSP none resid norm 3.501082228626e-15 true resid norm 3.501082228626e-15 ||r(i)||/||b|| 3.501082228626e-15 > Residual norms for firedrake_0_fieldsplit_1_ solve. > 0 KSP Residual norm 0.000000000000e+00 > Residual norms for firedrake_0_fieldsplit_1_ solve. > 0 KSP none resid norm 0.000000000000e+00 true resid norm 0.000000000000e+00 ||r(i)||/||b|| inf > 1 KSP Residual norm 0.000000000000e+00 > 1 KSP none resid norm 0.000000000000e+00 true resid norm 0.000000000000e+00 ||r(i)||/||b|| inf > 1 KSP Residual norm 1.612167203819e-12 > 1 KSP unpreconditioned resid norm 1.612167203819e-12 true resid norm 1.589286585800e-12 ||r(i)||/||b|| 3.438995245681e-15 > 1 SNES Function norm 1.599350481360e-12 > > The true residual is very low. (3.501082228626e-15) > > > MIGUEL ANGEL SALAZAR DE TROYA > Head of Software Engineering > miguel.salazar at corintis.com > Corintis SA > EPFL Innovation Park Building C > 1015 Lausanne > > <2024-08-1609_02_11-re_mailmigrationfromgoogletooffice365-sebastien.gobel at corintis.com-co_4456a1bf-dd3e-46b7-85f9-21f278e66a79.png> > Here at Corintis we care for your privacy. That is why we have taken appropriate measures to ensure that the data you have provided to us is always secure. > From: Barry Smith > > Date: Monday, 16 September 2024 at 19:34 > To: miguel.salazar > > Cc: petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] Residual at the fieldsplit level > > > Try ksp_monitor_true_residual. Let's see if it is an issue of preconditioned vs unpreconditioned residual. > > > > > On Sep 16, 2024, at 1:05?PM, miguel.salazar > wrote: > > Hello, > > I have this simple example on Firedrake to illustrate my point. I am solving for a two-component poisson equation (uncoupled). Only the first component has a non-zero residual. > > ``` > import firedrake as fd > from firedrake import inner, grad, dx, sin, pi > > N = 10 > mesh = fd.UnitSquareMesh(N, N) > V = fd.FunctionSpace(mesh, "CG", 1) > W = V * V > u = fd.Function(W) > v = fd.TestFunction(W) > a = inner(grad(u[0]), grad(v[0])) * dx + inner(grad(u[1]), grad(v[1])) * dx > x = fd.SpatialCoordinate(mesh) > f = fd.Function(V) > f.interpolate(fd.Constant(1e4) * sin(x[0] * pi) * sin(2 * x[1] * pi)) > L = f * v[0] * dx > F = a - L > bcs = [fd.DirichletBC(W.sub(0), fd.Constant(2.0), (1,))] > > > def snes_firedrake_residual(F, u, bcs): > for bcs_ in bcs: > bcs_.apply(u) > residual = fd.assemble(F, bcs=bcs, zero_bc_nodes=True) > with residual.dat.vec_ro as r: > print("Initial residual:", r.norm()) > > > snes_firedrake_residual(F, u, bcs) > > > problem = fd.NonlinearVariationalProblem(F, u, bcs=bcs) > solver_mumps_assembled = { > "ksp_type": "preonly", > "ksp_monitor": None, > "pc_type": "python", > "pc_python_type": "firedrake.AssembledPC", > "assembled_pc_type": "lu", > "assembled_pc_factor_mat_solver_type": "mumps", > "assembled_mat_mumps_icntl_14": 200, > "assembled_mat_mumps_icntl_24": 1, > } > solver_fieldsplit = { > "mat_type": "matfree", > "snes_type": "newtonls", > "ksp_type": "fgmres", > "ksp_rtol": 1e-1, > "ksp_monitor": None, > "pc_type": "fieldsplit", > "pc_fieldsplit_type": "additive", > "fieldsplit_0": solver_mumps_assembled, > "fieldsplit_1": solver_mumps_assembled, > } > > solver = fd.NonlinearVariationalSolver(problem, solver_parameters=solver_fieldsplit) > solver.solve() > ``` > > The PETSc output is as follow > > ``` > Initial residual: 462.13689530272404 > 0 SNES Function norm 4.621368953027e+02 > Residual norms for firedrake_0_ solve. > 0 KSP Residual norm 4.621368953027e+02 > Residual norms for firedrake_0_fieldsplit_0_ solve. > 0 KSP Residual norm 1.000000000000e+00 > 1 KSP Residual norm 3.501082228626e-15 > Residual norms for firedrake_0_fieldsplit_1_ solve. > 0 KSP Residual norm 0.000000000000e+00 > 1 KSP Residual norm 0.000000000000e+00 > 1 KSP Residual norm 1.612167203819e-12 > 1 SNES Function norm 1.599350481360e-12 > ``` > > Using the fieldsplit additive preconditioner, the problem converges in a single KSP iteration, as expected. However, I do not understand why the residual of fieldsplit_0 (1e+0) does not coincide with the outer residual (462.13689530272404). It should be the case given that only fieldsplit_0 has a non-zero residual contribution. The fact that it is just 1 is suspicious. Is there something about how the fieldsplit works that I am missing? > > Thanks, > Miguel > > MIGUEL ANGEL SALAZAR DE TROYA > Head of Software Engineering > miguel.salazar at corintis.com > Corintis SA > EPFL Innovation Park Building C > 1015 Lausanne > > <2024-08-1609_02_11-re_mailmigrationfromgoogletooffice365-sebastien.gobel at corintis.com-co_4456a1bf-dd3e-46b7-85f9-21f278e66a79.png> > Here at Corintis we care for your privacy. That is why we have taken appropriate measures to ensure that the data you have provided to us is always secure. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Pierre.LEDAC at cea.fr Tue Sep 17 12:43:32 2024 From: Pierre.LEDAC at cea.fr (LEDAC Pierre) Date: Tue, 17 Sep 2024 17:43:32 +0000 Subject: [petsc-users] [MPI GPU Aware] KSP_DIVERGED In-Reply-To: References: <99843d6c-206c-07bd-33a6-dce8e311643a@fastmail.org> <7cb0513bb7a14f1eaca0a314b6e55749@cea.fr> <4544e89c-7611-dfc4-7295-15d57e2d9395@fastmail.org> , Message-ID: <9ab9417628a6456b81b65507297daed1@cea.fr> Yes. Only OpenMPI 5.0.5 with Petsc 3.20. Pierre LEDAC Commissariat ? l??nergie atomique et aux ?nergies alternatives Centre de SACLAY DES/ISAS/DM2S/SGLS/LCAN B?timent 451 ? point courrier n?43 F-91191 Gif-sur-Yvette +33 1 69 08 04 03 +33 6 83 42 05 79 ________________________________ De : Junchao Zhang Envoy? : mardi 17 septembre 2024 18:09:44 ? : LEDAC Pierre Cc : petsc-users; ROUMET Elie Objet : Re: [petsc-users] [MPI GPU Aware] KSP_DIVERGED Did you "fix" the problem with OpenMPI 5, but keep petsc unchanged (ie., still 3.20)? --Junchao Zhang On Tue, Sep 17, 2024 at 9:47?AM LEDAC Pierre > wrote: Thanks Satish, and nice guess for OpenMPI 5 ! It seems it solves the issue (at least on my GPU box where I reproduced the issue with 8 MPI ranks with OpenMPI 4.x). Unhappily, all the clusters we currently use have no module with OpenMPI 5.x. Seems I need to build it to really confirm. Probably we will prevent users from configuring our code with OpenMPI-cuda 4.x cause it is really a weird bug. Pierre LEDAC Commissariat ? l??nergie atomique et aux ?nergies alternatives Centre de SACLAY DES/ISAS/DM2S/SGLS/LCAN B?timent 451 ? point courrier n?43 F-91191 Gif-sur-Yvette +33 1 69 08 04 03 +33 6 83 42 05 79 ________________________________ De : Satish Balay > Envoy? : mardi 17 septembre 2024 15:39:22 ? : LEDAC Pierre Cc : Junchao Zhang; petsc-users; ROUMET Elie Objet : Re: [petsc-users] [MPI GPU Aware] KSP_DIVERGED On Tue, 17 Sep 2024, LEDAC Pierre wrote: > Thanks all, I will try and report. > > > Last question, if I use "-use_gpu_aware_mpi 0" flag with a MPI GPU Aware library, do PETSc > > disable GPU intra/inter communications and send MPI buffers as usual (with extra Device<->Host copies) ? Yes. Not: Wrt using MPI that is not GPU-aware - we are changing the default behavior - to not require "-use_gpu_aware_mpi 0" flag. https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/merge_requests/7813__;!!G_uCfscf7eWS!a7fRmVH3eqR_0f_HiFDjLrRAYPYq1kBnMtoRTo97kVEXQj-HKv_0UKQF-sOah1FOXhIZhEXytfhmQBviU5wLFvXOR5DJ$ Satish > > > Thanks, > > > Pierre LEDAC > Commissariat ? l??nergie atomique et aux ?nergies alternatives > Centre de SACLAY > DES/ISAS/DM2S/SGLS/LCAN > B?timent 451 ? point courrier n?43 > F-91191 Gif-sur-Yvette > +33 1 69 08 04 03 > +33 6 83 42 05 79 > ________________________________ > De : Satish Balay > > Envoy? : lundi 16 septembre 2024 18:57:02 > ? : Junchao Zhang > Cc : LEDAC Pierre; petsc-users at mcs.anl.gov; ROUMET Elie > Objet : Re: [petsc-users] [MPI GPU Aware] KSP_DIVERGED > > And/Or - try latest OpenMPI [or MPICH] and see if that makes a difference. > > --download-mpich or --download-openmpi with latest petsc should build gpu-aware-mpi > > Satish > > On Mon, 16 Sep 2024, Junchao Zhang wrote: > > > Could you try petsc/main to see if the problem persists? > > > > --Junchao Zhang > > > > > > On Mon, Sep 16, 2024 at 10:51?AM LEDAC Pierre > wrote: > > > > > Hi all, > > > > > > > > > We are using PETSc 3.20 in our code and running succesfully several > > > solvers on Nvidia GPU with OpenMPI library which are not GPU aware (so I > > > need to add the flag -use_gpu_aware_mpi 0). > > > > > > > > > But now, when using OpenMPI GPU Aware library (OpenMPI 4.0.5 ou 4.1.5 from > > > NVHPC), some parallel calculations failed with *KSP_DIVERGED_ITS* or > > > *KSP_DIVERGED_DTOL* > > > > > > with several configurations. It may run wells on a small test case with > > > (matrix is symmetric): > > > > > > > > > *-ksp_type cg -pc_type gamg -pc_gamg_type classical* > > > > > > > > > But suddenly with a number of devices for instance bigger than 4 or 8, it > > > may fail. > > > > > > > > > If I switch to another solver (BiCGstab), it may converge: > > > > > > > > > *-ksp_type bcgs -pc_type gamg -pc_gamg_type classical* > > > > > > > > > The more sensitive cases where it diverges are the following: > > > > > > > > > *-ksp_type cg -pc_type hypre -pc_hypre_type boomeramg * > > > > > > *-ksp_type cg -pc_type gamg -pc_gamg_type classical* > > > > > > > > > And the *bcgs* turnaroud doesn't work each time... > > > > > > > > > It seems to work without problem with aggregation (at least 128 GPUs on my > > > simulation): > > > > > > *-ksp_type cg -pc_type gamg -pc_gamg_type agg* > > > > > > > > > So I guess there is a weird thing happening in my code during the solve in > > > PETSc with MPI GPU Aware, as all the previous configurations works with non > > > GPU aware MPI. > > > > > > > > > Here is the -ksp_view log during one fail with the first configuration: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *KSP Object: () 8 MPI processes type: cg maximum iterations=10000, > > > nonzero initial guess tolerances: relative=0., absolute=0.0001, > > > divergence=10000. left preconditioning using UNPRECONDITIONED norm type > > > for convergence test PC Object: () 8 MPI processes type: hypre HYPRE > > > BoomerAMG preconditioning Cycle type V Maximum number of levels > > > 25 Maximum number of iterations PER hypre call 1 Convergence > > > tolerance PER hypre call 0. Threshold for strong coupling 0.7 > > > Interpolation truncation factor 0. Interpolation: max elements per > > > row 0 Number of levels of aggressive coarsening 0 Number of > > > paths for aggressive coarsening 1 Maximum row sums 0.9 Sweeps > > > down 1 Sweeps up 1 Sweeps on coarse 1 > > > Relax down l1scaled-Jacobi Relax up > > > l1scaled-Jacobi Relax on coarse Gaussian-elimination Relax > > > weight (all) 1. Outer relax weight (all) 1. Maximum size > > > of coarsest grid 9 Minimum size of coarsest grid 1 Not using > > > CF-relaxation Not using more complex smoothers. Measure > > > type local Coarsen type PMIS Interpolation type > > > ext+i SpGEMM type cusparse linear system matrix = precond > > > matrix: Mat Object: () 8 MPI processes type: mpiaijcusparse > > > rows=64000, cols=64000 total: nonzeros=311040, allocated > > > nonzeros=311040 total number of mallocs used during MatSetValues > > > calls=0 not using I-node (on process 0) routines* > > > > > > > > > I didn't succeed for the moment creating a reproducer with ex.c examples... > > > > > > > > > Did you see this kind of behaviour before? > > > > > > Should I update my PETSc version ? > > > > > > > > > Thanks for any advice, > > > > > > > > > Pierre LEDAC > > > Commissariat ? l??nergie atomique et aux ?nergies alternatives > > > Centre de SACLAY > > > DES/ISAS/DM2S/SGLS/LCAN > > > B?timent 451 ? point courrier n?43 > > > F-91191 Gif-sur-Yvette > > > +33 1 69 08 04 03 > > > +33 6 83 42 05 79 > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Pierre.LEDAC at cea.fr Wed Sep 18 05:09:26 2024 From: Pierre.LEDAC at cea.fr (LEDAC Pierre) Date: Wed, 18 Sep 2024 10:09:26 +0000 Subject: [petsc-users] [MPI GPU Aware] KSP_DIVERGED In-Reply-To: <9ab9417628a6456b81b65507297daed1@cea.fr> References: <99843d6c-206c-07bd-33a6-dce8e311643a@fastmail.org> <7cb0513bb7a14f1eaca0a314b6e55749@cea.fr> <4544e89c-7611-dfc4-7295-15d57e2d9395@fastmail.org> , , <9ab9417628a6456b81b65507297daed1@cea.fr> Message-ID: <8fc0e5d2115c4b018e223e60ffd84e8d@cea.fr> Junchao, I just tried PETSc last version in main branch with OpenMPI 4.1.5-cuda. I still get KSP_DIVERGED on some parallel calculations. Solved when moving to OpenMPI 5.0.5-cuda. Pierre LEDAC Commissariat ? l??nergie atomique et aux ?nergies alternatives Centre de SACLAY DES/ISAS/DM2S/SGLS/LCAN B?timent 451 ? point courrier n?43 F-91191 Gif-sur-Yvette +33 1 69 08 04 03 +33 6 83 42 05 79 ________________________________ De : LEDAC Pierre Envoy? : mardi 17 septembre 2024 19:43:32 ? : Junchao Zhang Cc : petsc-users; ROUMET Elie Objet : RE: [petsc-users] [MPI GPU Aware] KSP_DIVERGED Yes. Only OpenMPI 5.0.5 with Petsc 3.20. Pierre LEDAC Commissariat ? l??nergie atomique et aux ?nergies alternatives Centre de SACLAY DES/ISAS/DM2S/SGLS/LCAN B?timent 451 ? point courrier n?43 F-91191 Gif-sur-Yvette +33 1 69 08 04 03 +33 6 83 42 05 79 ________________________________ De : Junchao Zhang Envoy? : mardi 17 septembre 2024 18:09:44 ? : LEDAC Pierre Cc : petsc-users; ROUMET Elie Objet : Re: [petsc-users] [MPI GPU Aware] KSP_DIVERGED Did you "fix" the problem with OpenMPI 5, but keep petsc unchanged (ie., still 3.20)? --Junchao Zhang On Tue, Sep 17, 2024 at 9:47?AM LEDAC Pierre > wrote: Thanks Satish, and nice guess for OpenMPI 5 ! It seems it solves the issue (at least on my GPU box where I reproduced the issue with 8 MPI ranks with OpenMPI 4.x). Unhappily, all the clusters we currently use have no module with OpenMPI 5.x. Seems I need to build it to really confirm. Probably we will prevent users from configuring our code with OpenMPI-cuda 4.x cause it is really a weird bug. Pierre LEDAC Commissariat ? l??nergie atomique et aux ?nergies alternatives Centre de SACLAY DES/ISAS/DM2S/SGLS/LCAN B?timent 451 ? point courrier n?43 F-91191 Gif-sur-Yvette +33 1 69 08 04 03 +33 6 83 42 05 79 ________________________________ De : Satish Balay > Envoy? : mardi 17 septembre 2024 15:39:22 ? : LEDAC Pierre Cc : Junchao Zhang; petsc-users; ROUMET Elie Objet : Re: [petsc-users] [MPI GPU Aware] KSP_DIVERGED On Tue, 17 Sep 2024, LEDAC Pierre wrote: > Thanks all, I will try and report. > > > Last question, if I use "-use_gpu_aware_mpi 0" flag with a MPI GPU Aware library, do PETSc > > disable GPU intra/inter communications and send MPI buffers as usual (with extra Device<->Host copies) ? Yes. Not: Wrt using MPI that is not GPU-aware - we are changing the default behavior - to not require "-use_gpu_aware_mpi 0" flag. https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/merge_requests/7813__;!!G_uCfscf7eWS!db-DltLLuITAzFAoa15HftvoW9oirhN17uvo8HA7xQJQcN0G0fC_qzo5CIfet-vIvfaA3DjlkImfaOwx6IFL7nSAcQRd$ Satish > > > Thanks, > > > Pierre LEDAC > Commissariat ? l??nergie atomique et aux ?nergies alternatives > Centre de SACLAY > DES/ISAS/DM2S/SGLS/LCAN > B?timent 451 ? point courrier n?43 > F-91191 Gif-sur-Yvette > +33 1 69 08 04 03 > +33 6 83 42 05 79 > ________________________________ > De : Satish Balay > > Envoy? : lundi 16 septembre 2024 18:57:02 > ? : Junchao Zhang > Cc : LEDAC Pierre; petsc-users at mcs.anl.gov; ROUMET Elie > Objet : Re: [petsc-users] [MPI GPU Aware] KSP_DIVERGED > > And/Or - try latest OpenMPI [or MPICH] and see if that makes a difference. > > --download-mpich or --download-openmpi with latest petsc should build gpu-aware-mpi > > Satish > > On Mon, 16 Sep 2024, Junchao Zhang wrote: > > > Could you try petsc/main to see if the problem persists? > > > > --Junchao Zhang > > > > > > On Mon, Sep 16, 2024 at 10:51?AM LEDAC Pierre > wrote: > > > > > Hi all, > > > > > > > > > We are using PETSc 3.20 in our code and running succesfully several > > > solvers on Nvidia GPU with OpenMPI library which are not GPU aware (so I > > > need to add the flag -use_gpu_aware_mpi 0). > > > > > > > > > But now, when using OpenMPI GPU Aware library (OpenMPI 4.0.5 ou 4.1.5 from > > > NVHPC), some parallel calculations failed with *KSP_DIVERGED_ITS* or > > > *KSP_DIVERGED_DTOL* > > > > > > with several configurations. It may run wells on a small test case with > > > (matrix is symmetric): > > > > > > > > > *-ksp_type cg -pc_type gamg -pc_gamg_type classical* > > > > > > > > > But suddenly with a number of devices for instance bigger than 4 or 8, it > > > may fail. > > > > > > > > > If I switch to another solver (BiCGstab), it may converge: > > > > > > > > > *-ksp_type bcgs -pc_type gamg -pc_gamg_type classical* > > > > > > > > > The more sensitive cases where it diverges are the following: > > > > > > > > > *-ksp_type cg -pc_type hypre -pc_hypre_type boomeramg * > > > > > > *-ksp_type cg -pc_type gamg -pc_gamg_type classical* > > > > > > > > > And the *bcgs* turnaroud doesn't work each time... > > > > > > > > > It seems to work without problem with aggregation (at least 128 GPUs on my > > > simulation): > > > > > > *-ksp_type cg -pc_type gamg -pc_gamg_type agg* > > > > > > > > > So I guess there is a weird thing happening in my code during the solve in > > > PETSc with MPI GPU Aware, as all the previous configurations works with non > > > GPU aware MPI. > > > > > > > > > Here is the -ksp_view log during one fail with the first configuration: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *KSP Object: () 8 MPI processes type: cg maximum iterations=10000, > > > nonzero initial guess tolerances: relative=0., absolute=0.0001, > > > divergence=10000. left preconditioning using UNPRECONDITIONED norm type > > > for convergence test PC Object: () 8 MPI processes type: hypre HYPRE > > > BoomerAMG preconditioning Cycle type V Maximum number of levels > > > 25 Maximum number of iterations PER hypre call 1 Convergence > > > tolerance PER hypre call 0. Threshold for strong coupling 0.7 > > > Interpolation truncation factor 0. Interpolation: max elements per > > > row 0 Number of levels of aggressive coarsening 0 Number of > > > paths for aggressive coarsening 1 Maximum row sums 0.9 Sweeps > > > down 1 Sweeps up 1 Sweeps on coarse 1 > > > Relax down l1scaled-Jacobi Relax up > > > l1scaled-Jacobi Relax on coarse Gaussian-elimination Relax > > > weight (all) 1. Outer relax weight (all) 1. Maximum size > > > of coarsest grid 9 Minimum size of coarsest grid 1 Not using > > > CF-relaxation Not using more complex smoothers. Measure > > > type local Coarsen type PMIS Interpolation type > > > ext+i SpGEMM type cusparse linear system matrix = precond > > > matrix: Mat Object: () 8 MPI processes type: mpiaijcusparse > > > rows=64000, cols=64000 total: nonzeros=311040, allocated > > > nonzeros=311040 total number of mallocs used during MatSetValues > > > calls=0 not using I-node (on process 0) routines* > > > > > > > > > I didn't succeed for the moment creating a reproducer with ex.c examples... > > > > > > > > > Did you see this kind of behaviour before? > > > > > > Should I update my PETSc version ? > > > > > > > > > Thanks for any advice, > > > > > > > > > Pierre LEDAC > > > Commissariat ? l??nergie atomique et aux ?nergies alternatives > > > Centre de SACLAY > > > DES/ISAS/DM2S/SGLS/LCAN > > > B?timent 451 ? point courrier n?43 > > > F-91191 Gif-sur-Yvette > > > +33 1 69 08 04 03 > > > +33 6 83 42 05 79 > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sayosale at hotmail.com Wed Sep 18 07:20:54 2024 From: sayosale at hotmail.com (dazza simplythebest) Date: Wed, 18 Sep 2024 12:20:54 +0000 Subject: [petsc-users] How to create an array of vectors of type Vec (for use with slepc EPSGetInvariantSubspace) Message-ID: Dear Jose, Many thanks for the guidance. I haven't yet extensively tested the program, but I will cautiously claim that it seems to be working on the small test problem described below (i.e. a small modification of your ex1f.F program). I am currently using version 3.20, but I will upgrade to 3.22 when it comes out. For the Laplace problem with n=50 , using the gd method and seeking to find the two leading eigenmodes (nev=2) using a space of dimension ncv = 30, with default tolerance, on the first solution (using a randomized initial subspace) the algorithm requires 91 iterations to converge, whereas for the second solution (using the previous solution invariant subspace as an initial subspace) it finds the solution in 2 iterations. Many thanks once again and best wishes, Dan. >The Fortran interface is not fully tested. In particular, this function is not employed in any of the Fortran examples. >The Fortran interface has undergone successive improvements in recent times. I have tried the following: > Vec, pointer :: subspace_receive(:) > ... > PetscCall(EPSGetConverged(eps, nconv, ierr)) > PetscCall(MatCreateVecs(A, PETSC_NULL_VEC, xr, ierr)) > PetscCall(VecDuplicateVecsF90(xr, nconv, subspace_receive, ierr)) > PetscCall(EPSGetInvariantSubspace(eps, subspace_receive, ierr)) > ... > PetscCall(VecDestroyVecsF90(nconv, subspace_receive, ierr)) > PetscCall(VecDestroy(xr, ierr)) >I have tested this in the 'main' branch and it works as expected (note that 'main' will become version 3.22 in two weeks). You can try with version 3.21 and it may work, > probably not in older versions. * Jose > El 16 sept 2024, a las 11:07, dazza simplythebest escribi?: > > Dear All and Jose, > I am now trying to code up the extraction of the invariant subspace, but I seem to be running into > a compilation problem. I am using the slepc example file ex1f.F as a base, and have added the following lines: > > !Declaration > Vec subspace_receive > . > . > . > ! Executed code (following EPSSolve) > call EPSGetConverged(eps,nconv,ierr); CHKERRA(ierr) > call VecDuplicateVecs(xr, nconv, subspace_receive, ierr); CHKERRA(ierr) > call EPSGetInvariantSubspace(eps, subspace_receive, ierr) ; CHKERRA(ierr). > . > . > > However, I get the following compilation error when I include the call EPSGetInvariantSubspace(eps, subspace_receive, ierr) line: > > > minitest.F(133): error #8284: If the actual argument is scalar, the dummy argument shall be scalar unless the actual argument is of type character or is an element of an array that is not assumed shape, pointer, or polymorphic. [B] > call EPSGetInvariantSubspace(eps, subspace_receive, ierr) ; if (ierr .ne. 0) then;call PetscErrorF(ierr,133,"minitest.F");call MPIU_Abort(PETSC_COMM_SELF,ierr);endif > -----------^ > compilation aborted for minitest.F (code 1) > > > If I comment out the call EPSGetInvariantSubspace(eps, subspace_receive, ierr) line the code compiles, links and runs successfully. > I presume that I must be making a silly mistake in the above lines, but I can't quite see what it is. (There is some discussion in the > manual of there being a different situation if the matrix is real but the eigenvalues are complex, but for this case the eigenvalues > are real). Can you offer any guidance ?! I can provide the whole code of course if that would help. > Many thanks and best wishes, > Dan. > > > > > > From: Jose E. Roman > Sent: Monday, September 2, 2024 7:06 AM > To: dazza simplythebest > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] How to create an array of vectors of type Vec (for use with slepc EPSGetInvariantSubspace) > You can use VecDuplicateVecs() to create an array of Vec. > Yes, you can use VecLoad() to read the data from a binary file. > > Jose > > > > El 2 sept 2024, a las 8:56, dazza simplythebest escribi?: > > > > Dear All, > > I am seeking to perform a large number of eigenvalue calculations (related to a fluid dynamics > > problem) using the slepc framework. I have written the program that calls slepc and validated its solutions against > > known results for special cases - the Jacobi-Davidson iteration seemed to perform best on this problem. > > Since in the 'production-level' calculation I will step through a parameter space which should > > mean only small changes to the eigenproblem between each calculation, it seems to make sense > > to use the previous solution as the initial guess for the next calculation. > > > > The sequence to execute this strategy would seem to be: > > To write out the current soln subspace: > > > > call EPSGetInvariantSubspace > > [ call VecView -> binary file (if a file storage is required for possible restart)] > > > > To then read in this data to use it as the initial subspace for the next run: > > > > call EPSSetInitialSpace > > > > My question is that EPSGetInvariantSubspace needs an 'array of nconv vectors', how does one create > > such an array ? My understanding is that the Vec structure is just a one-dimensional array (with contents > > possibly scattered in non-contiguous memory locations !?) , so do we just need to create a long Vec > > that stores multiple vectors ? How can we do this (the examples I have seen thus far use a MatCreateVecs to specify the size > > of the Vec)? > > > > If loading such a subspace from a binary file (created maybe by VecView), presumably VecLoad is the > > best way to load such an array of vectors from the binary file ? > > > > Best wishes and many thanks, > > Dan. -------------- next part -------------- An HTML attachment was scrubbed... URL: From junchao.zhang at gmail.com Wed Sep 18 09:58:54 2024 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Wed, 18 Sep 2024 09:58:54 -0500 Subject: [petsc-users] [MPI GPU Aware] KSP_DIVERGED In-Reply-To: <8fc0e5d2115c4b018e223e60ffd84e8d@cea.fr> References: <99843d6c-206c-07bd-33a6-dce8e311643a@fastmail.org> <7cb0513bb7a14f1eaca0a314b6e55749@cea.fr> <4544e89c-7611-dfc4-7295-15d57e2d9395@fastmail.org> <9ab9417628a6456b81b65507297daed1@cea.fr> <8fc0e5d2115c4b018e223e60ffd84e8d@cea.fr> Message-ID: Pierre, thanks for the additional information. This is scary. If this is really an OpenMPI bug, we don't have a petsc test to catch it and currently have no clue what went wrong. --Junchao Zhang On Wed, Sep 18, 2024 at 5:09?AM LEDAC Pierre wrote: > Junchao, I just tried PETSc last version in main branch with OpenMPI > 4.1.5-cuda. I still get KSP_DIVERGED on > > some parallel calculations. Solved when moving to OpenMPI 5.0.5-cuda. > > > Pierre LEDAC > Commissariat ? l??nergie atomique et aux ?nergies alternatives > Centre de SACLAY > DES/ISAS/DM2S/SGLS/LCAN > B?timent 451 ? point courrier n?43 > F-91191 Gif-sur-Yvette > +33 1 69 08 04 03 > +33 6 83 42 05 79 > ------------------------------ > *De :* LEDAC Pierre > *Envoy? :* mardi 17 septembre 2024 19:43:32 > *? :* Junchao Zhang > *Cc :* petsc-users; ROUMET Elie > *Objet :* RE: [petsc-users] [MPI GPU Aware] KSP_DIVERGED > > > Yes. Only OpenMPI 5.0.5 with Petsc 3.20. > > > Pierre LEDAC > Commissariat ? l??nergie atomique et aux ?nergies alternatives > Centre de SACLAY > DES/ISAS/DM2S/SGLS/LCAN > B?timent 451 ? point courrier n?43 > F-91191 Gif-sur-Yvette > +33 1 69 08 04 03 > +33 6 83 42 05 79 > ------------------------------ > *De :* Junchao Zhang > *Envoy? :* mardi 17 septembre 2024 18:09:44 > *? :* LEDAC Pierre > *Cc :* petsc-users; ROUMET Elie > *Objet :* Re: [petsc-users] [MPI GPU Aware] KSP_DIVERGED > > Did you "fix" the problem with OpenMPI 5, but keep petsc unchanged (ie., > still 3.20)? > > --Junchao Zhang > > > On Tue, Sep 17, 2024 at 9:47?AM LEDAC Pierre wrote: > >> Thanks Satish, and nice guess for OpenMPI 5 ! >> >> >> It seems it solves the issue (at least on my GPU box where I reproduced >> the issue with 8 MPI ranks with OpenMPI 4.x). >> >> >> Unhappily, all the clusters we currently use have no module with OpenMPI >> 5.x. Seems I need to build it to really confirm. >> >> >> Probably we will prevent users from configuring our code with >> OpenMPI-cuda 4.x cause it is really a weird bug. >> >> >> Pierre LEDAC >> Commissariat ? l??nergie atomique et aux ?nergies alternatives >> Centre de SACLAY >> DES/ISAS/DM2S/SGLS/LCAN >> B?timent 451 ? point courrier n?43 >> F-91191 Gif-sur-Yvette >> +33 1 69 08 04 03 >> +33 6 83 42 05 79 >> ------------------------------ >> *De :* Satish Balay >> *Envoy? :* mardi 17 septembre 2024 15:39:22 >> *? :* LEDAC Pierre >> *Cc :* Junchao Zhang; petsc-users; ROUMET Elie >> *Objet :* Re: [petsc-users] [MPI GPU Aware] KSP_DIVERGED >> >> On Tue, 17 Sep 2024, LEDAC Pierre wrote: >> >> > Thanks all, I will try and report. >> > >> > >> > Last question, if I use "-use_gpu_aware_mpi 0" flag with a MPI GPU >> Aware library, do PETSc >> > >> > disable GPU intra/inter communications and send MPI buffers as usual >> (with extra Device<->Host copies) ? >> >> Yes. >> >> Not: Wrt using MPI that is not GPU-aware - we are changing the default >> behavior - to not require "-use_gpu_aware_mpi 0" flag. >> >> https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/merge_requests/7813__;!!G_uCfscf7eWS!fqIBdGt7byg9rSjen4SjjKyivmWEe_CLpOXOzAM1m3NAKmGoS2jsQQaeww4yYT0379vyu8UdqMoywsMM6JJJS1kdDwvw$ >> >> >> Satish >> >> > >> > >> > Thanks, >> > >> > >> > Pierre LEDAC >> > Commissariat ? l??nergie atomique et aux ?nergies alternatives >> > Centre de SACLAY >> > DES/ISAS/DM2S/SGLS/LCAN >> > B?timent 451 ? point courrier n?43 >> > F-91191 Gif-sur-Yvette >> > +33 1 69 08 04 03 >> > +33 6 83 42 05 79 >> > ________________________________ >> > De : Satish Balay >> > Envoy? : lundi 16 septembre 2024 18:57:02 >> > ? : Junchao Zhang >> > Cc : LEDAC Pierre; petsc-users at mcs.anl.gov; ROUMET Elie >> > Objet : Re: [petsc-users] [MPI GPU Aware] KSP_DIVERGED >> > >> > And/Or - try latest OpenMPI [or MPICH] and see if that makes a >> difference. >> > >> > --download-mpich or --download-openmpi with latest petsc should build >> gpu-aware-mpi >> > >> > Satish >> > >> > On Mon, 16 Sep 2024, Junchao Zhang wrote: >> > >> > > Could you try petsc/main to see if the problem persists? >> > > >> > > --Junchao Zhang >> > > >> > > >> > > On Mon, Sep 16, 2024 at 10:51?AM LEDAC Pierre >> wrote: >> > > >> > > > Hi all, >> > > > >> > > > >> > > > We are using PETSc 3.20 in our code and running succesfully several >> > > > solvers on Nvidia GPU with OpenMPI library which are not GPU aware >> (so I >> > > > need to add the flag -use_gpu_aware_mpi 0). >> > > > >> > > > >> > > > But now, when using OpenMPI GPU Aware library (OpenMPI 4.0.5 ou >> 4.1.5 from >> > > > NVHPC), some parallel calculations failed with *KSP_DIVERGED_ITS* or >> > > > *KSP_DIVERGED_DTOL* >> > > > >> > > > with several configurations. It may run wells on a small test case >> with >> > > > (matrix is symmetric): >> > > > >> > > > >> > > > *-ksp_type cg -pc_type gamg -pc_gamg_type classical* >> > > > >> > > > >> > > > But suddenly with a number of devices for instance bigger than 4 or >> 8, it >> > > > may fail. >> > > > >> > > > >> > > > If I switch to another solver (BiCGstab), it may converge: >> > > > >> > > > >> > > > *-ksp_type bcgs -pc_type gamg -pc_gamg_type classical* >> > > > >> > > > >> > > > The more sensitive cases where it diverges are the following: >> > > > >> > > > >> > > > *-ksp_type cg -pc_type hypre -pc_hypre_type boomeramg * >> > > > >> > > > *-ksp_type cg -pc_type gamg -pc_gamg_type classical* >> > > > >> > > > >> > > > And the *bcgs* turnaroud doesn't work each time... >> > > > >> > > > >> > > > It seems to work without problem with aggregation (at least 128 >> GPUs on my >> > > > simulation): >> > > > >> > > > *-ksp_type cg -pc_type gamg -pc_gamg_type agg* >> > > > >> > > > >> > > > So I guess there is a weird thing happening in my code during the >> solve in >> > > > PETSc with MPI GPU Aware, as all the previous configurations works >> with non >> > > > GPU aware MPI. >> > > > >> > > > >> > > > Here is the -ksp_view log during one fail with the first >> configuration: >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > *KSP Object: () 8 MPI processes type: cg maximum >> iterations=10000, >> > > > nonzero initial guess tolerances: relative=0., absolute=0.0001, >> > > > divergence=10000. left preconditioning using UNPRECONDITIONED >> norm type >> > > > for convergence test PC Object: () 8 MPI processes type: >> hypre HYPRE >> > > > BoomerAMG preconditioning Cycle type V Maximum number >> of levels >> > > > 25 Maximum number of iterations PER hypre call 1 >> Convergence >> > > > tolerance PER hypre call 0. Threshold for strong coupling 0.7 >> > > > Interpolation truncation factor 0. Interpolation: max >> elements per >> > > > row 0 Number of levels of aggressive coarsening 0 >> Number of >> > > > paths for aggressive coarsening 1 Maximum row sums 0.9 >> Sweeps >> > > > down 1 Sweeps up 1 Sweeps on >> coarse 1 >> > > > Relax down l1scaled-Jacobi Relax up >> > > > l1scaled-Jacobi Relax on coarse >> Gaussian-elimination Relax >> > > > weight (all) 1. Outer relax weight (all) 1. >> Maximum size >> > > > of coarsest grid 9 Minimum size of coarsest grid 1 Not >> using >> > > > CF-relaxation Not using more complex smoothers. Measure >> > > > type local Coarsen type PMIS >> Interpolation type >> > > > ext+i SpGEMM type cusparse linear system matrix = >> precond >> > > > matrix: Mat Object: () 8 MPI processes type: mpiaijcusparse >> > > > rows=64000, cols=64000 total: nonzeros=311040, allocated >> > > > nonzeros=311040 total number of mallocs used during MatSetValues >> > > > calls=0 not using I-node (on process 0) routines* >> > > > >> > > > >> > > > I didn't succeed for the moment creating a reproducer with ex.c >> examples... >> > > > >> > > > >> > > > Did you see this kind of behaviour before? >> > > > >> > > > Should I update my PETSc version ? >> > > > >> > > > >> > > > Thanks for any advice, >> > > > >> > > > >> > > > Pierre LEDAC >> > > > Commissariat ? l??nergie atomique et aux ?nergies alternatives >> > > > Centre de SACLAY >> > > > DES/ISAS/DM2S/SGLS/LCAN >> > > > B?timent 451 ? point courrier n?43 >> > > > F-91191 Gif-sur-Yvette >> > > > +33 1 69 08 04 03 >> > > > +33 6 83 42 05 79 >> > > > >> > > >> > >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From edoardo.alinovi at gmail.com Thu Sep 19 07:53:06 2024 From: edoardo.alinovi at gmail.com (Edoardo alinovi) Date: Thu, 19 Sep 2024 14:53:06 +0200 Subject: [petsc-users] Question on PCGALERKIN from command line Message-ID: Hi Barry, I have a quick question on PC Galerking as I am trying to do the same thing reported in this post: https://urldefense.us/v3/__https://lists.mcs.anl.gov/pipermail/petsc-users/2017-June/032934.html__;!!G_uCfscf7eWS!bQ0VYCU_TuGCB-803OHYrdOURUwlOpEAp1j5pu1v1Kh6eUgPPdRF_wkiLfrW342u8LNZ4mpCiqz4OxsAOywju5ZUi9bIZUs$ This my command line: mpirun -np 4 flubio_coupled* -UPeqn_pc_type composite -UPeqn_pc_composite_pcs galerkin,bjacobi -UPeqn_sub_0_galerkin_ksp_type preonly -UPeqn_sub_0_galerkin_pc_type none* And this is the error: [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Object is in wrong state [0]PETSC ERROR: Must set operator of PCGALERKIN KSP with PCGalerkinGetKSP()/KSPSetOperators() Do I need to set something for PCGALERKING in my code to make it work from command line? Thank you! -------------- next part -------------- An HTML attachment was scrubbed... URL: From karthikeyan.chockalingam at stfc.ac.uk Thu Sep 19 08:32:06 2024 From: karthikeyan.chockalingam at stfc.ac.uk (Karthikeyan Chockalingam - STFC UKRI) Date: Thu, 19 Sep 2024 13:32:06 +0000 Subject: [petsc-users] Error: PCHYPRESetPoissonMatrix_HYPRE Message-ID: Hello, I would like to make the following hypre call HYPRE_AMSSetBetaPoissonMatrix(ams, NULL); So it does look like ams_beta_is_zero has to be true https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/hypre/hypre.c?ref_type=heads*L325__;Iw!!G_uCfscf7eWS!bi4rdSrrVzBRHrD32vObvn-LFoZba_XXh9z72ayyPQYw-X-nyeVCVp60RVCQVwTZHjxW-LZch_l-DhhxHn9BXpWjJIqS1m2SRZrJ$ So would I end up calling PCHYPRESetPoissonMatrix_HYPRE with isalpha being false. https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/hypre/hypre.c?ref_type=heads*L1640__;Iw!!G_uCfscf7eWS!bi4rdSrrVzBRHrD32vObvn-LFoZba_XXh9z72ayyPQYw-X-nyeVCVp60RVCQVwTZHjxW-LZch_l-DhhxHn9BXpWjJIqS1lTIq-Ra$ But I get the following error from Libmesh: is it because of version incompatibility between PETSc and Hypre? vector_fe_ex3.C:141:3: error: 'PCHYPRESetPoissonMatrix_HYPRE' was not declared in this scope; did you mean 'PCHYPRESetBetaPoissonMatrix'? 141 | PCHYPRESetPoissonMatrix_HYPRE(pc, A, false); | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~ I look forward to your response. Kind regards, Karthik -- Karthik Chockalingam, Ph.D. Senior Research Software Engineer High Performance Systems Engineering Group Hartree Centre | Science and Technology Facilities Council karthikeyan.chockalingam at stfc.ac.uk [signature_3970890138] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 18270 bytes Desc: image001.png URL: From knepley at gmail.com Thu Sep 19 09:20:39 2024 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 19 Sep 2024 10:20:39 -0400 Subject: [petsc-users] Error: PCHYPRESetPoissonMatrix_HYPRE In-Reply-To: References: Message-ID: On Thu, Sep 19, 2024 at 9:33?AM Karthikeyan Chockalingam - STFC UKRI via petsc-users wrote: > Hello, > > > > I would like to make the following hypre call > HYPRE_AMSSetBetaPoissonMatrix(ams, NULL); > > > > So it does look like ams_beta_is_zero has to be true > > > https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/hypre/hypre.c?ref_type=heads*L325__;Iw!!G_uCfscf7eWS!eK8fCGlsadPLSpAOiMsKgfe5Rvn5AFmIy6c98fmu8MdOAd4PzvTw4sthhEHfPWCQCq2wOpwArn1DAETA90wv$ > > > > > So would I end up calling PCHYPRESetPoissonMatrix_HYPRE with isalpha being > false. > > > https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/hypre/hypre.c?ref_type=heads*L1640__;Iw!!G_uCfscf7eWS!eK8fCGlsadPLSpAOiMsKgfe5Rvn5AFmIy6c98fmu8MdOAd4PzvTw4sthhEHfPWCQCq2wOpwArn1DAAWh3H1U$ > > > > > But I get the following error from Libmesh: is it because of version > incompatibility between PETSc and Hypre? > > > > vector_fe_ex3.C:141:3: error: 'PCHYPRESetPoissonMatrix_HYPRE' was not > declared in this scope; did you mean 'PCHYPRESetBetaPoissonMatrix'? > > 141 | PCHYPRESetPoissonMatrix_HYPRE(pc, A, false); > > | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~ > You are misunderstanding the API organization. You are intended to call PCHYPRESetBetaPoissonMatrix(), which calls PCHYPRESetPoissonMatrix_HYPRE(..., PETSC_FALSE); Thanks, Matt > I look forward to your response. > > > > Kind regards, > > Karthik > > > > -- > > *Karthik Chockalingam, Ph.D.* > > Senior Research Software Engineer > > High Performance Systems Engineering Group > > Hartree Centre | Science and Technology Facilities Council > > karthikeyan.chockalingam at stfc.ac.uk > > > > [image: signature_3970890138] > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!eK8fCGlsadPLSpAOiMsKgfe5Rvn5AFmIy6c98fmu8MdOAd4PzvTw4sthhEHfPWCQCq2wOpwArn1DAB18Lnc5$ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 18270 bytes Desc: not available URL: From bsmith at petsc.dev Thu Sep 19 09:25:47 2024 From: bsmith at petsc.dev (Barry Smith) Date: Thu, 19 Sep 2024 10:25:47 -0400 Subject: [petsc-users] Question on PCGALERKIN from command line In-Reply-To: References: Message-ID: Galerkin cannot be used completely from the command line since the Galerkin subspace needs to be defined. This is done with `PCGalerkinSetRestriction`(pc,R) and/or `PCGalerkinSetInterpolation`(pc,P). So you must construct and provide an appropriate R and or P for your problem. Sorry the error message is not very good and is confusing. Barry > On Sep 19, 2024, at 8:53?AM, Edoardo alinovi wrote: > > Hi Barry, > > I have a quick question on PC Galerking as I am trying to do the same thing reported in this post: https://urldefense.us/v3/__https://lists.mcs.anl.gov/pipermail/petsc-users/2017-June/032934.html__;!!G_uCfscf7eWS!eDk5piTM0B8YRN07CUs3CsN1qDacIINuAycHQ4UmA5iM2PrAKth8OlTpf-gz8BGsRMVavnxc0O-22jWV2PBPvvc$ > > This my command line: > mpirun -np 4 flubio_coupled -UPeqn_pc_type composite -UPeqn_pc_composite_pcs galerkin,bjacobi -UPeqn_sub_0_galerkin_ksp_type preonly -UPeqn_sub_0_galerkin_pc_type none > > And this is the error: > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Object is in wrong state > [0]PETSC ERROR: Must set operator of PCGALERKIN KSP with PCGalerkinGetKSP()/KSPSetOperators() > > Do I need to set something for PCGALERKING in my code to make it work from command line? > > Thank you! > -------------- next part -------------- An HTML attachment was scrubbed... URL: From liufield at gmail.com Fri Sep 20 08:34:58 2024 From: liufield at gmail.com (neil liu) Date: Fri, 20 Sep 2024 09:34:58 -0400 Subject: [petsc-users] Inquiry about reading exodus file for Petsc. Message-ID: Dear Petsc developers and users, I am trying to import an exodus II file from coreform Cubit. It is a little -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: test.e Type: application/octet-stream Size: 6588 bytes Desc: not available URL: From liufield at gmail.com Fri Sep 20 08:44:24 2024 From: liufield at gmail.com (neil liu) Date: Fri, 20 Sep 2024 09:44:24 -0400 Subject: [petsc-users] Inquiry about reading an exodus II file from coreform Cubit Message-ID: Dear Petsc developers and users, I am trying to read an exodus II file from coreform cubit, but without success, Then I used petsc's built-in exodus file, /share/petsc/datafiles/meshes/sevenside.exo. This file can be read by petsc successfully. And I did a test. This file, sevenside.exo, is imported into coreform Cubit, and then is saved as a new exodus file. This new exodus file can not be read by petsc successfully. [0]PETSC ERROR: Invalid argument [0]PETSC ERROR: Unrecognized element type SHELL [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!fcN-KL0rs3WRHNRfT47zioS4djjyXM7OCr32LT2olmZHm3X1NIk-WMdU9-UR3rNotQSpINEm8NrcMtWTCFqKEg$ for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.21.1, Apr 26, 2024 [0]PETSC ERROR: ./app on a arch-linux-c-opt named localhost.localdomain by xiaodongliu Fri Sep 20 09:36:16 2024 [0]PETSC ERROR: Configure options -download-mumps -download-scalapack --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --download-fblaslapack --download-mpich --with-scalar-type=complex --download-triangle --with-debugging=no --download-parmetis --download-metis -download-ptscotch --download-bison --download-hdf5 -download-mmg=/home/xiaodongliu/Documents/3rdPartyLib/d5c43d1bcefe598d51428f6a7fee10ec29478b79.tar.gz --download-ctetgen --download-pragmatic --download-eigen --download-netcdf=/home/xiaodongliu/Documents/3rdPartyLib/netcdf-c-4.9.2-p1.tar.gz --download-zlib --download-pnetcdf --download-exodusii [0]PETSC ERROR: #1 ExodusGetCellType_Internal() at /home/xiaodongliu/Documents/petsc-with-docs-3.21.1/petsc-3.21.1/src/dm/impls/plex/plexexodusii.c:1470 [0]PETSC ERROR: #2 DMPlexCreateExodus() at /home/xiaodongliu/Documents/petsc-with-docs-3.21.1/petsc-3.21.1/src/dm/impls/plex/plexexodusii.c:1551 [0]PETSC ERROR: #3 DMPlexCreateExodusFromFile() at /home/xiaodongliu/Documents/petsc-with-docs-3.21.1/petsc-3.21.1/src/dm/impls/plex/plexexodusii.c:1390 Thanks a lot. Xiaodong Compose: Inquiry about reading an exodus II file from coreform Cubit [image: Minimize][image: Pop-out][image: Close] PETSc users list -------------- next part -------------- An HTML attachment was scrubbed... URL: From qiyuelu1 at gmail.com Fri Sep 20 11:42:02 2024 From: qiyuelu1 at gmail.com (Qiyue Lu) Date: Fri, 20 Sep 2024 11:42:02 -0500 Subject: [petsc-users] Compiling 3.21.5 version on a Grace Hopper node failed Message-ID: Hello, I am trying to build the PETSc version 3.21.5 (the latest, cloned from git) version on a Grace Hopper node with this configuration options: *--with-cc=cc --with-cxx=CC --with-fc=ftn --download-f2cblaslapack=1 --with-cudac=nvcc --with-cuda=1 --with-threadsafety=1* with module cudatoolkit/23.11_12.3. And I got this error: =============== /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/../vecseqcupm_impl.hpp: In instantiation of ?Petsc::vec::cupm::impl::VecSeq_CUPM< >::SetAsync(Vec, PetscScalar, PetscDeviceContext):: [with Petsc::device::cupm::DeviceType T = (Petsc::device::cupm::DeviceType)0]?: /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/../vecseqcupm_impl.hpp:1313:157: required from ?struct Petsc::vec::cupm::impl::VecSeq_CUPM< >::SetAsync(Vec, PetscScalar, PetscDeviceContext) [with Petsc::device::cupm::DeviceType T = (Petsc::device::cupm::DeviceType)0; PetscErrorCode = int; Vec = _p_Vec*; PetscScalar = double; PetscDeviceContext = _p_PetscDeviceContext*]::? /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/../vecseqcupm_impl.hpp:1313:167: required from ?static PetscErrorCode Petsc::vec::cupm::impl::VecSeq_CUPM< >::SetAsync(Vec, PetscScalar, PetscDeviceContext) [with Petsc::device::cupm::DeviceType T = (Petsc::device::cupm::DeviceType)0; PetscErrorCode = int; Vec = _p_Vec*; PetscScalar = double; PetscDeviceContext = _p_PetscDeviceContext*]? /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/vecseqcupm.cu:7:40: required from here /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/../vecseqcupm_impl.hpp:1313:150: internal compiler error: in maybe_undo_parenthesized_ref, at cp/semantics.c:1740 PetscCallThrust(THRUST_CALL(thrust::fill, stream, dptr, dptr + n, alpha)); =============== Could you please show some hints on this issue? Thanks, Qiyue Lu -------------- next part -------------- An HTML attachment was scrubbed... URL: From junchao.zhang at gmail.com Fri Sep 20 11:58:46 2024 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Fri, 20 Sep 2024 11:58:46 -0500 Subject: [petsc-users] Compiling 3.21.5 version on a Grace Hopper node failed In-Reply-To: References: Message-ID: /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/../vecseqcupm_impl.hpp:1313:150: *internal compiler error*: in maybe_undo_parenthesized_ref, at cp/semantics.c:1740 Did you use nvidia compilers, e.g. nvc, nvc++? If yes, you may want to change to other compilers by loading other cray PrgEnvs --Junchao Zhang On Fri, Sep 20, 2024 at 11:42?AM Qiyue Lu wrote: > Hello, > I am trying to build the PETSc version 3.21.5 (the latest, cloned from > git) version on a Grace Hopper node with this configuration options: > *--with-cc=cc --with-cxx=CC --with-fc=ftn --download-f2cblaslapack=1 > --with-cudac=nvcc --with-cuda=1 --with-threadsafety=1* > with module cudatoolkit/23.11_12.3. > > And I got this error: > =============== > /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/../vecseqcupm_impl.hpp: > In instantiation of ?Petsc::vec::cupm::impl::VecSeq_CUPM< > >::SetAsync(Vec, PetscScalar, PetscDeviceContext):: [with > Petsc::device::cupm::DeviceType T = (Petsc::device::cupm::DeviceType)0]?: > /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/../vecseqcupm_impl.hpp:1313:157: > required from ?struct Petsc::vec::cupm::impl::VecSeq_CUPM< > >::SetAsync(Vec, PetscScalar, PetscDeviceContext) [with > Petsc::device::cupm::DeviceType T = (Petsc::device::cupm::DeviceType)0; > PetscErrorCode = int; Vec = _p_Vec*; PetscScalar = double; > PetscDeviceContext = _p_PetscDeviceContext*]::? > /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/../vecseqcupm_impl.hpp:1313:167: > required from ?static PetscErrorCode > Petsc::vec::cupm::impl::VecSeq_CUPM< >::SetAsync(Vec, > PetscScalar, PetscDeviceContext) [with Petsc::device::cupm::DeviceType T = > (Petsc::device::cupm::DeviceType)0; PetscErrorCode = int; Vec = _p_Vec*; > PetscScalar = double; PetscDeviceContext = _p_PetscDeviceContext*]? > /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/vecseqcupm.cu:7:40: > required from here > /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/../vecseqcupm_impl.hpp:1313:150: > internal compiler error: in maybe_undo_parenthesized_ref, at > cp/semantics.c:1740 > PetscCallThrust(THRUST_CALL(thrust::fill, stream, dptr, dptr + n, > alpha)); > > =============== > > Could you please show some hints on this issue? > > Thanks, > Qiyue Lu > -------------- next part -------------- An HTML attachment was scrubbed... URL: From qiyuelu1 at gmail.com Fri Sep 20 13:33:13 2024 From: qiyuelu1 at gmail.com (Qiyue Lu) Date: Fri, 20 Sep 2024 13:33:13 -0500 Subject: [petsc-users] Compiling 3.21.5 version on a Grace Hopper node failed In-Reply-To: References: Message-ID: Thanks for your response. Here is my module loaded: *Currently Loaded Modules: 1) craype-arm-grace 4) perftools-base/24.03.0 7) craype/2.7.31 10) cray-libsci/24.03.0 13) default 2) libfabric/1.15.2.0 5) xpmem/2.8.4-1.0_7.1__ga37cbd9.shasta 8) cray-dsmml/0.3.0 11) PrgEnv-gnu/8.5.0 3) craype-network-ofi 6) gcc-native/13.2 9) cray-mpich/8.1.29 12) cudatoolkit/23.11_12.3* And here is the configuration option now: *--with-cc=cc --with-cxx=CC --with-fc=ftn --download-f2cblaslapack=1 --with-cuda=1 * That error is still there. Best, Qiyue Lu On Fri, Sep 20, 2024 at 11:58?AM Junchao Zhang wrote: > /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/../vecseqcupm_impl.hpp:1313:150: > *internal compiler error*: in maybe_undo_parenthesized_ref, at > cp/semantics.c:1740 > > Did you use nvidia compilers, e.g. nvc, nvc++? If yes, you may want to > change to other compilers by loading other cray PrgEnvs > > --Junchao Zhang > > > On Fri, Sep 20, 2024 at 11:42?AM Qiyue Lu wrote: > >> Hello, >> I am trying to build the PETSc version 3.21.5 (the latest, cloned from >> git) version on a Grace Hopper node with this configuration options: >> *--with-cc=cc --with-cxx=CC --with-fc=ftn --download-f2cblaslapack=1 >> --with-cudac=nvcc --with-cuda=1 --with-threadsafety=1* >> with module cudatoolkit/23.11_12.3. >> >> And I got this error: >> =============== >> /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/../vecseqcupm_impl.hpp: >> In instantiation of ?Petsc::vec::cupm::impl::VecSeq_CUPM< >> >::SetAsync(Vec, PetscScalar, PetscDeviceContext):: [with >> Petsc::device::cupm::DeviceType T = (Petsc::device::cupm::DeviceType)0]?: >> /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/../vecseqcupm_impl.hpp:1313:157: >> required from ?struct Petsc::vec::cupm::impl::VecSeq_CUPM< >> >::SetAsync(Vec, PetscScalar, PetscDeviceContext) [with >> Petsc::device::cupm::DeviceType T = (Petsc::device::cupm::DeviceType)0; >> PetscErrorCode = int; Vec = _p_Vec*; PetscScalar = double; >> PetscDeviceContext = _p_PetscDeviceContext*]::? >> /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/../vecseqcupm_impl.hpp:1313:167: >> required from ?static PetscErrorCode >> Petsc::vec::cupm::impl::VecSeq_CUPM< >::SetAsync(Vec, >> PetscScalar, PetscDeviceContext) [with Petsc::device::cupm::DeviceType T = >> (Petsc::device::cupm::DeviceType)0; PetscErrorCode = int; Vec = _p_Vec*; >> PetscScalar = double; PetscDeviceContext = _p_PetscDeviceContext*]? >> /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/vecseqcupm.cu:7:40: >> required from here >> /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/../vecseqcupm_impl.hpp:1313:150: >> internal compiler error: in maybe_undo_parenthesized_ref, at >> cp/semantics.c:1740 >> PetscCallThrust(THRUST_CALL(thrust::fill, stream, dptr, dptr + n, >> alpha)); >> >> =============== >> >> Could you please show some hints on this issue? >> >> Thanks, >> Qiyue Lu >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From junchao.zhang at gmail.com Fri Sep 20 14:07:21 2024 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Fri, 20 Sep 2024 14:07:21 -0500 Subject: [petsc-users] Compiling 3.21.5 version on a Grace Hopper node failed In-Reply-To: References: Message-ID: Could you send the configure.log file? --Junchao Zhang On Fri, Sep 20, 2024 at 1:33?PM Qiyue Lu wrote: > Thanks for your response. > Here is my module loaded: > > > > *Currently Loaded Modules: 1) craype-arm-grace 4) > perftools-base/24.03.0 7) craype/2.7.31 10) > cray-libsci/24.03.0 13) default 2) libfabric/1.15.2.0 > 5) xpmem/2.8.4-1.0_7.1__ga37cbd9.shasta 8) > cray-dsmml/0.3.0 11) PrgEnv-gnu/8.5.0 3) craype-network-ofi 6) > gcc-native/13.2 9) cray-mpich/8.1.29 12) > cudatoolkit/23.11_12.3* > And here is the configuration option now: > > *--with-cc=cc --with-cxx=CC --with-fc=ftn --download-f2cblaslapack=1 > --with-cuda=1 * > > That error is still there. > > Best, > Qiyue Lu > > On Fri, Sep 20, 2024 at 11:58?AM Junchao Zhang > wrote: > >> /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/../vecseqcupm_impl.hpp:1313:150: >> *internal compiler error*: in maybe_undo_parenthesized_ref, at >> cp/semantics.c:1740 >> >> Did you use nvidia compilers, e.g. nvc, nvc++? If yes, you may want to >> change to other compilers by loading other cray PrgEnvs >> >> --Junchao Zhang >> >> >> On Fri, Sep 20, 2024 at 11:42?AM Qiyue Lu wrote: >> >>> Hello, >>> I am trying to build the PETSc version 3.21.5 (the latest, cloned from >>> git) version on a Grace Hopper node with this configuration options: >>> *--with-cc=cc --with-cxx=CC --with-fc=ftn --download-f2cblaslapack=1 >>> --with-cudac=nvcc --with-cuda=1 --with-threadsafety=1* >>> with module cudatoolkit/23.11_12.3. >>> >>> And I got this error: >>> =============== >>> /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/../vecseqcupm_impl.hpp: >>> In instantiation of ?Petsc::vec::cupm::impl::VecSeq_CUPM< >>> >::SetAsync(Vec, PetscScalar, PetscDeviceContext):: [with >>> Petsc::device::cupm::DeviceType T = (Petsc::device::cupm::DeviceType)0]?: >>> /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/../vecseqcupm_impl.hpp:1313:157: >>> required from ?struct Petsc::vec::cupm::impl::VecSeq_CUPM< >>> >::SetAsync(Vec, PetscScalar, PetscDeviceContext) [with >>> Petsc::device::cupm::DeviceType T = (Petsc::device::cupm::DeviceType)0; >>> PetscErrorCode = int; Vec = _p_Vec*; PetscScalar = double; >>> PetscDeviceContext = _p_PetscDeviceContext*]::? >>> /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/../vecseqcupm_impl.hpp:1313:167: >>> required from ?static PetscErrorCode >>> Petsc::vec::cupm::impl::VecSeq_CUPM< >::SetAsync(Vec, >>> PetscScalar, PetscDeviceContext) [with Petsc::device::cupm::DeviceType T = >>> (Petsc::device::cupm::DeviceType)0; PetscErrorCode = int; Vec = _p_Vec*; >>> PetscScalar = double; PetscDeviceContext = _p_PetscDeviceContext*]? >>> /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/vecseqcupm.cu:7:40: >>> required from here >>> /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/../vecseqcupm_impl.hpp:1313:150: >>> internal compiler error: in maybe_undo_parenthesized_ref, at >>> cp/semantics.c:1740 >>> PetscCallThrust(THRUST_CALL(thrust::fill, stream, dptr, dptr + n, >>> alpha)); >>> >>> =============== >>> >>> Could you please show some hints on this issue? >>> >>> Thanks, >>> Qiyue Lu >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrsd at gmail.com Fri Sep 20 14:14:49 2024 From: andrsd at gmail.com (David Andrs) Date: Fri, 20 Sep 2024 13:14:49 -0600 Subject: [petsc-users] Inquiry about reading an exodus II file from coreform Cubit In-Reply-To: References: Message-ID: Before you export the mesh from Cubit, change the element type to something like QUAD4. PETSc does not automatically remap SHELL elements to QUADs. -- David On Fri, Sep 20, 2024 at 8:05?AM neil liu wrote: > Dear Petsc developers and users, > > I am trying to read an exodus II file from coreform cubit, but without > success, > Then I used petsc's built-in exodus file, > /share/petsc/datafiles/meshes/sevenside.exo. > This file can be read by petsc successfully. > > And I did a test. This file, sevenside.exo, is imported into coreform > Cubit, and then is saved as a new exodus file. This new exodus file can not > be read by petsc successfully. > > [0]PETSC ERROR: Invalid argument > [0]PETSC ERROR: Unrecognized element type SHELL > [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!YXFbgIYxxvzCFais_YdSXz_Ud3HzkOxfL59IAAUpV7ib_eKf36dajg1egysD6JLSeii4EZ_NXsx3DrXzCiY$ > for > trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.21.1, Apr 26, 2024 > [0]PETSC ERROR: ./app on a arch-linux-c-opt named localhost.localdomain by > xiaodongliu Fri Sep 20 09:36:16 2024 > [0]PETSC ERROR: Configure options -download-mumps -download-scalapack > --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --download-fblaslapack > --download-mpich --with-scalar-type=complex --download-triangle > --with-debugging=no --download-parmetis --download-metis -download-ptscotch > --download-bison --download-hdf5 > -download-mmg=/home/xiaodongliu/Documents/3rdPartyLib/d5c43d1bcefe598d51428f6a7fee10ec29478b79.tar.gz > --download-ctetgen --download-pragmatic --download-eigen > --download-netcdf=/home/xiaodongliu/Documents/3rdPartyLib/netcdf-c-4.9.2-p1.tar.gz > --download-zlib --download-pnetcdf --download-exodusii > [0]PETSC ERROR: #1 ExodusGetCellType_Internal() at > /home/xiaodongliu/Documents/petsc-with-docs-3.21.1/petsc-3.21.1/src/dm/impls/plex/plexexodusii.c:1470 > [0]PETSC ERROR: #2 DMPlexCreateExodus() at > /home/xiaodongliu/Documents/petsc-with-docs-3.21.1/petsc-3.21.1/src/dm/impls/plex/plexexodusii.c:1551 > [0]PETSC ERROR: #3 DMPlexCreateExodusFromFile() at > /home/xiaodongliu/Documents/petsc-with-docs-3.21.1/petsc-3.21.1/src/dm/impls/plex/plexexodusii.c:1390 > > Thanks a lot. > > Xiaodong > Compose: > Inquiry about reading an exodus II file from coreform Cubit > [image: Minimize][image: Pop-out][image: Close] > PETSc users list > -------------- next part -------------- An HTML attachment was scrubbed... URL: From qiyuelu1 at gmail.com Fri Sep 20 14:22:14 2024 From: qiyuelu1 at gmail.com (Qiyue Lu) Date: Fri, 20 Sep 2024 14:22:14 -0500 Subject: [petsc-users] Compiling 3.21.5 version on a Grace Hopper node failed In-Reply-To: References: Message-ID: Sure, here it is. Thanks. On Fri, Sep 20, 2024 at 2:07?PM Junchao Zhang wrote: > Could you send the configure.log file? > --Junchao Zhang > > > On Fri, Sep 20, 2024 at 1:33?PM Qiyue Lu wrote: > >> Thanks for your response. >> Here is my module loaded: >> >> >> >> *Currently Loaded Modules: 1) craype-arm-grace 4) >> perftools-base/24.03.0 7) craype/2.7.31 10) >> cray-libsci/24.03.0 13) default 2) libfabric/1.15.2.0 >> 5) xpmem/2.8.4-1.0_7.1__ga37cbd9.shasta 8) >> cray-dsmml/0.3.0 11) PrgEnv-gnu/8.5.0 3) craype-network-ofi 6) >> gcc-native/13.2 9) cray-mpich/8.1.29 12) >> cudatoolkit/23.11_12.3* >> And here is the configuration option now: >> >> *--with-cc=cc --with-cxx=CC --with-fc=ftn --download-f2cblaslapack=1 >> --with-cuda=1 * >> >> That error is still there. >> >> Best, >> Qiyue Lu >> >> On Fri, Sep 20, 2024 at 11:58?AM Junchao Zhang >> wrote: >> >>> /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/../vecseqcupm_impl.hpp:1313:150: >>> *internal compiler error*: in maybe_undo_parenthesized_ref, at >>> cp/semantics.c:1740 >>> >>> Did you use nvidia compilers, e.g. nvc, nvc++? If yes, you may want to >>> change to other compilers by loading other cray PrgEnvs >>> >>> --Junchao Zhang >>> >>> >>> On Fri, Sep 20, 2024 at 11:42?AM Qiyue Lu wrote: >>> >>>> Hello, >>>> I am trying to build the PETSc version 3.21.5 (the latest, cloned from >>>> git) version on a Grace Hopper node with this configuration options: >>>> *--with-cc=cc --with-cxx=CC --with-fc=ftn --download-f2cblaslapack=1 >>>> --with-cudac=nvcc --with-cuda=1 --with-threadsafety=1* >>>> with module cudatoolkit/23.11_12.3. >>>> >>>> And I got this error: >>>> =============== >>>> /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/../vecseqcupm_impl.hpp: >>>> In instantiation of ?Petsc::vec::cupm::impl::VecSeq_CUPM< >>>> >::SetAsync(Vec, PetscScalar, PetscDeviceContext):: [with >>>> Petsc::device::cupm::DeviceType T = (Petsc::device::cupm::DeviceType)0]?: >>>> /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/../vecseqcupm_impl.hpp:1313:157: >>>> required from ?struct Petsc::vec::cupm::impl::VecSeq_CUPM< >>>> >::SetAsync(Vec, PetscScalar, PetscDeviceContext) [with >>>> Petsc::device::cupm::DeviceType T = (Petsc::device::cupm::DeviceType)0; >>>> PetscErrorCode = int; Vec = _p_Vec*; PetscScalar = double; >>>> PetscDeviceContext = _p_PetscDeviceContext*]::? >>>> /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/../vecseqcupm_impl.hpp:1313:167: >>>> required from ?static PetscErrorCode >>>> Petsc::vec::cupm::impl::VecSeq_CUPM< >::SetAsync(Vec, >>>> PetscScalar, PetscDeviceContext) [with Petsc::device::cupm::DeviceType T = >>>> (Petsc::device::cupm::DeviceType)0; PetscErrorCode = int; Vec = _p_Vec*; >>>> PetscScalar = double; PetscDeviceContext = _p_PetscDeviceContext*]? >>>> /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/vecseqcupm.cu:7:40: >>>> required from here >>>> /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/../vecseqcupm_impl.hpp:1313:150: >>>> internal compiler error: in maybe_undo_parenthesized_ref, at >>>> cp/semantics.c:1740 >>>> PetscCallThrust(THRUST_CALL(thrust::fill, stream, dptr, dptr + >>>> n, alpha)); >>>> >>>> =============== >>>> >>>> Could you please show some hints on this issue? >>>> >>>> Thanks, >>>> Qiyue Lu >>>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: text/x-log Size: 1396537 bytes Desc: not available URL: From liufield at gmail.com Sat Sep 21 08:54:32 2024 From: liufield at gmail.com (neil liu) Date: Sat, 21 Sep 2024 09:54:32 -0400 Subject: [petsc-users] Inquiry about reading an exodus II file from coreform Cubit In-Reply-To: References: Message-ID: Thanks a lot, David. That works. Then I tried another example from Cubit. The script is defined as follows, reset brick x 1 mesh vol 1 block 1 volume 1 sideset 1 surface 1 block 2 surface 1 #sides of these elements can now be referenced in a sideset block 2 element type quad sideset 2 curve 1 Then the mesh was exported as a exodus file (attached), and imported into Petsc. Here, the code was stuck by the function, DMPlexCreateExodus(). I think maybe this is due to block 1 is 3D while block 2 is 2D. But this seems necessary to define an edge using sideset. Thanks, Xiaodong On Fri, Sep 20, 2024 at 3:15?PM David Andrs wrote: > Before you export the mesh from Cubit, change the element type to > something like QUAD4. PETSc does not automatically remap SHELL elements to > QUADs. > > -- > David > > On Fri, Sep 20, 2024 at 8:05?AM neil liu wrote: > >> Dear Petsc developers and users, >> >> I am trying to read an exodus II file from coreform cubit, but without >> success, >> Then I used petsc's built-in exodus file, >> /share/petsc/datafiles/meshes/sevenside.exo. >> This file can be read by petsc successfully. >> >> And I did a test. This file, sevenside.exo, is imported into coreform >> Cubit, and then is saved as a new exodus file. This new exodus file can not >> be read by petsc successfully. >> >> [0]PETSC ERROR: Invalid argument >> [0]PETSC ERROR: Unrecognized element type SHELL >> [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!ZpvOEyVekHc8aSibQ_b2zv7q7cr-DxnkTfOTSvDcUuydfpfd6rwcbvH1VRCh3MREIeAyEQ64ZxReP77OQsQN9A$ >> for >> trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.21.1, Apr 26, 2024 >> [0]PETSC ERROR: ./app on a arch-linux-c-opt named localhost.localdomain >> by xiaodongliu Fri Sep 20 09:36:16 2024 >> [0]PETSC ERROR: Configure options -download-mumps -download-scalapack >> --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --download-fblaslapack >> --download-mpich --with-scalar-type=complex --download-triangle >> --with-debugging=no --download-parmetis --download-metis -download-ptscotch >> --download-bison --download-hdf5 >> -download-mmg=/home/xiaodongliu/Documents/3rdPartyLib/d5c43d1bcefe598d51428f6a7fee10ec29478b79.tar.gz >> --download-ctetgen --download-pragmatic --download-eigen >> --download-netcdf=/home/xiaodongliu/Documents/3rdPartyLib/netcdf-c-4.9.2-p1.tar.gz >> --download-zlib --download-pnetcdf --download-exodusii >> [0]PETSC ERROR: #1 ExodusGetCellType_Internal() at >> /home/xiaodongliu/Documents/petsc-with-docs-3.21.1/petsc-3.21.1/src/dm/impls/plex/plexexodusii.c:1470 >> [0]PETSC ERROR: #2 DMPlexCreateExodus() at >> /home/xiaodongliu/Documents/petsc-with-docs-3.21.1/petsc-3.21.1/src/dm/impls/plex/plexexodusii.c:1551 >> [0]PETSC ERROR: #3 DMPlexCreateExodusFromFile() at >> /home/xiaodongliu/Documents/petsc-with-docs-3.21.1/petsc-3.21.1/src/dm/impls/plex/plexexodusii.c:1390 >> >> Thanks a lot. >> >> Xiaodong >> Compose: >> Inquiry about reading an exodus II file from coreform Cubit >> [image: Minimize][image: Pop-out][image: Close] >> PETSc users list >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: test.e Type: application/octet-stream Size: 87816 bytes Desc: not available URL: From bourdin at mcmaster.ca Sun Sep 22 08:26:17 2024 From: bourdin at mcmaster.ca (Blaise Bourdin) Date: Sun, 22 Sep 2024 13:26:17 +0000 Subject: [petsc-users] Inquiry about reading an exodus II file from coreform Cubit In-Reply-To: References: Message-ID: <0B8C6F87-E3F7-408A-A792-9F426394B77D@mcmaster.ca> An HTML attachment was scrubbed... URL: From liufield at gmail.com Sun Sep 22 09:03:06 2024 From: liufield at gmail.com (neil liu) Date: Sun, 22 Sep 2024 22:03:06 +0800 Subject: [petsc-users] Inquiry about reading an exodus II file from coreform Cubit In-Reply-To: <0B8C6F87-E3F7-408A-A792-9F426394B77D@mcmaster.ca> References: <0B8C6F87-E3F7-408A-A792-9F426394B77D@mcmaster.ca> Message-ID: Thanks a lot, Blaise. Yes. As you said, I have to define a node set to find the edges we are interested in for Exodus II file. On Sun, Sep 22, 2024 at 9:26?AM Blaise Bourdin wrote: > > > On Sep 21, 2024, at 9:54?AM, neil liu wrote: > > Caution: External email. > > Thanks a lot, David. That works. > Then I tried another example from Cubit. The script is defined as follows, > > reset > brick x 1 > mesh vol 1 > block 1 volume 1 > sideset 1 surface 1 > block 2 surface 1 #sides of these elements can now be referenced in a > sideset > block 2 element type quad > sideset 2 curve 1 > > Then the mesh was exported as a exodus file (attached), and imported into > Petsc. > Here, the code was stuck by the function, DMPlexCreateExodus(). > I think maybe this is due to block 1 is 3D while block 2 is 2D. > But this seems necessary to define an edge using sideset. > > > > That is corrrect. > As far as I understand, one of the few assumptions in dmplex is that > ?cells? have the same topological dimension > Here sideset 1 and 2 have different dimensions. > As far as I remember, exodusII does not have ?Edge sets?, so your best bet > would be to define a node set for curve 1. > > Regards, > Blaise > > > Thanks, > > Xiaodong > > On Fri, Sep 20, 2024 at 3:15?PM David Andrs wrote: > >> Before you export the mesh from Cubit, change the element type to >> something like QUAD4. PETSc does not automatically remap SHELL elements to >> QUADs. >> >> -- >> David >> >> On Fri, Sep 20, 2024 at 8:05?AM neil liu wrote: >> >>> Dear Petsc developers and users, >>> >>> I am trying to read an exodus II file from coreform cubit, but without >>> success, >>> Then I used petsc's built-in exodus file, >>> /share/petsc/datafiles/meshes/sevenside.exo. >>> This file can be read by petsc successfully. >>> >>> And I did a test. This file, sevenside.exo, is imported into coreform >>> Cubit, and then is saved as a new exodus file. This new exodus file can not >>> be read by petsc successfully. >>> >>> [0]PETSC ERROR: Invalid argument >>> [0]PETSC ERROR: Unrecognized element type SHELL >>> [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!behGY_YKaZKejR7vV7vkVTAHTWR_Y1UTNxgJ-4rMSCrvsMTFm_QsFVjhbC6HhjoPQ4ThPerrFQAqNC0N2WoY9A$ >>> for >>> trouble shooting. >>> [0]PETSC ERROR: Petsc Release Version 3.21.1, Apr 26, 2024 >>> [0]PETSC ERROR: ./app on a arch-linux-c-opt named localhost.localdomain >>> by xiaodongliu Fri Sep 20 09:36:16 2024 >>> [0]PETSC ERROR: Configure options -download-mumps -download-scalapack >>> --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --download-fblaslapack >>> --download-mpich --with-scalar-type=complex --download-triangle >>> --with-debugging=no --download-parmetis --download-metis -download-ptscotch >>> --download-bison --download-hdf5 >>> -download-mmg=/home/xiaodongliu/Documents/3rdPartyLib/d5c43d1bcefe598d51428f6a7fee10ec29478b79.tar.gz >>> --download-ctetgen --download-pragmatic --download-eigen >>> --download-netcdf=/home/xiaodongliu/Documents/3rdPartyLib/netcdf-c-4.9.2-p1.tar.gz >>> --download-zlib --download-pnetcdf --download-exodusii >>> [0]PETSC ERROR: #1 ExodusGetCellType_Internal() at >>> /home/xiaodongliu/Documents/petsc-with-docs-3.21.1/petsc-3.21.1/src/dm/impls/plex/plexexodusii.c:1470 >>> [0]PETSC ERROR: #2 DMPlexCreateExodus() at >>> /home/xiaodongliu/Documents/petsc-with-docs-3.21.1/petsc-3.21.1/src/dm/impls/plex/plexexodusii.c:1551 >>> [0]PETSC ERROR: #3 DMPlexCreateExodusFromFile() at >>> /home/xiaodongliu/Documents/petsc-with-docs-3.21.1/petsc-3.21.1/src/dm/impls/plex/plexexodusii.c:1390 >>> >>> Thanks a lot. >>> >>> Xiaodong >>> Compose: >>> Inquiry about reading an exodus II file from coreform Cubit >>> [image: Minimize][image: Pop-out][image: Close] >>> PETSc users list >>> >> > > > ? > Canada Research Chair in Mathematical and Computational Aspects of Solid > Mechanics (Tier 1) > Professor, Department of Mathematics & Statistics > Hamilton Hall room 409A, McMaster University > 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada > https://urldefense.us/v3/__https://www.math.mcmaster.ca/bourdin__;!!G_uCfscf7eWS!behGY_YKaZKejR7vV7vkVTAHTWR_Y1UTNxgJ-4rMSCrvsMTFm_QsFVjhbC6HhjoPQ4ThPerrFQAqNC3T_iAGGQ$ | +1 (905) 525 9140 ext. 27243 > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sun Sep 22 10:13:01 2024 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 22 Sep 2024 11:13:01 -0400 Subject: [petsc-users] Inquiry about reading an exodus II file from coreform Cubit In-Reply-To: <0B8C6F87-E3F7-408A-A792-9F426394B77D@mcmaster.ca> References: <0B8C6F87-E3F7-408A-A792-9F426394B77D@mcmaster.ca> Message-ID: On Sun, Sep 22, 2024 at 9:26?AM Blaise Bourdin wrote: > On Sep 21, 2024, at 9:54?AM, neil liu wrote: > > Caution: External email. > > Thanks a lot, David. That works. > Then I tried another example from Cubit. The script is defined as follows, > > reset > brick x 1 > mesh vol 1 > block 1 volume 1 > sideset 1 surface 1 > block 2 surface 1 #sides of these elements can now be referenced in a > sideset > block 2 element type quad > sideset 2 curve 1 > > Then the mesh was exported as a exodus file (attached), and imported into > Petsc. > Here, the code was stuck by the function, DMPlexCreateExodus(). > I think maybe this is due to block 1 is 3D while block 2 is 2D. > But this seems necessary to define an edge using sideset. > > > > That is corrrect. > As far as I understand, one of the few assumptions in dmplex is that > ?cells? have the same topological dimension > Here sideset 1 and 2 have different dimensions. > As far as I remember, exodusII does not have ?Edge sets?, so your best bet > would be to define a node set for curve 1. > It is, of course, possible to mark edges. The problem is that ExodusII makes it very hard to connect the edges in your sideset to edges in the actual mesh, which is really what you want. This, in my opinion, is why ExodusII is a failure as a mesh format. If we can explain, I you can make this connection, I have no problem writing this code. Thanks, Matt > Regards, > Blaise > > > Thanks, > > Xiaodong > > On Fri, Sep 20, 2024 at 3:15?PM David Andrs wrote: > >> Before you export the mesh from Cubit, change the element type to >> something like QUAD4. PETSc does not automatically remap SHELL elements to >> QUADs. >> >> -- >> David >> >> On Fri, Sep 20, 2024 at 8:05?AM neil liu wrote: >> >>> Dear Petsc developers and users, >>> >>> I am trying to read an exodus II file from coreform cubit, but without >>> success, >>> Then I used petsc's built-in exodus file, >>> /share/petsc/datafiles/meshes/sevenside.exo. >>> This file can be read by petsc successfully. >>> >>> And I did a test. This file, sevenside.exo, is imported into coreform >>> Cubit, and then is saved as a new exodus file. This new exodus file can not >>> be read by petsc successfully. >>> >>> [0]PETSC ERROR: Invalid argument >>> [0]PETSC ERROR: Unrecognized element type SHELL >>> [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!dee9Na-tSTsDKqZOWq2BLt5MPj0L04TPre2lr-cwZEDmoI_Cb8bezH_J6E6K7vJynTpnCxV-jbPjt4XvcUgN$ >>> for >>> trouble shooting. >>> [0]PETSC ERROR: Petsc Release Version 3.21.1, Apr 26, 2024 >>> [0]PETSC ERROR: ./app on a arch-linux-c-opt named localhost.localdomain >>> by xiaodongliu Fri Sep 20 09:36:16 2024 >>> [0]PETSC ERROR: Configure options -download-mumps -download-scalapack >>> --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --download-fblaslapack >>> --download-mpich --with-scalar-type=complex --download-triangle >>> --with-debugging=no --download-parmetis --download-metis -download-ptscotch >>> --download-bison --download-hdf5 >>> -download-mmg=/home/xiaodongliu/Documents/3rdPartyLib/d5c43d1bcefe598d51428f6a7fee10ec29478b79.tar.gz >>> --download-ctetgen --download-pragmatic --download-eigen >>> --download-netcdf=/home/xiaodongliu/Documents/3rdPartyLib/netcdf-c-4.9.2-p1.tar.gz >>> --download-zlib --download-pnetcdf --download-exodusii >>> [0]PETSC ERROR: #1 ExodusGetCellType_Internal() at >>> /home/xiaodongliu/Documents/petsc-with-docs-3.21.1/petsc-3.21.1/src/dm/impls/plex/plexexodusii.c:1470 >>> [0]PETSC ERROR: #2 DMPlexCreateExodus() at >>> /home/xiaodongliu/Documents/petsc-with-docs-3.21.1/petsc-3.21.1/src/dm/impls/plex/plexexodusii.c:1551 >>> [0]PETSC ERROR: #3 DMPlexCreateExodusFromFile() at >>> /home/xiaodongliu/Documents/petsc-with-docs-3.21.1/petsc-3.21.1/src/dm/impls/plex/plexexodusii.c:1390 >>> >>> Thanks a lot. >>> >>> Xiaodong >>> Compose: >>> Inquiry about reading an exodus II file from coreform Cubit >>> [image: Minimize][image: Pop-out][image: Close] >>> PETSc users list >>> >> > > > ? > Canada Research Chair in Mathematical and Computational Aspects of Solid > Mechanics (Tier 1) > Professor, Department of Mathematics & Statistics > Hamilton Hall room 409A, McMaster University > 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada > https://urldefense.us/v3/__https://www.math.mcmaster.ca/bourdin__;!!G_uCfscf7eWS!dee9Na-tSTsDKqZOWq2BLt5MPj0L04TPre2lr-cwZEDmoI_Cb8bezH_J6E6K7vJynTpnCxV-jbPjtwBSB1Pe$ > > | +1 (905) 525 9140 ext. 27243 > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!dee9Na-tSTsDKqZOWq2BLt5MPj0L04TPre2lr-cwZEDmoI_Cb8bezH_J6E6K7vJynTpnCxV-jbPjt7Gu_5vo$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrsd at gmail.com Sun Sep 22 10:51:39 2024 From: andrsd at gmail.com (David Andrs) Date: Sun, 22 Sep 2024 09:51:39 -0600 Subject: [petsc-users] Inquiry about reading an exodus II file from coreform Cubit In-Reply-To: <0B8C6F87-E3F7-408A-A792-9F426394B77D@mcmaster.ca> References: <0B8C6F87-E3F7-408A-A792-9F426394B77D@mcmaster.ca> Message-ID: <563F566D-39BB-4048-BE4E-D94C51F3E32A@gmail.com> > On Sep 22, 2024, at 07:26, Blaise Bourdin wrote: > > > >> On Sep 21, 2024, at 9:54?AM, neil liu > wrote: >> >> Caution: External email. >> >> Thanks a lot, David. That works. >> Then I tried another example from Cubit. The script is defined as follows, >> >> reset >> brick x 1 >> mesh vol 1 >> block 1 volume 1 >> sideset 1 surface 1 >> block 2 surface 1 #sides of these elements can now be referenced in a sideset >> block 2 element type quad >> sideset 2 curve 1 >> >> Then the mesh was exported as a exodus file (attached), and imported into Petsc. >> Here, the code was stuck by the function, DMPlexCreateExodus(). >> I think maybe this is due to block 1 is 3D while block 2 is 2D. >> But this seems necessary to define an edge using sideset. > > > That is corrrect. > As far as I understand, one of the few assumptions in dmplex is that ?cells? have the same topological dimension > Here sideset 1 and 2 have different dimensions. > As far as I remember, exodusII does not have ?Edge sets?, so your best bet would be to define a node set for curve 1. I have not seen the concept of edge sets discussed in the official exodusII doc [1]. However, they have an enum called ex_entity_type which does have a EX_EDGE_SET [2] value :thinking: IDK, how much developed that is. I have not used this myself. Most codes I have seen just use SIDE/NODE sets? Just sharing something I recently spotted? [1] https://urldefense.us/v3/__https://sandialabs.github.io/seacas-docs/exodusII-new.pdf__;!!G_uCfscf7eWS!ZXJZphkaSn_4MhiE6JwD78s0zYdnwitEqgRtuh997d8-EpF-ebRBxfAqlqjCsPAyPs6UAv6MOex2ksU4efI$ [2] https://urldefense.us/v3/__https://sandialabs.github.io/seacas-docs/html/exodusII_8h.html*a1f649ed288d305ab55773ffd1d220490__;Iw!!G_uCfscf7eWS!ZXJZphkaSn_4MhiE6JwD78s0zYdnwitEqgRtuh997d8-EpF-ebRBxfAqlqjCsPAyPs6UAv6MOex2PdimBKk$ ? David > > Regards, > Blaise > >> >> Thanks, >> >> Xiaodong >> >> On Fri, Sep 20, 2024 at 3:15?PM David Andrs > wrote: >>> Before you export the mesh from Cubit, change the element type to something like QUAD4. PETSc does not automatically remap SHELL elements to QUADs. >>> >>> -- >>> David >>> >>> On Fri, Sep 20, 2024 at 8:05?AM neil liu > wrote: >>>> Dear Petsc developers and users, >>>> >>>> I am trying to read an exodus II file from coreform cubit, but without success, >>>> Then I used petsc's built-in exodus file, /share/petsc/datafiles/meshes/sevenside.exo. >>>> This file can be read by petsc successfully. >>>> >>>> And I did a test. This file, sevenside.exo, is imported into coreform Cubit, and then is saved as a new exodus file. This new exodus file can not be read by petsc successfully. >>>> >>>> [0]PETSC ERROR: Invalid argument >>>> [0]PETSC ERROR: Unrecognized element type SHELL >>>> [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!ZXJZphkaSn_4MhiE6JwD78s0zYdnwitEqgRtuh997d8-EpF-ebRBxfAqlqjCsPAyPs6UAv6MOex2BOTP4BI$ for trouble shooting. >>>> [0]PETSC ERROR: Petsc Release Version 3.21.1, Apr 26, 2024 >>>> [0]PETSC ERROR: ./app on a arch-linux-c-opt named localhost.localdomain by xiaodongliu Fri Sep 20 09:36:16 2024 >>>> [0]PETSC ERROR: Configure options -download-mumps -download-scalapack --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --download-fblaslapack --download-mpich --with-scalar-type=complex --download-triangle --with-debugging=no --download-parmetis --download-metis -download-ptscotch --download-bison --download-hdf5 -download-mmg=/home/xiaodongliu/Documents/3rdPartyLib/d5c43d1bcefe598d51428f6a7fee10ec29478b79.tar.gz --download-ctetgen --download-pragmatic --download-eigen --download-netcdf=/home/xiaodongliu/Documents/3rdPartyLib/netcdf-c-4.9.2-p1.tar.gz --download-zlib --download-pnetcdf --download-exodusii >>>> [0]PETSC ERROR: #1 ExodusGetCellType_Internal() at /home/xiaodongliu/Documents/petsc-with-docs-3.21.1/petsc-3.21.1/src/dm/impls/plex/plexexodusii.c:1470 >>>> [0]PETSC ERROR: #2 DMPlexCreateExodus() at /home/xiaodongliu/Documents/petsc-with-docs-3.21.1/petsc-3.21.1/src/dm/impls/plex/plexexodusii.c:1551 >>>> [0]PETSC ERROR: #3 DMPlexCreateExodusFromFile() at /home/xiaodongliu/Documents/petsc-with-docs-3.21.1/petsc-3.21.1/src/dm/impls/plex/plexexodusii.c:1390 >>>> >>>> Thanks a lot. >>>> >>>> Xiaodong >>>> Compose: >>>> Inquiry about reading an exodus II file from coreform Cubit >>>> >>>> >>>> PETSc users list >>>> >> > > ? > Canada Research Chair in Mathematical and Computational Aspects of Solid Mechanics (Tier 1) > Professor, Department of Mathematics & Statistics > Hamilton Hall room 409A, McMaster University > 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada > https://urldefense.us/v3/__https://www.math.mcmaster.ca/bourdin__;!!G_uCfscf7eWS!ZXJZphkaSn_4MhiE6JwD78s0zYdnwitEqgRtuh997d8-EpF-ebRBxfAqlqjCsPAyPs6UAv6MOex2EiOeyPQ$ | +1 (905) 525 9140 ext. 27243 -------------- next part -------------- An HTML attachment was scrubbed... URL: From matteo.semplice at uninsubria.it Mon Sep 23 04:33:37 2024 From: matteo.semplice at uninsubria.it (Matteo Semplice) Date: Mon, 23 Sep 2024 11:33:37 +0200 Subject: [petsc-users] Mat indices for DMPlex jacobian Message-ID: Dear petsc, ??? I need to hand-code a jacobian and I can't figure out how to translate the DMplex points/fields to matrix indices. My DMPlex has a section with m fields per cell (which makes for n>m dof per cell since some fields are vector). Say I want to insert an nxn block for the row corresponding to cell c and coloumn of its neighbour d. I guess that I should call either MatSetValues/MatSetValuesLocal or the blocked variants, but how do I find the row/col indices to pass in starting from the c/d dmplex points? And, while I am at it, which MatSetValues version standard/Local standard/blocked is best? I looked for a petsc example, but fails to find it: if there's one, can you just point me to it? Thanks in advance. Matteo Semplice From knepley at gmail.com Mon Sep 23 10:16:12 2024 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 23 Sep 2024 11:16:12 -0400 Subject: [petsc-users] Mat indices for DMPlex jacobian In-Reply-To: References: Message-ID: On Mon, Sep 23, 2024 at 5:33?AM Matteo Semplice via petsc-users < petsc-users at mcs.anl.gov> wrote: > Dear petsc, > > I need to hand-code a jacobian and I can't figure out how to > translate the DMplex points/fields to matrix indices. > > My DMPlex has a section with m fields per cell (which makes for n>m dof > per cell since some fields are vector). Say I want to insert an nxn > block for the row corresponding to cell c and coloumn of its neighbour > d. I guess that I should call either MatSetValues/MatSetValuesLocal or > the blocked variants, but how do I find the row/col indices to pass in > starting from the c/d dmplex points? And, while I am at it, which > MatSetValues version standard/Local standard/blocked is best? > > I looked for a petsc example, but fails to find it: if there's one, can > you just point me to it? > 1. In FEM, the main unit of indices is usually the closure, so we provide https://urldefense.us/v3/__https://petsc.org/main/manualpages/DMPlex/DMPlexGetClosureIndices/__;!!G_uCfscf7eWS!fDvGoz5OloH7xryMIAIqyYsslt8U4HqegW7HeYDlwRPbkTloMU6k0kze8NlG7S2DxDuY2nUXRQcBIiuqLU1o$ which is what is used inside of https://urldefense.us/v3/__https://petsc.org/main/manualpages/DMPlex/DMPlexMatSetClosure/__;!!G_uCfscf7eWS!fDvGoz5OloH7xryMIAIqyYsslt8U4HqegW7HeYDlwRPbkTloMU6k0kze8NlG7S2DxDuY2nUXRQcBIi82N3yY$ 2. These indices are calculated using the global section, so you can just DMGetGlobalSection(dm, &gs); and then use PetscSectionGetDof() and PetscSectionGetOffset(), knowing that off-process values are encoded as -(off + 1), so you need to convert those. Does this make sense? Thanks, Matt > Thanks in advance. > > Matteo Semplice > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fDvGoz5OloH7xryMIAIqyYsslt8U4HqegW7HeYDlwRPbkTloMU6k0kze8NlG7S2DxDuY2nUXRQcBIjjfDigI$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From junchao.zhang at gmail.com Tue Sep 24 23:00:34 2024 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Tue, 24 Sep 2024 23:00:34 -0500 Subject: [petsc-users] Compiling 3.21.5 version on a Grace Hopper node failed In-Reply-To: References: Message-ID: Hi, Qiyue, I don't have the exact compilers as yours (cuda-12.3 + gcc-13.2.1). I tried petsc-3.21.5 with cuda-12.6 + gcc-13.3.0 and it compiled fine. Since it was an internal compiler error, it seems the only option is to change compilers. Thanks. --Junchao Zhang On Fri, Sep 20, 2024 at 2:22?PM Qiyue Lu wrote: > Sure, here it is. Thanks. > > On Fri, Sep 20, 2024 at 2:07?PM Junchao Zhang > wrote: > >> Could you send the configure.log file? >> --Junchao Zhang >> >> >> On Fri, Sep 20, 2024 at 1:33?PM Qiyue Lu wrote: >> >>> Thanks for your response. >>> Here is my module loaded: >>> >>> >>> >>> *Currently Loaded Modules: 1) craype-arm-grace 4) >>> perftools-base/24.03.0 7) craype/2.7.31 10) >>> cray-libsci/24.03.0 13) default 2) libfabric/1.15.2.0 >>> 5) xpmem/2.8.4-1.0_7.1__ga37cbd9.shasta 8) >>> cray-dsmml/0.3.0 11) PrgEnv-gnu/8.5.0 3) craype-network-ofi 6) >>> gcc-native/13.2 9) cray-mpich/8.1.29 12) >>> cudatoolkit/23.11_12.3* >>> And here is the configuration option now: >>> >>> *--with-cc=cc --with-cxx=CC --with-fc=ftn --download-f2cblaslapack=1 >>> --with-cuda=1 * >>> >>> That error is still there. >>> >>> Best, >>> Qiyue Lu >>> >>> On Fri, Sep 20, 2024 at 11:58?AM Junchao Zhang >>> wrote: >>> >>>> /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/../vecseqcupm_impl.hpp:1313:150: >>>> *internal compiler error*: in maybe_undo_parenthesized_ref, at >>>> cp/semantics.c:1740 >>>> >>>> Did you use nvidia compilers, e.g. nvc, nvc++? If yes, you may want to >>>> change to other compilers by loading other cray PrgEnvs >>>> >>>> --Junchao Zhang >>>> >>>> >>>> On Fri, Sep 20, 2024 at 11:42?AM Qiyue Lu wrote: >>>> >>>>> Hello, >>>>> I am trying to build the PETSc version 3.21.5 (the latest, cloned from >>>>> git) version on a Grace Hopper node with this configuration options: >>>>> *--with-cc=cc --with-cxx=CC --with-fc=ftn --download-f2cblaslapack=1 >>>>> --with-cudac=nvcc --with-cuda=1 --with-threadsafety=1* >>>>> with module cudatoolkit/23.11_12.3. >>>>> >>>>> And I got this error: >>>>> =============== >>>>> /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/../vecseqcupm_impl.hpp: >>>>> In instantiation of ?Petsc::vec::cupm::impl::VecSeq_CUPM< >>>>> >::SetAsync(Vec, PetscScalar, PetscDeviceContext):: [with >>>>> Petsc::device::cupm::DeviceType T = (Petsc::device::cupm::DeviceType)0]?: >>>>> /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/../vecseqcupm_impl.hpp:1313:157: >>>>> required from ?struct Petsc::vec::cupm::impl::VecSeq_CUPM< >>>>> >::SetAsync(Vec, PetscScalar, PetscDeviceContext) [with >>>>> Petsc::device::cupm::DeviceType T = (Petsc::device::cupm::DeviceType)0; >>>>> PetscErrorCode = int; Vec = _p_Vec*; PetscScalar = double; >>>>> PetscDeviceContext = _p_PetscDeviceContext*]::? >>>>> /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/../vecseqcupm_impl.hpp:1313:167: >>>>> required from ?static PetscErrorCode >>>>> Petsc::vec::cupm::impl::VecSeq_CUPM< >::SetAsync(Vec, >>>>> PetscScalar, PetscDeviceContext) [with Petsc::device::cupm::DeviceType T = >>>>> (Petsc::device::cupm::DeviceType)0; PetscErrorCode = int; Vec = _p_Vec*; >>>>> PetscScalar = double; PetscDeviceContext = _p_PetscDeviceContext*]? >>>>> /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/vecseqcupm.cu:7:40: >>>>> required from here >>>>> /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/../vecseqcupm_impl.hpp:1313:150: >>>>> internal compiler error: in maybe_undo_parenthesized_ref, at >>>>> cp/semantics.c:1740 >>>>> PetscCallThrust(THRUST_CALL(thrust::fill, stream, dptr, dptr + >>>>> n, alpha)); >>>>> >>>>> =============== >>>>> >>>>> Could you please show some hints on this issue? >>>>> >>>>> Thanks, >>>>> Qiyue Lu >>>>> >>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay.anl at fastmail.org Tue Sep 24 23:09:29 2024 From: balay.anl at fastmail.org (Satish Balay) Date: Tue, 24 Sep 2024 23:09:29 -0500 (CDT) Subject: [petsc-users] Compiling 3.21.5 version on a Grace Hopper node failed In-Reply-To: References: Message-ID: <6bdace85-516c-3d6a-e9bb-6b961649e9fe@fastmail.org> Can you try the following and see if it makes a difference make V=1 CUDAFLAGS="-g -O0" all Satish On Tue, 24 Sep 2024, Junchao Zhang wrote: > Hi, Qiyue, > I don't have the exact compilers as yours (cuda-12.3 + gcc-13.2.1). I > tried petsc-3.21.5 with cuda-12.6 + gcc-13.3.0 and it compiled fine. > Since it was an internal compiler error, it seems the only option is to > change compilers. > > Thanks. > --Junchao Zhang > > > On Fri, Sep 20, 2024 at 2:22?PM Qiyue Lu wrote: > > > Sure, here it is. Thanks. > > > > On Fri, Sep 20, 2024 at 2:07?PM Junchao Zhang > > wrote: > > > >> Could you send the configure.log file? > >> --Junchao Zhang > >> > >> > >> On Fri, Sep 20, 2024 at 1:33?PM Qiyue Lu wrote: > >> > >>> Thanks for your response. > >>> Here is my module loaded: > >>> > >>> > >>> > >>> *Currently Loaded Modules: 1) craype-arm-grace 4) > >>> perftools-base/24.03.0 7) craype/2.7.31 10) > >>> cray-libsci/24.03.0 13) default 2) libfabric/1.15.2.0 > >>> 5) xpmem/2.8.4-1.0_7.1__ga37cbd9.shasta 8) > >>> cray-dsmml/0.3.0 11) PrgEnv-gnu/8.5.0 3) craype-network-ofi 6) > >>> gcc-native/13.2 9) cray-mpich/8.1.29 12) > >>> cudatoolkit/23.11_12.3* > >>> And here is the configuration option now: > >>> > >>> *--with-cc=cc --with-cxx=CC --with-fc=ftn --download-f2cblaslapack=1 > >>> --with-cuda=1 * > >>> > >>> That error is still there. > >>> > >>> Best, > >>> Qiyue Lu > >>> > >>> On Fri, Sep 20, 2024 at 11:58?AM Junchao Zhang > >>> wrote: > >>> > >>>> /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/../vecseqcupm_impl.hpp:1313:150: > >>>> *internal compiler error*: in maybe_undo_parenthesized_ref, at > >>>> cp/semantics.c:1740 > >>>> > >>>> Did you use nvidia compilers, e.g. nvc, nvc++? If yes, you may want to > >>>> change to other compilers by loading other cray PrgEnvs > >>>> > >>>> --Junchao Zhang > >>>> > >>>> > >>>> On Fri, Sep 20, 2024 at 11:42?AM Qiyue Lu wrote: > >>>> > >>>>> Hello, > >>>>> I am trying to build the PETSc version 3.21.5 (the latest, cloned from > >>>>> git) version on a Grace Hopper node with this configuration options: > >>>>> *--with-cc=cc --with-cxx=CC --with-fc=ftn --download-f2cblaslapack=1 > >>>>> --with-cudac=nvcc --with-cuda=1 --with-threadsafety=1* > >>>>> with module cudatoolkit/23.11_12.3. > >>>>> > >>>>> And I got this error: > >>>>> =============== > >>>>> /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/../vecseqcupm_impl.hpp: > >>>>> In instantiation of ?Petsc::vec::cupm::impl::VecSeq_CUPM< > >>>>> >::SetAsync(Vec, PetscScalar, PetscDeviceContext):: [with > >>>>> Petsc::device::cupm::DeviceType T = (Petsc::device::cupm::DeviceType)0]?: > >>>>> /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/../vecseqcupm_impl.hpp:1313:157: > >>>>> required from ?struct Petsc::vec::cupm::impl::VecSeq_CUPM< > >>>>> >::SetAsync(Vec, PetscScalar, PetscDeviceContext) [with > >>>>> Petsc::device::cupm::DeviceType T = (Petsc::device::cupm::DeviceType)0; > >>>>> PetscErrorCode = int; Vec = _p_Vec*; PetscScalar = double; > >>>>> PetscDeviceContext = _p_PetscDeviceContext*]::? > >>>>> /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/../vecseqcupm_impl.hpp:1313:167: > >>>>> required from ?static PetscErrorCode > >>>>> Petsc::vec::cupm::impl::VecSeq_CUPM< >::SetAsync(Vec, > >>>>> PetscScalar, PetscDeviceContext) [with Petsc::device::cupm::DeviceType T = > >>>>> (Petsc::device::cupm::DeviceType)0; PetscErrorCode = int; Vec = _p_Vec*; > >>>>> PetscScalar = double; PetscDeviceContext = _p_PetscDeviceContext*]? > >>>>> /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/vecseqcupm.cu:7:40: > >>>>> required from here > >>>>> /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/../vecseqcupm_impl.hpp:1313:150: > >>>>> internal compiler error: in maybe_undo_parenthesized_ref, at > >>>>> cp/semantics.c:1740 > >>>>> PetscCallThrust(THRUST_CALL(thrust::fill, stream, dptr, dptr + > >>>>> n, alpha)); > >>>>> > >>>>> =============== > >>>>> > >>>>> Could you please show some hints on this issue? > >>>>> > >>>>> Thanks, > >>>>> Qiyue Lu > >>>>> > >>>> > From matteo.semplice at uninsubria.it Wed Sep 25 08:28:34 2024 From: matteo.semplice at uninsubria.it (Matteo Semplice) Date: Wed, 25 Sep 2024 15:28:34 +0200 Subject: [petsc-users] Mat indices for DMPlex jacobian In-Reply-To: References: Message-ID: On 23/09/2024 17:16, Matthew Knepley wrote: > On Mon, Sep 23, 2024 at 5:33?AM Matteo Semplice via petsc-users > wrote: > > Dear petsc, > > ???? I need to hand-code a jacobian and I can't figure out how to > translate the DMplex points/fields to matrix indices. > > My DMPlex has a section with m fields per cell (which makes for > n>m dof > per cell since some fields are vector). Say I want to insert an nxn > block for the row corresponding to cell c and coloumn of its > neighbour > d. I guess that I should call either > MatSetValues/MatSetValuesLocal or > the blocked variants, but how do I find the row/col indices to > pass in > starting from the c/d dmplex points? And, while I am at it, which > MatSetValues version standard/Local standard/blocked is best? > > I looked for a petsc example, but fails to find it: if there's > one, can > you just point me to it? > > > 1. In FEM, the main unit of indices is usually the closure, so we provide > > https://urldefense.us/v3/__https://petsc.org/main/manualpages/DMPlex/DMPlexGetClosureIndices/__;!!G_uCfscf7eWS!YwF8ebZtFAZdHXKPu98AWB7KPuScfxMu4dRoe8ZqVaPVoEO_nX_qJa-zS6pOWHnPUs2ME6GB7l_SlHBl8ypyOPOQXzbVSC9xXhd8tw$ > > which is what is used inside of > > https://urldefense.us/v3/__https://petsc.org/main/manualpages/DMPlex/DMPlexMatSetClosure/__;!!G_uCfscf7eWS!YwF8ebZtFAZdHXKPu98AWB7KPuScfxMu4dRoe8ZqVaPVoEO_nX_qJa-zS6pOWHnPUs2ME6GB7l_SlHBl8ypyOPOQXzbVSC8i-x5AVA$ > > 2. These indices are calculated using the global section, so you can just > > ? DMGetGlobalSection(dm, &gs); > > and then use PetscSectionGetDof() and PetscSectionGetOffset(), knowing > that > off-process values are encoded as -(off + 1), so you need to convert > those. > > Does this make sense? Yes! Thank you! Matteo -------------- next part -------------- An HTML attachment was scrubbed... URL: From qiyuelu1 at gmail.com Wed Sep 25 08:50:14 2024 From: qiyuelu1 at gmail.com (Qiyue Lu) Date: Wed, 25 Sep 2024 08:50:14 -0500 Subject: [petsc-users] Compiling 3.21.5 version on a Grace Hopper node failed In-Reply-To: <6bdace85-516c-3d6a-e9bb-6b961649e9fe@fastmail.org> References: <6bdace85-516c-3d6a-e9bb-6b961649e9fe@fastmail.org> Message-ID: Thanks Junchao and Satish, However, using 'make V=1 CUDAFLAGS="-g -O0" all' produces the same error. I will forward these discussions to our system admin, hoping they can get some hints on this internal compiler error. Thanks, Qiyue Lu On Tue, Sep 24, 2024 at 11:09?PM Satish Balay wrote: > Can you try the following and see if it makes a difference > > make V=1 CUDAFLAGS="-g -O0" all > > Satish > > On Tue, 24 Sep 2024, Junchao Zhang wrote: > > > Hi, Qiyue, > > I don't have the exact compilers as yours (cuda-12.3 + gcc-13.2.1). I > > tried petsc-3.21.5 with cuda-12.6 + gcc-13.3.0 and it compiled fine. > > Since it was an internal compiler error, it seems the only option is to > > change compilers. > > > > Thanks. > > --Junchao Zhang > > > > > > On Fri, Sep 20, 2024 at 2:22?PM Qiyue Lu wrote: > > > > > Sure, here it is. Thanks. > > > > > > On Fri, Sep 20, 2024 at 2:07?PM Junchao Zhang > > > > wrote: > > > > > >> Could you send the configure.log file? > > >> --Junchao Zhang > > >> > > >> > > >> On Fri, Sep 20, 2024 at 1:33?PM Qiyue Lu wrote: > > >> > > >>> Thanks for your response. > > >>> Here is my module loaded: > > >>> > > >>> > > >>> > > >>> *Currently Loaded Modules: 1) craype-arm-grace 4) > > >>> perftools-base/24.03.0 7) craype/2.7.31 10) > > >>> cray-libsci/24.03.0 13) default 2) libfabric/1.15.2.0 > > >>> < > https://urldefense.us/v3/__http://1.15.2.0__;!!G_uCfscf7eWS!ZHTIMyeRVq_2VWhDO4gvbHRyVBXfTyJ7D25fWIZqvdeq5izIpEwmCtqUmYOjItUtNDX2wgMxilvBpjEZv0WMfOTeAAxd$ > > 5) xpmem/2.8.4-1.0_7.1__ga37cbd9.shasta 8) > > >>> cray-dsmml/0.3.0 11) PrgEnv-gnu/8.5.0 3) craype-network-ofi 6) > > >>> gcc-native/13.2 9) cray-mpich/8.1.29 12) > > >>> cudatoolkit/23.11_12.3* > > >>> And here is the configuration option now: > > >>> > > >>> *--with-cc=cc --with-cxx=CC --with-fc=ftn --download-f2cblaslapack=1 > > >>> --with-cuda=1 * > > >>> > > >>> That error is still there. > > >>> > > >>> Best, > > >>> Qiyue Lu > > >>> > > >>> On Fri, Sep 20, 2024 at 11:58?AM Junchao Zhang < > junchao.zhang at gmail.com> > > >>> wrote: > > >>> > > >>>> > /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/../vecseqcupm_impl.hpp:1313:150: > > >>>> *internal compiler error*: in maybe_undo_parenthesized_ref, at > > >>>> cp/semantics.c:1740 > > >>>> > > >>>> Did you use nvidia compilers, e.g. nvc, nvc++? If yes, you may > want to > > >>>> change to other compilers by loading other cray PrgEnvs > > >>>> > > >>>> --Junchao Zhang > > >>>> > > >>>> > > >>>> On Fri, Sep 20, 2024 at 11:42?AM Qiyue Lu > wrote: > > >>>> > > >>>>> Hello, > > >>>>> I am trying to build the PETSc version 3.21.5 (the latest, cloned > from > > >>>>> git) version on a Grace Hopper node with this configuration > options: > > >>>>> *--with-cc=cc --with-cxx=CC --with-fc=ftn > --download-f2cblaslapack=1 > > >>>>> --with-cudac=nvcc --with-cuda=1 --with-threadsafety=1* > > >>>>> with module cudatoolkit/23.11_12.3. > > >>>>> > > >>>>> And I got this error: > > >>>>> =============== > > >>>>> > /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/../vecseqcupm_impl.hpp: > > >>>>> In instantiation of > ?Petsc::vec::cupm::impl::VecSeq_CUPM< > > >>>>> >::SetAsync(Vec, PetscScalar, PetscDeviceContext):: [with > > >>>>> Petsc::device::cupm::DeviceType T = > (Petsc::device::cupm::DeviceType)0]?: > > >>>>> > /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/../vecseqcupm_impl.hpp:1313:157: > > >>>>> required from ?struct > Petsc::vec::cupm::impl::VecSeq_CUPM< > > >>>>> >::SetAsync(Vec, PetscScalar, PetscDeviceContext) [with > > >>>>> Petsc::device::cupm::DeviceType T = > (Petsc::device::cupm::DeviceType)0; > > >>>>> PetscErrorCode = int; Vec = _p_Vec*; PetscScalar = double; > > >>>>> PetscDeviceContext = _p_PetscDeviceContext*]::? > > >>>>> > /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/../vecseqcupm_impl.hpp:1313:167: > > >>>>> required from ?static PetscErrorCode > > >>>>> Petsc::vec::cupm::impl::VecSeq_CUPM< >::SetAsync(Vec, > > >>>>> PetscScalar, PetscDeviceContext) [with > Petsc::device::cupm::DeviceType T = > > >>>>> (Petsc::device::cupm::DeviceType)0; PetscErrorCode = int; Vec = > _p_Vec*; > > >>>>> PetscScalar = double; PetscDeviceContext = _p_PetscDeviceContext*]? > > >>>>> /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/vecseqcupm.cu:7 > :40: > > >>>>> required from here > > >>>>> > /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/../vecseqcupm_impl.hpp:1313:150: > > >>>>> internal compiler error: in maybe_undo_parenthesized_ref, at > > >>>>> cp/semantics.c:1740 > > >>>>> PetscCallThrust(THRUST_CALL(thrust::fill, stream, dptr, > dptr + > > >>>>> n, alpha)); > > >>>>> > > >>>>> =============== > > >>>>> > > >>>>> Could you please show some hints on this issue? > > >>>>> > > >>>>> Thanks, > > >>>>> Qiyue Lu > > >>>>> > > >>>> > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay.anl at fastmail.org Wed Sep 25 09:03:00 2024 From: balay.anl at fastmail.org (Satish Balay) Date: Wed, 25 Sep 2024 09:03:00 -0500 (CDT) Subject: [petsc-users] Compiling 3.21.5 version on a Grace Hopper node failed In-Reply-To: References: <6bdace85-516c-3d6a-e9bb-6b961649e9fe@fastmail.org> Message-ID: V=1 should print the exact compile command used - that produced that internal compiler error. Satish On Wed, 25 Sep 2024, Qiyue Lu wrote: > Thanks Junchao and Satish, > However, using 'make V=1 CUDAFLAGS="-g -O0" all' produces the same error. > > I will forward these discussions to our system admin, hoping they can get > some hints on this internal compiler error. > > Thanks, > Qiyue Lu > > On Tue, Sep 24, 2024 at 11:09?PM Satish Balay > wrote: > > > Can you try the following and see if it makes a difference > > > > make V=1 CUDAFLAGS="-g -O0" all > > > > Satish > > > > On Tue, 24 Sep 2024, Junchao Zhang wrote: > > > > > Hi, Qiyue, > > > I don't have the exact compilers as yours (cuda-12.3 + gcc-13.2.1). I > > > tried petsc-3.21.5 with cuda-12.6 + gcc-13.3.0 and it compiled fine. > > > Since it was an internal compiler error, it seems the only option is to > > > change compilers. > > > > > > Thanks. > > > --Junchao Zhang > > > > > > > > > On Fri, Sep 20, 2024 at 2:22?PM Qiyue Lu wrote: > > > > > > > Sure, here it is. Thanks. > > > > > > > > On Fri, Sep 20, 2024 at 2:07?PM Junchao Zhang > > > > > > wrote: > > > > > > > >> Could you send the configure.log file? > > > >> --Junchao Zhang > > > >> > > > >> > > > >> On Fri, Sep 20, 2024 at 1:33?PM Qiyue Lu wrote: > > > >> > > > >>> Thanks for your response. > > > >>> Here is my module loaded: > > > >>> > > > >>> > > > >>> > > > >>> *Currently Loaded Modules: 1) craype-arm-grace 4) > > > >>> perftools-base/24.03.0 7) craype/2.7.31 10) > > > >>> cray-libsci/24.03.0 13) default 2) libfabric/1.15.2.0 > > > >>> < > > https://urldefense.us/v3/__http://1.15.2.0__;!!G_uCfscf7eWS!ZHTIMyeRVq_2VWhDO4gvbHRyVBXfTyJ7D25fWIZqvdeq5izIpEwmCtqUmYOjItUtNDX2wgMxilvBpjEZv0WMfOTeAAxd$ > > > 5) xpmem/2.8.4-1.0_7.1__ga37cbd9.shasta 8) > > > >>> cray-dsmml/0.3.0 11) PrgEnv-gnu/8.5.0 3) craype-network-ofi 6) > > > >>> gcc-native/13.2 9) cray-mpich/8.1.29 12) > > > >>> cudatoolkit/23.11_12.3* > > > >>> And here is the configuration option now: > > > >>> > > > >>> *--with-cc=cc --with-cxx=CC --with-fc=ftn --download-f2cblaslapack=1 > > > >>> --with-cuda=1 * > > > >>> > > > >>> That error is still there. > > > >>> > > > >>> Best, > > > >>> Qiyue Lu > > > >>> > > > >>> On Fri, Sep 20, 2024 at 11:58?AM Junchao Zhang < > > junchao.zhang at gmail.com> > > > >>> wrote: > > > >>> > > > >>>> > > /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/../vecseqcupm_impl.hpp:1313:150: > > > >>>> *internal compiler error*: in maybe_undo_parenthesized_ref, at > > > >>>> cp/semantics.c:1740 > > > >>>> > > > >>>> Did you use nvidia compilers, e.g. nvc, nvc++? If yes, you may > > want to > > > >>>> change to other compilers by loading other cray PrgEnvs > > > >>>> > > > >>>> --Junchao Zhang > > > >>>> > > > >>>> > > > >>>> On Fri, Sep 20, 2024 at 11:42?AM Qiyue Lu > > wrote: > > > >>>> > > > >>>>> Hello, > > > >>>>> I am trying to build the PETSc version 3.21.5 (the latest, cloned > > from > > > >>>>> git) version on a Grace Hopper node with this configuration > > options: > > > >>>>> *--with-cc=cc --with-cxx=CC --with-fc=ftn > > --download-f2cblaslapack=1 > > > >>>>> --with-cudac=nvcc --with-cuda=1 --with-threadsafety=1* > > > >>>>> with module cudatoolkit/23.11_12.3. > > > >>>>> > > > >>>>> And I got this error: > > > >>>>> =============== > > > >>>>> > > /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/../vecseqcupm_impl.hpp: > > > >>>>> In instantiation of > > ?Petsc::vec::cupm::impl::VecSeq_CUPM< > > > >>>>> >::SetAsync(Vec, PetscScalar, PetscDeviceContext):: [with > > > >>>>> Petsc::device::cupm::DeviceType T = > > (Petsc::device::cupm::DeviceType)0]?: > > > >>>>> > > /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/../vecseqcupm_impl.hpp:1313:157: > > > >>>>> required from ?struct > > Petsc::vec::cupm::impl::VecSeq_CUPM< > > > >>>>> >::SetAsync(Vec, PetscScalar, PetscDeviceContext) [with > > > >>>>> Petsc::device::cupm::DeviceType T = > > (Petsc::device::cupm::DeviceType)0; > > > >>>>> PetscErrorCode = int; Vec = _p_Vec*; PetscScalar = double; > > > >>>>> PetscDeviceContext = _p_PetscDeviceContext*]::? > > > >>>>> > > /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/../vecseqcupm_impl.hpp:1313:167: > > > >>>>> required from ?static PetscErrorCode > > > >>>>> Petsc::vec::cupm::impl::VecSeq_CUPM< >::SetAsync(Vec, > > > >>>>> PetscScalar, PetscDeviceContext) [with > > Petsc::device::cupm::DeviceType T = > > > >>>>> (Petsc::device::cupm::DeviceType)0; PetscErrorCode = int; Vec = > > _p_Vec*; > > > >>>>> PetscScalar = double; PetscDeviceContext = _p_PetscDeviceContext*]? > > > >>>>> /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/vecseqcupm.cu:7 > > :40: > > > >>>>> required from here > > > >>>>> > > /u/qiyuelu1/petsc/src/vec/vec/impls/seq/cupm/cuda/../vecseqcupm_impl.hpp:1313:150: > > > >>>>> internal compiler error: in maybe_undo_parenthesized_ref, at > > > >>>>> cp/semantics.c:1740 > > > >>>>> PetscCallThrust(THRUST_CALL(thrust::fill, stream, dptr, > > dptr + > > > >>>>> n, alpha)); > > > >>>>> > > > >>>>> =============== > > > >>>>> > > > >>>>> Could you please show some hints on this issue? > > > >>>>> > > > >>>>> Thanks, > > > >>>>> Qiyue Lu > > > >>>>> > > > >>>> > > > > > > From pierreseize at gmail.com Wed Sep 25 13:10:33 2024 From: pierreseize at gmail.com (Pierre Seize) Date: Wed, 25 Sep 2024 20:10:33 +0200 Subject: [petsc-users] DMPlex boundary condition from array Message-ID: Hi I would like to solve div(u) = 0 with a FE method in a spherical mesh made of tetrahedron. My problem concern the boundary condition : a Dirichlet boundary condition on the surface of the sphere where the value is given by another program, in the form of a list of indices corresponding to the vertices on the boundary and a list of values to be applied on those vertices. I am able to use DMPlexCreateFromCellListPetsc to create the DM on process 0, but the moment I distribute the DM the vertices are renumbered and I loose the correspondence with the boundary describing arrays. I have two questions: - is there a way to get the mapping between old indices and new global indices after the distribute operation? - how can I apply a boundary condition such as mine, i.e. set given values at given vertices? Thanks in advance Pierre Seize -------------- next part -------------- An HTML attachment was scrubbed... URL: From mmolinos at us.es Thu Sep 26 05:30:56 2024 From: mmolinos at us.es (MIGUEL MOLINOS PEREZ) Date: Thu, 26 Sep 2024 10:30:56 +0000 Subject: [petsc-users] Ghost particles for DMSWARM (or similar) In-Reply-To: References: <8FBAC7A5-B6AE-4B21-8FEB-52BE1C04A265@us.es> <1B9B1277-9566-444C-9DA8-7ED17684FE01@us.es> Message-ID: <24337E11-33D2-4FFC-89E2-12520AD487FF@us.es> Hi Matt et al, I?ve been working on the scheme that you proposed to create ghost particles (atoms in my case), and it works! With a couple of caveats: -1? In general the overlap particles will be migrate from their own rank to more than one neighbor rank, this is specially relevant for those located close to the corners. Therefore, you'll need to call DMSwarmMigrate several times (27 times for 3D cells), during the migration process. -2? You need to set DMSWARM_MIGRATE_BASIC. Otherwise the proposed algorithm will not work at all! Hope this helps to other folks! I have a follow-up question about periodic bcc on this context, should I open a new thread of keep posting here? Thanks, Miguel On Aug 7, 2024, at 4:22?AM, MIGUEL MOLINOS PEREZ wrote: Thanks Matt, I think I'll start by making a small program as a proof of concept. Then, if it works I'll implement it in my code and I'll be happy to share it too :-) Miguel On Aug 4, 2024, at 3:30?AM, Matthew Knepley wrote: On Fri, Aug 2, 2024 at 7:15?PM MIGUEL MOLINOS PEREZ > wrote: Thanks again Matt, that makes a lot more sense !! Just to check that we are on the same page. You are saying: 1. create a field define a field called "owner rank" for each particle. 2. Identify the phantom particles and modify the internal variable defined by the DMSwarmField_rank variable. 3. Call DMSwarmMigrate(*,PETSC_FALSE), do the calculations using the new local vector including the ghost particles. 4. Then, once the calculations are done, rename the DMSwarmField_rank variable using the "owner rank" variable and call DMSwarmMigrate(*,PETSC_FALSE) once again. I don't think we need this last step. We can just remove those ghost particles for the next step I think. Thanks, Matt Thank you, Miguel On Aug 2, 2024, at 5:33?PM, Matthew Knepley > wrote: On Fri, Aug 2, 2024 at 11:15?AM MIGUEL MOLINOS PEREZ > wrote: Thank you Matt for your time, What you describe seems to me the ideal approach. 1) Add a particle field 'ghost' that identifies ghost vs owned particles. I think it needs options OWNED, OVERLAP, and GHOST This means, locally, I need to allocate Nlocal + ghost particles (duplicated) for my model? I would do it another way. I would allocate the particles with no overlap and set them up. Then I would identify the halo particles, mark them as OVERLAP, call DMSwarmMigrate(), and mark the migrated particles as GHOST, then unmark the OVERLAP particles. Shoot! That marking will not work since we cannot tell the difference between particles we received and particles we sent. Okay, instead of the `ghost` field we need an `owner rank` field. So then we 1) Setup the non-overlapping particles 2) Identify the halo particles 3) Change the `rank`, but not the `owner rank` 4) Call DMSwarmMigrate() Now we can identify ghost particles by the `owner rank` If that so, how to do the communication between the ghost particles living in the rank i and their ?real? counterpart in the rank j. Algo, as an alternative, what about: 1) Use an IS tag which contains, for each rank, a list of the global index of the neighbors particles outside of the rank. 2) Use VecCreateGhost to create a new vector which contains extra local space for the ghost components of the vector. 3) Use VecScatterCreate, VecScatterBegin, and VecScatterEnd to do the transference of data between a vector obtained with DMSwarmCreateGlobalVectorFromField 4) Do necessary computations using the vectors created with VecCreateGhost. This is essentially what Migrate() does. I was trying to reuse the code. Thanks, Matt Thanks, Miguel On Aug 2, 2024, at 8:58?AM, Matthew Knepley > wrote: On Thu, Aug 1, 2024 at 4:40?PM MIGUEL MOLINOS PEREZ > wrote: This Message Is From an External Sender This message came from outside your organization. Dear all, I am implementing a Molecular Dynamics (MD) code using the DMSWARM interface. In the MD simulations we evaluate on each particle (atoms) some kind of scalar functional using data from the neighbouring atoms. My problem lies in the parallel implementation of the model, because sometimes, some of these neighbours lie on a different processor. This is usually solved by using ghost particles. A similar approach (with nodes instead) is already implemented for other PETSc mesh structures like DMPlexConstructGhostCells. Unfortunately, I don't see this kind of constructs for DMSWARM. Am I missing something? I this could be done by applying a buffer region by exploiting the background DMDA mesh that I already use to do domain decomposition. Then using the buffer region of each cell to locate the ghost particles and finally using VecCreateGhost. Is this feasible? Or is there an easier approach using other PETSc functions. This is feasible, but it would be good to develop a set of best practices, since we have been mainly focused on the case of non-redundant particles. Here is how I think I would do what you want. 1) Add a particle field 'ghost' that identifies ghost vs owned particles. I think it needs options OWNED, OVERLAP, and GHOST 2) At some interval identify particles that should be sent to other processes as ghosts. I would call these "overlap particles". The determination seems application specific, so I would leave this determination to the user right now. We do two things to these particles a) Mark chosen particles as OVERLAP b) Change rank to process we are sending to 3) Call DMSwarmMigrate with PETSC_FALSE for the particle deletion flag 4) Mark OVERLAP particles as GHOST when they arrive There is one problem in the above algorithm. It does not allow sending particles to multiple ranks. We would have to do this in phases right now, or make a small adjustment to the interface allowing replication of particles when a set of ranks is specified. THanks, Matt Thank you, Miguel -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!eB1Yyad9mlB8tjMOEd99Bj5SDzHWN37Uw8Bew-IJDgi-9Yy5R460d-GklyL1CM91YOYPHeczTSce1ATv5mR8ZQ$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!eB1Yyad9mlB8tjMOEd99Bj5SDzHWN37Uw8Bew-IJDgi-9Yy5R460d-GklyL1CM91YOYPHeczTSce1ATv5mR8ZQ$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!eB1Yyad9mlB8tjMOEd99Bj5SDzHWN37Uw8Bew-IJDgi-9Yy5R460d-GklyL1CM91YOYPHeczTSce1ATv5mR8ZQ$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Sep 26 08:53:36 2024 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 26 Sep 2024 09:53:36 -0400 Subject: [petsc-users] Ghost particles for DMSWARM (or similar) In-Reply-To: <24337E11-33D2-4FFC-89E2-12520AD487FF@us.es> References: <8FBAC7A5-B6AE-4B21-8FEB-52BE1C04A265@us.es> <1B9B1277-9566-444C-9DA8-7ED17684FE01@us.es> <24337E11-33D2-4FFC-89E2-12520AD487FF@us.es> Message-ID: On Thu, Sep 26, 2024 at 6:31?AM MIGUEL MOLINOS PEREZ wrote: > Hi Matt et al, > > I?ve been working on the scheme that you proposed to create ghost > particles (atoms in my case), and it works! With a couple of caveats: > -1? In general the overlap particles will be migrate from their own rank > to more than one neighbor rank, this is specially relevant for those > located close to the corners. Therefore, you'll need to call DMSwarmMigrate > several times (27 times for 3D cells), during the migration process. > That is terrible. Let's just fix DMSwarmMigrate to have a mode that sends the particle to all overlapping neighbors at once. It can't be that hard. > -2? You need to set DMSWARM_MIGRATE_BASIC. Otherwise the proposed > algorithm will not work at all! > Oh, I should have thought of that. Sorry. I can help code up that extension. Can you take a quick look at the BASIC code? Right now, we just use the rank attached to the particle to send it. We could have an arrays of ranks, but that seems crazy, and would blow up particle storage. How about just adding new particles with the other ranks right before migration? Thanks, Matt > Hope this helps to other folks! > > I have a follow-up question about periodic bcc on this context, should I > open a new thread of keep posting here? > > Thanks, > Miguel > > On Aug 7, 2024, at 4:22?AM, MIGUEL MOLINOS PEREZ wrote: > > Thanks Matt, I think I'll start by making a small program as a proof of > concept. Then, if it works I'll implement it in my code and I'll be happy > to share it too :-) > > Miguel > > On Aug 4, 2024, at 3:30?AM, Matthew Knepley wrote: > > On Fri, Aug 2, 2024 at 7:15?PM MIGUEL MOLINOS PEREZ > wrote: > >> Thanks again Matt, that makes a lot more sense !! >> >> Just to check that we are on the same page. You are saying: >> >> 1. create a field define a field called "owner rank" for each particle. >> >> 2. Identify the phantom particles and modify the internal variable >> defined by the DMSwarmField_rank variable. >> >> 3. Call DMSwarmMigrate(*,PETSC_FALSE), do the calculations using the new >> local vector including the ghost particles. >> >> 4. Then, once the calculations are done, rename the DMSwarmField_rank >> variable using the "owner rank" variable and call >> DMSwarmMigrate(*,PETSC_FALSE) once again. >> > > I don't think we need this last step. We can just remove those ghost > particles for the next step I think. > > Thanks, > > Matt > > >> Thank you, >> Miguel >> >> >> On Aug 2, 2024, at 5:33?PM, Matthew Knepley wrote: >> >> On Fri, Aug 2, 2024 at 11:15?AM MIGUEL MOLINOS PEREZ >> wrote: >> >>> Thank you Matt for your time, >>> >>> What you describe seems to me the ideal approach. >>> >>> 1) Add a particle field 'ghost' that identifies ghost vs owned >>> particles. I think it needs options OWNED, OVERLAP, and GHOST >>> >>> This means, locally, I need to allocate Nlocal + ghost particles >>> (duplicated) for my model? >>> >> >> I would do it another way. I would allocate the particles with no overlap >> and set them up. Then I would identify the halo particles, mark them as >> OVERLAP, call DMSwarmMigrate(), and mark the migrated particles as GHOST, >> then unmark the OVERLAP particles. Shoot! That marking will not work since >> we cannot tell the difference between particles we received and particles >> we sent. Okay, instead of the `ghost` field we need an `owner rank` field. >> So then we >> >> 1) Setup the non-overlapping particles >> >> 2) Identify the halo particles >> >> 3) Change the `rank`, but not the `owner rank` >> >> 4) Call DMSwarmMigrate() >> >> Now we can identify ghost particles by the `owner rank` >> >> >>> If that so, how to do the communication between the ghost particles >>> living in the rank i and their ?real? counterpart in the rank j. >>> >>> Algo, as an alternative, what about: >>> 1) Use an IS tag which contains, for each rank, a list of the global >>> index of the neighbors particles outside of the rank. >>> 2) Use VecCreateGhost to create a new vector which contains extra local >>> space for the ghost components of the vector. >>> 3) Use VecScatterCreate, VecScatterBegin, and VecScatterEnd to do the >>> transference of data between a vector obtained with >>> DMSwarmCreateGlobalVectorFromField >>> 4) Do necessary computations using the vectors created with >>> VecCreateGhost. >>> >> >> This is essentially what Migrate() does. I was trying to reuse the code. >> >> Thanks, >> >> Matt >> >> >>> Thanks, >>> Miguel >>> >>> On Aug 2, 2024, at 8:58?AM, Matthew Knepley wrote: >>> >>> On Thu, Aug 1, 2024 at 4:40?PM MIGUEL MOLINOS PEREZ >>> wrote: >>> >>>> This Message Is From an External Sender >>>> This message came from outside your organization. >>>> >>>> >>>> Dear all, >>>> >>>> I am implementing a Molecular Dynamics (MD) code using the DMSWARM interface. In the MD simulations we evaluate on each particle (atoms) some kind of scalar functional using data from the neighbouring atoms. My problem lies in the parallel implementation of the model, because sometimes, some of these neighbours lie on a different processor. >>>> >>>> This is usually solved by using ghost particles. A similar approach (with nodes instead) is already implemented for other PETSc mesh structures like DMPlexConstructGhostCells. Unfortunately, I don't see this kind of constructs for DMSWARM. Am I missing something? >>>> >>>> I this could be done by applying a buffer region by exploiting the background DMDA mesh that I already use to do domain decomposition. Then using the buffer region of each cell to locate the ghost particles and finally using VecCreateGhost. Is this feasible? Or is there an easier approach using other PETSc functions. >>>> >>>> >>> This is feasible, but it would be good to develop a set of best >>> practices, since we have been mainly focused on the case of non-redundant >>> particles. Here is how I think I would do what you want. >>> >>> 1) Add a particle field 'ghost' that identifies ghost vs owned >>> particles. I think it needs options OWNED, OVERLAP, and GHOST >>> >>> 2) At some interval identify particles that should be sent to other >>> processes as ghosts. I would call these "overlap particles". The >>> determination >>> seems application specific, so I would leave this determination to >>> the user right now. We do two things to these particles >>> >>> a) Mark chosen particles as OVERLAP >>> >>> b) Change rank to process we are sending to >>> >>> 3) Call DMSwarmMigrate with PETSC_FALSE for the particle deletion flag >>> >>> 4) Mark OVERLAP particles as GHOST when they arrive >>> >>> There is one problem in the above algorithm. It does not allow sending >>> particles to multiple ranks. We would have to do this >>> in phases right now, or make a small adjustment to the interface >>> allowing replication of particles when a set of ranks is specified. >>> >>> THanks, >>> >>> Matt >>> >>> >>>> Thank you, >>>> Miguel >>>> >>>> >>>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!b8H6cvtpkxnXlyLYZttri3coxtP79HW3qEuGjuEMbiOmEC6fq3ilao2gtsQ1I5M5Ov09lgyu3rybTv8Yk_Xk$ >>> >>> >>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!b8H6cvtpkxnXlyLYZttri3coxtP79HW3qEuGjuEMbiOmEC6fq3ilao2gtsQ1I5M5Ov09lgyu3rybTv8Yk_Xk$ >> >> >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!b8H6cvtpkxnXlyLYZttri3coxtP79HW3qEuGjuEMbiOmEC6fq3ilao2gtsQ1I5M5Ov09lgyu3rybTv8Yk_Xk$ > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!b8H6cvtpkxnXlyLYZttri3coxtP79HW3qEuGjuEMbiOmEC6fq3ilao2gtsQ1I5M5Ov09lgyu3rybTv8Yk_Xk$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mmolinos at us.es Thu Sep 26 10:20:11 2024 From: mmolinos at us.es (MIGUEL MOLINOS PEREZ) Date: Thu, 26 Sep 2024 15:20:11 +0000 Subject: [petsc-users] Ghost particles for DMSWARM (or similar) In-Reply-To: References: <8FBAC7A5-B6AE-4B21-8FEB-52BE1C04A265@us.es> <1B9B1277-9566-444C-9DA8-7ED17684FE01@us.es> <24337E11-33D2-4FFC-89E2-12520AD487FF@us.es> Message-ID: <681E96BD-62A4-4566-A13E-E034B2F19D54@us.es> Thank you Matt. Okey, let me have a careful look to the DMSwarmMigrate_Push_Basic implementation to see if there is some workaround. The idea of adding new particles is interesting. However, in that case, we need to initialize the new (ghost) particles using the fields of the ?real? particle, right? This can be done using something like: VecGhostUpdateBegin(Vec globalout,InsertMode ADD_VALUES, ScatterMode SCATTER_REVERSE); VecGhostUpdateEnd(Vec globalout,InsertMode ADD_VALUES, ScatterMode SCATTER_REVERSE); for the particle fields (?). Thanks, Miguel On Sep 26, 2024, at 3:53?PM, Matthew Knepley wrote: On Thu, Sep 26, 2024 at 6:31?AM MIGUEL MOLINOS PEREZ > wrote: Hi Matt et al, I?ve been working on the scheme that you proposed to create ghost particles (atoms in my case), and it works! With a couple of caveats: -1? In general the overlap particles will be migrate from their own rank to more than one neighbor rank, this is specially relevant for those located close to the corners. Therefore, you'll need to call DMSwarmMigrate several times (27 times for 3D cells), during the migration process. That is terrible. Let's just fix DMSwarmMigrate to have a mode that sends the particle to all overlapping neighbors at once. It can't be that hard. -2? You need to set DMSWARM_MIGRATE_BASIC. Otherwise the proposed algorithm will not work at all! Oh, I should have thought of that. Sorry. I can help code up that extension. Can you take a quick look at the BASIC code? Right now, we just use the rank attached to the particle to send it. We could have an arrays of ranks, but that seems crazy, and would blow up particle storage. How about just adding new particles with the other ranks right before migration? Thanks, Matt Hope this helps to other folks! I have a follow-up question about periodic bcc on this context, should I open a new thread of keep posting here? Thanks, Miguel On Aug 7, 2024, at 4:22?AM, MIGUEL MOLINOS PEREZ > wrote: Thanks Matt, I think I'll start by making a small program as a proof of concept. Then, if it works I'll implement it in my code and I'll be happy to share it too :-) Miguel On Aug 4, 2024, at 3:30?AM, Matthew Knepley > wrote: On Fri, Aug 2, 2024 at 7:15?PM MIGUEL MOLINOS PEREZ > wrote: Thanks again Matt, that makes a lot more sense !! Just to check that we are on the same page. You are saying: 1. create a field define a field called "owner rank" for each particle. 2. Identify the phantom particles and modify the internal variable defined by the DMSwarmField_rank variable. 3. Call DMSwarmMigrate(*,PETSC_FALSE), do the calculations using the new local vector including the ghost particles. 4. Then, once the calculations are done, rename the DMSwarmField_rank variable using the "owner rank" variable and call DMSwarmMigrate(*,PETSC_FALSE) once again. I don't think we need this last step. We can just remove those ghost particles for the next step I think. Thanks, Matt Thank you, Miguel On Aug 2, 2024, at 5:33?PM, Matthew Knepley > wrote: On Fri, Aug 2, 2024 at 11:15?AM MIGUEL MOLINOS PEREZ > wrote: Thank you Matt for your time, What you describe seems to me the ideal approach. 1) Add a particle field 'ghost' that identifies ghost vs owned particles. I think it needs options OWNED, OVERLAP, and GHOST This means, locally, I need to allocate Nlocal + ghost particles (duplicated) for my model? I would do it another way. I would allocate the particles with no overlap and set them up. Then I would identify the halo particles, mark them as OVERLAP, call DMSwarmMigrate(), and mark the migrated particles as GHOST, then unmark the OVERLAP particles. Shoot! That marking will not work since we cannot tell the difference between particles we received and particles we sent. Okay, instead of the `ghost` field we need an `owner rank` field. So then we 1) Setup the non-overlapping particles 2) Identify the halo particles 3) Change the `rank`, but not the `owner rank` 4) Call DMSwarmMigrate() Now we can identify ghost particles by the `owner rank` If that so, how to do the communication between the ghost particles living in the rank i and their ?real? counterpart in the rank j. Algo, as an alternative, what about: 1) Use an IS tag which contains, for each rank, a list of the global index of the neighbors particles outside of the rank. 2) Use VecCreateGhost to create a new vector which contains extra local space for the ghost components of the vector. 3) Use VecScatterCreate, VecScatterBegin, and VecScatterEnd to do the transference of data between a vector obtained with DMSwarmCreateGlobalVectorFromField 4) Do necessary computations using the vectors created with VecCreateGhost. This is essentially what Migrate() does. I was trying to reuse the code. Thanks, Matt Thanks, Miguel On Aug 2, 2024, at 8:58?AM, Matthew Knepley > wrote: On Thu, Aug 1, 2024 at 4:40?PM MIGUEL MOLINOS PEREZ > wrote: This Message Is From an External Sender This message came from outside your organization. Dear all, I am implementing a Molecular Dynamics (MD) code using the DMSWARM interface. In the MD simulations we evaluate on each particle (atoms) some kind of scalar functional using data from the neighbouring atoms. My problem lies in the parallel implementation of the model, because sometimes, some of these neighbours lie on a different processor. This is usually solved by using ghost particles. A similar approach (with nodes instead) is already implemented for other PETSc mesh structures like DMPlexConstructGhostCells. Unfortunately, I don't see this kind of constructs for DMSWARM. Am I missing something? I this could be done by applying a buffer region by exploiting the background DMDA mesh that I already use to do domain decomposition. Then using the buffer region of each cell to locate the ghost particles and finally using VecCreateGhost. Is this feasible? Or is there an easier approach using other PETSc functions. This is feasible, but it would be good to develop a set of best practices, since we have been mainly focused on the case of non-redundant particles. Here is how I think I would do what you want. 1) Add a particle field 'ghost' that identifies ghost vs owned particles. I think it needs options OWNED, OVERLAP, and GHOST 2) At some interval identify particles that should be sent to other processes as ghosts. I would call these "overlap particles". The determination seems application specific, so I would leave this determination to the user right now. We do two things to these particles a) Mark chosen particles as OVERLAP b) Change rank to process we are sending to 3) Call DMSwarmMigrate with PETSC_FALSE for the particle deletion flag 4) Mark OVERLAP particles as GHOST when they arrive There is one problem in the above algorithm. It does not allow sending particles to multiple ranks. We would have to do this in phases right now, or make a small adjustment to the interface allowing replication of particles when a set of ranks is specified. THanks, Matt Thank you, Miguel -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fP7OWxTbYKTmKcfmVZotGtqEWwn7qb1U_unUjri1y_h3eMgj6qvPCAKa4jvQNLIS8llUk_EbKNwPj4vDGF_YTQ$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fP7OWxTbYKTmKcfmVZotGtqEWwn7qb1U_unUjri1y_h3eMgj6qvPCAKa4jvQNLIS8llUk_EbKNwPj4vDGF_YTQ$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fP7OWxTbYKTmKcfmVZotGtqEWwn7qb1U_unUjri1y_h3eMgj6qvPCAKa4jvQNLIS8llUk_EbKNwPj4vDGF_YTQ$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fP7OWxTbYKTmKcfmVZotGtqEWwn7qb1U_unUjri1y_h3eMgj6qvPCAKa4jvQNLIS8llUk_EbKNwPj4vDGF_YTQ$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bramkamp at nsc.liu.se Thu Sep 26 12:50:37 2024 From: bramkamp at nsc.liu.se (Frank Bramkamp) Date: Thu, 26 Sep 2024 19:50:37 +0200 Subject: [petsc-users] MatCopy question Message-ID: <38FFF448-9C43-4D1D-8892-E19F3970CD6D@nsc.liu.se> Dear PETSc team, I would like to create two matrices. The first matrix is based on a compact stencil (COMPACT), and the second one has an extended stencil (WIDE) So the non-zero elements of the COMPACT matrix is contained in the WIDE matrix as well. But the WIDE matrix shall contain some additional elements. I try to create the COMPACT matrix first, then copy it to the WIDE matrix and then add some more terms to the WIDE matrix. We want to create a matrix that is based on a lower order approximation of our scheme which serves as basis matrix for preconditioning, where we want to have a more accurate jacobian which has a larger stencil, that is based on the lower approximation plus some additional non-zero elements. We want to avoid that we have to call the same routines twice to setup the more accurate matrix, but copy the elements from the lower order (compact) matrix. I try to use MatCopy to copy the COMPACT matrix into the WIDE matrix. Somehow I cannot figure out to do this, since there is always a problem wirh non conforming sizes [0]PETSC ERROR: Nonconforming object sizes [0]PETSC ERROR: Mat A,Mat B: global dim (4,8) (4,8) Here is a basic example that I try. Adding additional elements to the WIDE matrix is disabled right now, since MatCopy already gives a problem. I thought that in MatCopy(COMPACT, WIDE, DIFFERENT_NONZERO_PATTERN, ierr) the option DIFFERENT_NONZERO_PATTERN should enable to copy different matrix sizes into each other. Or is that not possible to do ?! Here the COMPACT matrix is a 4x4 matrix and the WIDE matrix 8x8, for a basic test. Thanks, Frank Bramkamp program petsc_matrix_example #include use petsc implicit none PetscErrorCode :: ierr Mat :: COMPACT, WIDE PetscInt :: m, n, MM, NN, d_nz PetscInt :: i, j PetscScalar :: additional_value ! Initialize PETSc call PetscInitialize(PETSC_NULL_CHARACTER, ierr) ! Set dimensions for COMPACT matrix m = 4 ! Local rows n = 4 ! Local columns MM = 4 ! Global rows NN = 4 ! Global columns d_nz = 4 ! Estimated non-zeros per row (diagonal) o_nz = 0 ! Estimated non-zeros per row (off-diagonal) ! Create COMPACT matrix call MatCreateSeqAIJ(PETSC_COMM_SELF, MM, NN, PETSC_DECIDE, PETSC_NULL_INTEGER, COMPACT, ierr) ! Set some values in COMPACT matrix (example) do i = 0, M-1 do i = 0, NN-1 !do j = max(0, i-1), min(NN-1, i+1) do j = 0,NN-1 call MatSetValue(COMPACT, i, j, 1.0_PETSC_REAL_KIND, INSERT_VALUES, ierr) end do end do ! Assemble COMPACT matrix call MatAssemblyBegin(COMPACT, MAT_FINAL_ASSEMBLY, ierr) call MatAssemblyEnd(COMPACT, MAT_FINAL_ASSEMBLY, ierr) ! Set dimensions for WIDE matrix n = 8 ! Increase local columns NN = 8 ! Increase global columns m = 8 MM = 8 d_nz = 8 ! Create WIDE matrix call MatCreateSeqAIJ(PETSC_COMM_SELF, MM, NN, PETSC_DECIDE, PETSC_NULL_INTEGER, WIDE, ierr) ! Assemble WIDE matrix: MUST WE ASSMBLE THE MATRIX BEFORE MatCOPY ?! call MatAssemblyBegin(WIDE, MAT_FINAL_ASSEMBLY, ierr) call MatAssemblyEnd(WIDE, MAT_FINAL_ASSEMBLY, ierr) ! Copy elements from COMPACT to WIDE call MatCopy(COMPACT, WIDE, DIFFERENT_NONZERO_PATTERN, ierr) !call MatAssemblyBegin(WIDE, MAT_FINAL_ASSEMBLY, ierr) ! Add additional elements to WIDE matrix !!$ additional_value = 10.0 !!$ !!$ ! original !!$ !do i = 0, MM-1 !!$ ! do j = NN/2, NN-1 !!$ ! call MatSetValue(WIDE, i, j, additional_value, INSERT_VALUES, ierr) !!$ ! end do !!$ !end do !!$ !!$ do i = 0, NN-1 !!$ do j = max(0, i-1), min(NN-1, i+1) !!$ call MatSetValue(WIDE, i, j, 10.0_PETSC_REAL_KIND, ADD_VALUES, ierr) !!$ end do !!$ end do ! Assemble WIDE matrix again call MatAssemblyBegin(WIDE, MAT_FINAL_ASSEMBLY, ierr) call MatAssemblyEnd(WIDE, MAT_FINAL_ASSEMBLY, ierr) ! View matrices (optional) call PetscPrintf(PETSC_COMM_WORLD, "COMPACT matrix:\n", ierr) call MatView(COMPACT, PETSC_VIEWER_STDOUT_WORLD, ierr) call PetscPrintf(PETSC_COMM_WORLD, "WIDE matrix:\n", ierr) call MatView(WIDE, PETSC_VIEWER_STDOUT_WORLD, ierr) ! Clean up call MatDestroy(COMPACT, ierr) call MatDestroy(WIDE, ierr) ! Finalize PETSc call PetscFinalize(ierr) end program petsc_matrix_example From bsmith at petsc.dev Thu Sep 26 13:13:22 2024 From: bsmith at petsc.dev (Barry Smith) Date: Thu, 26 Sep 2024 14:13:22 -0400 Subject: [petsc-users] MatCopy question In-Reply-To: <38FFF448-9C43-4D1D-8892-E19F3970CD6D@nsc.liu.se> References: <38FFF448-9C43-4D1D-8892-E19F3970CD6D@nsc.liu.se> Message-ID: <1E632159-7C9C-45EA-B66A-A95313C65D80@petsc.dev> Frank, Sorry for the confusion. MatCopy() and MatAXPY() only work for identically sized matrices. Even if the two matrices are the same size but one, say A, has a subset of nonzeros than the other, say B, what you want to do is non-trivial. This is because to do a copy you must MatAssembly the matrix first and this squeezes out the zero locations that you would like to fill in B. A simple solution is in the code that computes the subset of entries, call MatSetValues() twice, once for A and once for B. Not entirely satisfying but it will work and be reasonably efficient. Barry > On Sep 26, 2024, at 1:50?PM, Frank Bramkamp wrote: > > Dear PETSc team, > > > I would like to create two matrices. The first matrix is based on a compact stencil (COMPACT), and the second one has an extended stencil (WIDE) > So the non-zero elements of the COMPACT matrix is contained in the WIDE matrix as well. But the WIDE matrix shall contain some additional elements. > I try to create the COMPACT matrix first, then copy it to the WIDE matrix and then add some more terms to the WIDE matrix. > We want to create a matrix that is based on a lower order approximation of our scheme which serves as basis matrix for preconditioning, > where we want to have a more accurate jacobian which has a larger stencil, that is based on the lower approximation plus some additional non-zero elements. > We want to avoid that we have to call the same routines twice to setup the more accurate matrix, but copy the elements from the lower order (compact) matrix. > > > I try to use MatCopy to copy the COMPACT matrix into the WIDE matrix. Somehow I cannot figure out to do this, > since there is always a problem wirh non conforming sizes > > [0]PETSC ERROR: Nonconforming object sizes > [0]PETSC ERROR: Mat A,Mat B: global dim (4,8) (4,8) > > > Here is a basic example that I try. > Adding additional elements to the WIDE matrix is disabled right now, since MatCopy already gives a problem. > I thought that in MatCopy(COMPACT, WIDE, DIFFERENT_NONZERO_PATTERN, ierr) > the option DIFFERENT_NONZERO_PATTERN should enable to copy different matrix sizes into each other. > Or is that not possible to do ?! > > Here the COMPACT matrix is a 4x4 matrix and the WIDE matrix 8x8, for a basic test. > > Thanks, Frank Bramkamp > > > > > program petsc_matrix_example > #include > use petsc > implicit none > > PetscErrorCode :: ierr > Mat :: COMPACT, WIDE > PetscInt :: m, n, MM, NN, d_nz > PetscInt :: i, j > PetscScalar :: additional_value > > ! Initialize PETSc > call PetscInitialize(PETSC_NULL_CHARACTER, ierr) > > ! Set dimensions for COMPACT matrix > m = 4 ! Local rows > n = 4 ! Local columns > MM = 4 ! Global rows > NN = 4 ! Global columns > d_nz = 4 ! Estimated non-zeros per row (diagonal) o_nz = 0 ! Estimated non-zeros per row (off-diagonal) > > ! Create COMPACT matrix > call MatCreateSeqAIJ(PETSC_COMM_SELF, MM, NN, PETSC_DECIDE, PETSC_NULL_INTEGER, COMPACT, ierr) > > ! Set some values in COMPACT matrix (example) do i = 0, M-1 > do i = 0, NN-1 > !do j = max(0, i-1), min(NN-1, i+1) > do j = 0,NN-1 > call MatSetValue(COMPACT, i, j, 1.0_PETSC_REAL_KIND, INSERT_VALUES, ierr) > end do > end do > > ! Assemble COMPACT matrix > call MatAssemblyBegin(COMPACT, MAT_FINAL_ASSEMBLY, ierr) > call MatAssemblyEnd(COMPACT, MAT_FINAL_ASSEMBLY, ierr) > > ! Set dimensions for WIDE matrix > n = 8 ! Increase local columns > NN = 8 ! Increase global columns > m = 8 > MM = 8 > d_nz = 8 > > ! Create WIDE matrix > call MatCreateSeqAIJ(PETSC_COMM_SELF, MM, NN, PETSC_DECIDE, PETSC_NULL_INTEGER, WIDE, ierr) > > ! Assemble WIDE matrix: MUST WE ASSMBLE THE MATRIX BEFORE MatCOPY ?! > call MatAssemblyBegin(WIDE, MAT_FINAL_ASSEMBLY, ierr) > call MatAssemblyEnd(WIDE, MAT_FINAL_ASSEMBLY, ierr) > > ! Copy elements from COMPACT to WIDE > call MatCopy(COMPACT, WIDE, DIFFERENT_NONZERO_PATTERN, ierr) > > > > !call MatAssemblyBegin(WIDE, MAT_FINAL_ASSEMBLY, ierr) > > ! Add additional elements to WIDE matrix > !!$ additional_value = 10.0 > !!$ > !!$ ! original > !!$ !do i = 0, MM-1 > !!$ ! do j = NN/2, NN-1 > !!$ ! call MatSetValue(WIDE, i, j, additional_value, INSERT_VALUES, ierr) > !!$ ! end do > !!$ !end do > !!$ > !!$ do i = 0, NN-1 > !!$ do j = max(0, i-1), min(NN-1, i+1) > !!$ call MatSetValue(WIDE, i, j, 10.0_PETSC_REAL_KIND, ADD_VALUES, ierr) > !!$ end do > !!$ end do > > > ! Assemble WIDE matrix again > call MatAssemblyBegin(WIDE, MAT_FINAL_ASSEMBLY, ierr) > call MatAssemblyEnd(WIDE, MAT_FINAL_ASSEMBLY, ierr) > > ! View matrices (optional) > call PetscPrintf(PETSC_COMM_WORLD, "COMPACT matrix:\n", ierr) > call MatView(COMPACT, PETSC_VIEWER_STDOUT_WORLD, ierr) > call PetscPrintf(PETSC_COMM_WORLD, "WIDE matrix:\n", ierr) > call MatView(WIDE, PETSC_VIEWER_STDOUT_WORLD, ierr) > > ! Clean up > call MatDestroy(COMPACT, ierr) > call MatDestroy(WIDE, ierr) > > ! Finalize PETSc > call PetscFinalize(ierr) > > end program petsc_matrix_example > > From junchao.zhang at gmail.com Thu Sep 26 13:27:41 2024 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Thu, 26 Sep 2024 13:27:41 -0500 Subject: [petsc-users] MatCopy question In-Reply-To: <1E632159-7C9C-45EA-B66A-A95313C65D80@petsc.dev> References: <38FFF448-9C43-4D1D-8892-E19F3970CD6D@nsc.liu.se> <1E632159-7C9C-45EA-B66A-A95313C65D80@petsc.dev> Message-ID: Is it possible to set the size of the COMPACT matrix as large as the WIDE matrix, then do MatAYPX(WIDE, 0, COMPACT, SUBSET_NONZERO_PATTERN), i.e, WIDE = 0*WIDE + COMPACT? You need to build WIDE upfront once, but if you repeat, the other MatAYPX will be fast. --Junchao Zhang On Thu, Sep 26, 2024 at 1:13?PM Barry Smith wrote: > > Frank, > > Sorry for the confusion. MatCopy() and MatAXPY() only work for > identically sized matrices. > > Even if the two matrices are the same size but one, say A, has a subset > of nonzeros than the other, say B, what you want to do is non-trivial. This > is because to do a copy you must MatAssembly the matrix first and this > squeezes out the zero locations that you would like to fill in B. > > A simple solution is in the code that computes the subset of entries, > call MatSetValues() twice, once for A and once for B. Not entirely > satisfying but it will work and be reasonably efficient. > > > Barry > > > > On Sep 26, 2024, at 1:50?PM, Frank Bramkamp wrote: > > > > Dear PETSc team, > > > > > > I would like to create two matrices. The first matrix is based on a > compact stencil (COMPACT), and the second one has an extended stencil (WIDE) > > So the non-zero elements of the COMPACT matrix is contained in the WIDE > matrix as well. But the WIDE matrix shall contain some additional elements. > > I try to create the COMPACT matrix first, then copy it to the WIDE > matrix and then add some more terms to the WIDE matrix. > > We want to create a matrix that is based on a lower order approximation > of our scheme which serves as basis matrix for preconditioning, > > where we want to have a more accurate jacobian which has a larger > stencil, that is based on the lower approximation plus some additional > non-zero elements. > > We want to avoid that we have to call the same routines twice to setup > the more accurate matrix, but copy the elements from the lower order > (compact) matrix. > > > > > > I try to use MatCopy to copy the COMPACT matrix into the WIDE matrix. > Somehow I cannot figure out to do this, > > since there is always a problem wirh non conforming sizes > > > > [0]PETSC ERROR: Nonconforming object sizes > > [0]PETSC ERROR: Mat A,Mat B: global dim (4,8) (4,8) > > > > > > Here is a basic example that I try. > > Adding additional elements to the WIDE matrix is disabled right now, > since MatCopy already gives a problem. > > I thought that in MatCopy(COMPACT, WIDE, DIFFERENT_NONZERO_PATTERN, ierr) > > the option DIFFERENT_NONZERO_PATTERN should enable to copy different > matrix sizes into each other. > > Or is that not possible to do ?! > > > > Here the COMPACT matrix is a 4x4 matrix and the WIDE matrix 8x8, for a > basic test. > > > > Thanks, Frank Bramkamp > > > > > > > > > > program petsc_matrix_example > > #include > > use petsc > > implicit none > > > > PetscErrorCode :: ierr > > Mat :: COMPACT, WIDE > > PetscInt :: m, n, MM, NN, d_nz > > PetscInt :: i, j > > PetscScalar :: additional_value > > > > ! Initialize PETSc > > call PetscInitialize(PETSC_NULL_CHARACTER, ierr) > > > > ! Set dimensions for COMPACT matrix > > m = 4 ! Local rows > > n = 4 ! Local columns > > MM = 4 ! Global rows > > NN = 4 ! Global columns > > d_nz = 4 ! Estimated non-zeros per row (diagonal) o_nz = 0 ! Estimated > non-zeros per row (off-diagonal) > > > > ! Create COMPACT matrix > > call MatCreateSeqAIJ(PETSC_COMM_SELF, MM, NN, PETSC_DECIDE, > PETSC_NULL_INTEGER, COMPACT, ierr) > > > > ! Set some values in COMPACT matrix (example) do i = 0, M-1 > > do i = 0, NN-1 > > !do j = max(0, i-1), min(NN-1, i+1) > > do j = 0,NN-1 > > call MatSetValue(COMPACT, i, j, 1.0_PETSC_REAL_KIND, INSERT_VALUES, > ierr) > > end do > > end do > > > > ! Assemble COMPACT matrix > > call MatAssemblyBegin(COMPACT, MAT_FINAL_ASSEMBLY, ierr) > > call MatAssemblyEnd(COMPACT, MAT_FINAL_ASSEMBLY, ierr) > > > > ! Set dimensions for WIDE matrix > > n = 8 ! Increase local columns > > NN = 8 ! Increase global columns > > m = 8 > > MM = 8 > > d_nz = 8 > > > > ! Create WIDE matrix > > call MatCreateSeqAIJ(PETSC_COMM_SELF, MM, NN, PETSC_DECIDE, > PETSC_NULL_INTEGER, WIDE, ierr) > > > > ! Assemble WIDE matrix: MUST WE ASSMBLE THE MATRIX BEFORE MatCOPY ?! > > call MatAssemblyBegin(WIDE, MAT_FINAL_ASSEMBLY, ierr) > > call MatAssemblyEnd(WIDE, MAT_FINAL_ASSEMBLY, ierr) > > > > ! Copy elements from COMPACT to WIDE > > call MatCopy(COMPACT, WIDE, DIFFERENT_NONZERO_PATTERN, ierr) > > > > > > > > !call MatAssemblyBegin(WIDE, MAT_FINAL_ASSEMBLY, ierr) > > > > ! Add additional elements to WIDE matrix > > !!$ additional_value = 10.0 > > !!$ > > !!$ ! original > > !!$ !do i = 0, MM-1 > > !!$ ! do j = NN/2, NN-1 > > !!$ ! call MatSetValue(WIDE, i, j, additional_value, INSERT_VALUES, > ierr) > > !!$ ! end do > > !!$ !end do > > !!$ > > !!$ do i = 0, NN-1 > > !!$ do j = max(0, i-1), min(NN-1, i+1) > > !!$ call MatSetValue(WIDE, i, j, 10.0_PETSC_REAL_KIND, ADD_VALUES, > ierr) > > !!$ end do > > !!$ end do > > > > > > ! Assemble WIDE matrix again > > call MatAssemblyBegin(WIDE, MAT_FINAL_ASSEMBLY, ierr) > > call MatAssemblyEnd(WIDE, MAT_FINAL_ASSEMBLY, ierr) > > > > ! View matrices (optional) > > call PetscPrintf(PETSC_COMM_WORLD, "COMPACT matrix:\n", ierr) > > call MatView(COMPACT, PETSC_VIEWER_STDOUT_WORLD, ierr) > > call PetscPrintf(PETSC_COMM_WORLD, "WIDE matrix:\n", ierr) > > call MatView(WIDE, PETSC_VIEWER_STDOUT_WORLD, ierr) > > > > ! Clean up > > call MatDestroy(COMPACT, ierr) > > call MatDestroy(WIDE, ierr) > > > > ! Finalize PETSc > > call PetscFinalize(ierr) > > > > end program petsc_matrix_example > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From junchao.zhang at gmail.com Thu Sep 26 13:36:27 2024 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Thu, 26 Sep 2024 13:36:27 -0500 Subject: [petsc-users] MatCopy question In-Reply-To: References: <38FFF448-9C43-4D1D-8892-E19F3970CD6D@nsc.liu.se> <1E632159-7C9C-45EA-B66A-A95313C65D80@petsc.dev> Message-ID: No, Setting a bigger matrix size won't have a negative impact. Only the actual number of nonzeros matters. --Junchao Zhang On Thu, Sep 26, 2024 at 1:33?PM Frank Bramkamp wrote: > Thanks for the reply, > > when setting the size larger than needed, would that have any negative > impact somewhere else ?! > We typically pre-determine how many non-zero elements are present, when > using MatCreate. > > Then, the smaller size needs the same memory as the larger matrix ?! > > Thanks, Frank > > > > > On 26 Sep 2024, at 20:27, Junchao Zhang wrote: > > Is it possible to set the size of the COMPACT matrix as large as the WIDE > matrix, then do MatAYPX(WIDE, 0, COMPACT, SUBSET_NONZERO_PATTERN), i.e, > WIDE = 0*WIDE + COMPACT? You need to build WIDE upfront once, but if you > repeat, the other MatAYPX will be fast. > > --Junchao Zhang > > > On Thu, Sep 26, 2024 at 1:13?PM Barry Smith wrote: > >> >> Frank, >> >> Sorry for the confusion. MatCopy() and MatAXPY() only work for >> identically sized matrices. >> >> Even if the two matrices are the same size but one, say A, has a >> subset of nonzeros than the other, say B, what you want to do is >> non-trivial. This is because to do a copy you must MatAssembly the matrix >> first and this squeezes out the zero locations that you would like to fill >> in B. >> >> A simple solution is in the code that computes the subset of entries, >> call MatSetValues() twice, once for A and once for B. Not entirely >> satisfying but it will work and be reasonably efficient. >> >> >> Barry >> >> >> > On Sep 26, 2024, at 1:50?PM, Frank Bramkamp >> wrote: >> > >> > Dear PETSc team, >> > >> > >> > I would like to create two matrices. The first matrix is based on a >> compact stencil (COMPACT), and the second one has an extended stencil (WIDE) >> > So the non-zero elements of the COMPACT matrix is contained in the >> WIDE matrix as well. But the WIDE matrix shall contain some additional >> elements. >> > I try to create the COMPACT matrix first, then copy it to the WIDE >> matrix and then add some more terms to the WIDE matrix. >> > We want to create a matrix that is based on a lower order approximation >> of our scheme which serves as basis matrix for preconditioning, >> > where we want to have a more accurate jacobian which has a larger >> stencil, that is based on the lower approximation plus some additional >> non-zero elements. >> > We want to avoid that we have to call the same routines twice to setup >> the more accurate matrix, but copy the elements from the lower order >> (compact) matrix. >> > >> > >> > I try to use MatCopy to copy the COMPACT matrix into the WIDE matrix. >> Somehow I cannot figure out to do this, >> > since there is always a problem wirh non conforming sizes >> > >> > [0]PETSC ERROR: Nonconforming object sizes >> > [0]PETSC ERROR: Mat A,Mat B: global dim (4,8) (4,8) >> > >> > >> > Here is a basic example that I try. >> > Adding additional elements to the WIDE matrix is disabled right now, >> since MatCopy already gives a problem. >> > I thought that in MatCopy(COMPACT, WIDE, DIFFERENT_NONZERO_PATTERN, >> ierr) >> > the option DIFFERENT_NONZERO_PATTERN should enable to copy different >> matrix sizes into each other. >> > Or is that not possible to do ?! >> > >> > Here the COMPACT matrix is a 4x4 matrix and the WIDE matrix 8x8, for a >> basic test. >> > >> > Thanks, Frank Bramkamp >> > >> > >> > >> > >> > program petsc_matrix_example >> > #include >> > use petsc >> > implicit none >> > >> > PetscErrorCode :: ierr >> > Mat :: COMPACT, WIDE >> > PetscInt :: m, n, MM, NN, d_nz >> > PetscInt :: i, j >> > PetscScalar :: additional_value >> > >> > ! Initialize PETSc >> > call PetscInitialize(PETSC_NULL_CHARACTER, ierr) >> > >> > ! Set dimensions for COMPACT matrix >> > m = 4 ! Local rows >> > n = 4 ! Local columns >> > MM = 4 ! Global rows >> > NN = 4 ! Global columns >> > d_nz = 4 ! Estimated non-zeros per row (diagonal) o_nz = 0 ! Estimated >> non-zeros per row (off-diagonal) >> > >> > ! Create COMPACT matrix >> > call MatCreateSeqAIJ(PETSC_COMM_SELF, MM, NN, PETSC_DECIDE, >> PETSC_NULL_INTEGER, COMPACT, ierr) >> > >> > ! Set some values in COMPACT matrix (example) do i = 0, M-1 >> > do i = 0, NN-1 >> > !do j = max(0, i-1), min(NN-1, i+1) >> > do j = 0,NN-1 >> > call MatSetValue(COMPACT, i, j, 1.0_PETSC_REAL_KIND, >> INSERT_VALUES, ierr) >> > end do >> > end do >> > >> > ! Assemble COMPACT matrix >> > call MatAssemblyBegin(COMPACT, MAT_FINAL_ASSEMBLY, ierr) >> > call MatAssemblyEnd(COMPACT, MAT_FINAL_ASSEMBLY, ierr) >> > >> > ! Set dimensions for WIDE matrix >> > n = 8 ! Increase local columns >> > NN = 8 ! Increase global columns >> > m = 8 >> > MM = 8 >> > d_nz = 8 >> > >> > ! Create WIDE matrix >> > call MatCreateSeqAIJ(PETSC_COMM_SELF, MM, NN, PETSC_DECIDE, >> PETSC_NULL_INTEGER, WIDE, ierr) >> > >> > ! Assemble WIDE matrix: MUST WE ASSMBLE THE MATRIX BEFORE MatCOPY ?! >> > call MatAssemblyBegin(WIDE, MAT_FINAL_ASSEMBLY, ierr) >> > call MatAssemblyEnd(WIDE, MAT_FINAL_ASSEMBLY, ierr) >> > >> > ! Copy elements from COMPACT to WIDE >> > call MatCopy(COMPACT, WIDE, DIFFERENT_NONZERO_PATTERN, ierr) >> > >> > >> > >> > !call MatAssemblyBegin(WIDE, MAT_FINAL_ASSEMBLY, ierr) >> > >> > ! Add additional elements to WIDE matrix >> > !!$ additional_value = 10.0 >> > !!$ >> > !!$ ! original >> > !!$ !do i = 0, MM-1 >> > !!$ ! do j = NN/2, NN-1 >> > !!$ ! call MatSetValue(WIDE, i, j, additional_value, INSERT_VALUES, >> ierr) >> > !!$ ! end do >> > !!$ !end do >> > !!$ >> > !!$ do i = 0, NN-1 >> > !!$ do j = max(0, i-1), min(NN-1, i+1) >> > !!$ call MatSetValue(WIDE, i, j, 10.0_PETSC_REAL_KIND, ADD_VALUES, >> ierr) >> > !!$ end do >> > !!$ end do >> > >> > >> > ! Assemble WIDE matrix again >> > call MatAssemblyBegin(WIDE, MAT_FINAL_ASSEMBLY, ierr) >> > call MatAssemblyEnd(WIDE, MAT_FINAL_ASSEMBLY, ierr) >> > >> > ! View matrices (optional) >> > call PetscPrintf(PETSC_COMM_WORLD, "COMPACT matrix:\n", ierr) >> > call MatView(COMPACT, PETSC_VIEWER_STDOUT_WORLD, ierr) >> > call PetscPrintf(PETSC_COMM_WORLD, "WIDE matrix:\n", ierr) >> > call MatView(WIDE, PETSC_VIEWER_STDOUT_WORLD, ierr) >> > >> > ! Clean up >> > call MatDestroy(COMPACT, ierr) >> > call MatDestroy(WIDE, ierr) >> > >> > ! Finalize PETSc >> > call PetscFinalize(ierr) >> > >> > end program petsc_matrix_example >> > >> > >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Sep 26 16:09:22 2024 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 26 Sep 2024 17:09:22 -0400 Subject: [petsc-users] Ghost particles for DMSWARM (or similar) In-Reply-To: <681E96BD-62A4-4566-A13E-E034B2F19D54@us.es> References: <8FBAC7A5-B6AE-4B21-8FEB-52BE1C04A265@us.es> <1B9B1277-9566-444C-9DA8-7ED17684FE01@us.es> <24337E11-33D2-4FFC-89E2-12520AD487FF@us.es> <681E96BD-62A4-4566-A13E-E034B2F19D54@us.es> Message-ID: On Thu, Sep 26, 2024 at 11:20?AM MIGUEL MOLINOS PEREZ wrote: > Thank you Matt. > > Okey, let me have a careful look to the DMSwarmMigrate_Push_Basic implementation > to see if there is some workaround. > > The idea of adding new particles is interesting. However, in that case, we > need to initialize the new (ghost) particles using the fields of the > ?real? particle, right? This can be done using something like: > > VecGhostUpdateBegin (Vec globalout,InsertMode ADD_VALUES , ScatterMode SCATTER_REVERSE );VecGhostUpdateEnd (Vec globalout,InsertMode ADD_VALUES , ScatterMode SCATTER_REVERSE ); > > for the particle fields (?). > I think we can just copy from the local particle. For example, suppose I decide that particle 1 should go to rank 5, 12, and 27. Then I first set p1.rank = 5, then I add two new particles with the same values as particle 1, but with rank = 12 and 27. Then when I call migrate, it will move these three particles to the correct processes, and delete the original particles and the copies from the local set. Thanks, Matt > Thanks, > Miguel > > > On Sep 26, 2024, at 3:53?PM, Matthew Knepley wrote: > > On Thu, Sep 26, 2024 at 6:31?AM MIGUEL MOLINOS PEREZ > wrote: > >> Hi Matt et al, >> >> I?ve been working on the scheme that you proposed to create ghost >> particles (atoms in my case), and it works! With a couple of caveats: >> -1? In general the overlap particles will be migrate from their own rank >> to more than one neighbor rank, this is specially relevant for those >> located close to the corners. Therefore, you'll need to call DMSwarmMigrate >> several times (27 times for 3D cells), during the migration process. >> > > That is terrible. Let's just fix DMSwarmMigrate to have a mode that sends > the particle to all overlapping neighbors at once. It can't be that hard. > > >> -2? You need to set DMSWARM_MIGRATE_BASIC. Otherwise the proposed >> algorithm will not work at all! >> > > Oh, I should have thought of that. Sorry. > > I can help code up that extension. Can you take a quick look at the BASIC > code? Right now, we just use the rank attached to the particle > to send it. We could have an arrays of ranks, but that seems crazy, and > would blow up particle storage. How about just adding new particles > with the other ranks right before migration? > > Thanks, > > Matt > > >> Hope this helps to other folks! >> >> I have a follow-up question about periodic bcc on this context, should I >> open a new thread of keep posting here? >> >> Thanks, >> Miguel >> >> On Aug 7, 2024, at 4:22?AM, MIGUEL MOLINOS PEREZ wrote: >> >> Thanks Matt, I think I'll start by making a small program as a proof of >> concept. Then, if it works I'll implement it in my code and I'll be happy >> to share it too :-) >> >> Miguel >> >> On Aug 4, 2024, at 3:30?AM, Matthew Knepley wrote: >> >> On Fri, Aug 2, 2024 at 7:15?PM MIGUEL MOLINOS PEREZ >> wrote: >> >>> Thanks again Matt, that makes a lot more sense !! >>> >>> Just to check that we are on the same page. You are saying: >>> >>> 1. create a field define a field called "owner rank" for each particle. >>> >>> 2. Identify the phantom particles and modify the internal variable >>> defined by the DMSwarmField_rank variable. >>> >>> 3. Call DMSwarmMigrate(*,PETSC_FALSE), do the calculations using the new >>> local vector including the ghost particles. >>> >>> 4. Then, once the calculations are done, rename the DMSwarmField_rank >>> variable using the "owner rank" variable and call >>> DMSwarmMigrate(*,PETSC_FALSE) once again. >>> >> >> I don't think we need this last step. We can just remove those ghost >> particles for the next step I think. >> >> Thanks, >> >> Matt >> >> >>> Thank you, >>> Miguel >>> >>> >>> On Aug 2, 2024, at 5:33?PM, Matthew Knepley wrote: >>> >>> On Fri, Aug 2, 2024 at 11:15?AM MIGUEL MOLINOS PEREZ >>> wrote: >>> >>>> Thank you Matt for your time, >>>> >>>> What you describe seems to me the ideal approach. >>>> >>>> 1) Add a particle field 'ghost' that identifies ghost vs owned >>>> particles. I think it needs options OWNED, OVERLAP, and GHOST >>>> >>>> This means, locally, I need to allocate Nlocal + ghost particles >>>> (duplicated) for my model? >>>> >>> >>> I would do it another way. I would allocate the particles with no >>> overlap and set them up. Then I would identify the halo particles, mark >>> them as OVERLAP, call DMSwarmMigrate(), and mark the migrated particles as >>> GHOST, then unmark the OVERLAP particles. Shoot! That marking will not work >>> since we cannot tell the difference between particles we received and >>> particles we sent. Okay, instead of the `ghost` field we need an `owner >>> rank` field. So then we >>> >>> 1) Setup the non-overlapping particles >>> >>> 2) Identify the halo particles >>> >>> 3) Change the `rank`, but not the `owner rank` >>> >>> 4) Call DMSwarmMigrate() >>> >>> Now we can identify ghost particles by the `owner rank` >>> >>> >>>> If that so, how to do the communication between the ghost particles >>>> living in the rank i and their ?real? counterpart in the rank j. >>>> >>>> Algo, as an alternative, what about: >>>> 1) Use an IS tag which contains, for each rank, a list of the global >>>> index of the neighbors particles outside of the rank. >>>> 2) Use VecCreateGhost to create a new vector which contains extra >>>> local space for the ghost components of the vector. >>>> 3) Use VecScatterCreate, VecScatterBegin, and VecScatterEnd to do the >>>> transference of data between a vector obtained with >>>> DMSwarmCreateGlobalVectorFromField >>>> 4) Do necessary computations using the vectors created with >>>> VecCreateGhost. >>>> >>> >>> This is essentially what Migrate() does. I was trying to reuse the code. >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Thanks, >>>> Miguel >>>> >>>> On Aug 2, 2024, at 8:58?AM, Matthew Knepley wrote: >>>> >>>> On Thu, Aug 1, 2024 at 4:40?PM MIGUEL MOLINOS PEREZ >>>> wrote: >>>> >>>>> This Message Is From an External Sender >>>>> This message came from outside your organization. >>>>> >>>>> >>>>> Dear all, >>>>> >>>>> I am implementing a Molecular Dynamics (MD) code using the DMSWARM interface. In the MD simulations we evaluate on each particle (atoms) some kind of scalar functional using data from the neighbouring atoms. My problem lies in the parallel implementation of the model, because sometimes, some of these neighbours lie on a different processor. >>>>> >>>>> This is usually solved by using ghost particles. A similar approach (with nodes instead) is already implemented for other PETSc mesh structures like DMPlexConstructGhostCells. Unfortunately, I don't see this kind of constructs for DMSWARM. Am I missing something? >>>>> >>>>> I this could be done by applying a buffer region by exploiting the background DMDA mesh that I already use to do domain decomposition. Then using the buffer region of each cell to locate the ghost particles and finally using VecCreateGhost. Is this feasible? Or is there an easier approach using other PETSc functions. >>>>> >>>>> >>>> This is feasible, but it would be good to develop a set of best >>>> practices, since we have been mainly focused on the case of non-redundant >>>> particles. Here is how I think I would do what you want. >>>> >>>> 1) Add a particle field 'ghost' that identifies ghost vs owned >>>> particles. I think it needs options OWNED, OVERLAP, and GHOST >>>> >>>> 2) At some interval identify particles that should be sent to other >>>> processes as ghosts. I would call these "overlap particles". The >>>> determination >>>> seems application specific, so I would leave this determination to >>>> the user right now. We do two things to these particles >>>> >>>> a) Mark chosen particles as OVERLAP >>>> >>>> b) Change rank to process we are sending to >>>> >>>> 3) Call DMSwarmMigrate with PETSC_FALSE for the particle deletion flag >>>> >>>> 4) Mark OVERLAP particles as GHOST when they arrive >>>> >>>> There is one problem in the above algorithm. It does not allow sending >>>> particles to multiple ranks. We would have to do this >>>> in phases right now, or make a small adjustment to the interface >>>> allowing replication of particles when a set of ranks is specified. >>>> >>>> THanks, >>>> >>>> Matt >>>> >>>> >>>>> Thank you, >>>>> Miguel >>>>> >>>>> >>>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aikPEw2N9TJ1IX4nFiTcOEsUOqcmEW2AwEUjS3Pw-WhS71aoiH_YonU7jU5aHVRNC-CqA7OIsJSNgtTo5Ip8$ >>>> >>>> >>>> >>>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aikPEw2N9TJ1IX4nFiTcOEsUOqcmEW2AwEUjS3Pw-WhS71aoiH_YonU7jU5aHVRNC-CqA7OIsJSNgtTo5Ip8$ >>> >>> >>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aikPEw2N9TJ1IX4nFiTcOEsUOqcmEW2AwEUjS3Pw-WhS71aoiH_YonU7jU5aHVRNC-CqA7OIsJSNgtTo5Ip8$ >> >> >> >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aikPEw2N9TJ1IX4nFiTcOEsUOqcmEW2AwEUjS3Pw-WhS71aoiH_YonU7jU5aHVRNC-CqA7OIsJSNgtTo5Ip8$ > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aikPEw2N9TJ1IX4nFiTcOEsUOqcmEW2AwEUjS3Pw-WhS71aoiH_YonU7jU5aHVRNC-CqA7OIsJSNgtTo5Ip8$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mmolinos at us.es Thu Sep 26 18:18:40 2024 From: mmolinos at us.es (MIGUEL MOLINOS PEREZ) Date: Thu, 26 Sep 2024 23:18:40 +0000 Subject: [petsc-users] Ghost particles for DMSWARM (or similar) In-Reply-To: References: <8FBAC7A5-B6AE-4B21-8FEB-52BE1C04A265@us.es> <1B9B1277-9566-444C-9DA8-7ED17684FE01@us.es> <24337E11-33D2-4FFC-89E2-12520AD487FF@us.es> <681E96BD-62A4-4566-A13E-E034B2F19D54@us.es> Message-ID: I see, you mean: Create the ghost particles at the local cell with the same properties as particle 1 (duplicate the original particle) but different value DMSwarmField_rank. Then, call DMSwarmMigrate(*,PETSC_FALSE) so we do the migration and delete the local copies of the particle 1. Right? Thanks, Miguel On Sep 26, 2024, at 11:09?PM, Matthew Knepley wrote: On Thu, Sep 26, 2024 at 11:20?AM MIGUEL MOLINOS PEREZ > wrote: Thank you Matt. Okey, let me have a careful look to the DMSwarmMigrate_Push_Basic implementation to see if there is some workaround. The idea of adding new particles is interesting. However, in that case, we need to initialize the new (ghost) particles using the fields of the ?real? particle, right? This can be done using something like: VecGhostUpdateBegin(Vec globalout,InsertMode ADD_VALUES, ScatterMode SCATTER_REVERSE); VecGhostUpdateEnd(Vec globalout,InsertMode ADD_VALUES, ScatterMode SCATTER_REVERSE); for the particle fields (?). I think we can just copy from the local particle. For example, suppose I decide that particle 1 should go to rank 5, 12, and 27. Then I first set p1.rank = 5, then I add two new particles with the same values as particle 1, but with rank = 12 and 27. Then when I call migrate, it will move these three particles to the correct processes, and delete the original particles and the copies from the local set. Thanks, Matt Thanks, Miguel On Sep 26, 2024, at 3:53?PM, Matthew Knepley > wrote: On Thu, Sep 26, 2024 at 6:31?AM MIGUEL MOLINOS PEREZ > wrote: Hi Matt et al, I?ve been working on the scheme that you proposed to create ghost particles (atoms in my case), and it works! With a couple of caveats: -1? In general the overlap particles will be migrate from their own rank to more than one neighbor rank, this is specially relevant for those located close to the corners. Therefore, you'll need to call DMSwarmMigrate several times (27 times for 3D cells), during the migration process. That is terrible. Let's just fix DMSwarmMigrate to have a mode that sends the particle to all overlapping neighbors at once. It can't be that hard. -2? You need to set DMSWARM_MIGRATE_BASIC. Otherwise the proposed algorithm will not work at all! Oh, I should have thought of that. Sorry. I can help code up that extension. Can you take a quick look at the BASIC code? Right now, we just use the rank attached to the particle to send it. We could have an arrays of ranks, but that seems crazy, and would blow up particle storage. How about just adding new particles with the other ranks right before migration? Thanks, Matt Hope this helps to other folks! I have a follow-up question about periodic bcc on this context, should I open a new thread of keep posting here? Thanks, Miguel On Aug 7, 2024, at 4:22?AM, MIGUEL MOLINOS PEREZ > wrote: Thanks Matt, I think I'll start by making a small program as a proof of concept. Then, if it works I'll implement it in my code and I'll be happy to share it too :-) Miguel On Aug 4, 2024, at 3:30?AM, Matthew Knepley > wrote: On Fri, Aug 2, 2024 at 7:15?PM MIGUEL MOLINOS PEREZ > wrote: Thanks again Matt, that makes a lot more sense !! Just to check that we are on the same page. You are saying: 1. create a field define a field called "owner rank" for each particle. 2. Identify the phantom particles and modify the internal variable defined by the DMSwarmField_rank variable. 3. Call DMSwarmMigrate(*,PETSC_FALSE), do the calculations using the new local vector including the ghost particles. 4. Then, once the calculations are done, rename the DMSwarmField_rank variable using the "owner rank" variable and call DMSwarmMigrate(*,PETSC_FALSE) once again. I don't think we need this last step. We can just remove those ghost particles for the next step I think. Thanks, Matt Thank you, Miguel On Aug 2, 2024, at 5:33?PM, Matthew Knepley > wrote: On Fri, Aug 2, 2024 at 11:15?AM MIGUEL MOLINOS PEREZ > wrote: Thank you Matt for your time, What you describe seems to me the ideal approach. 1) Add a particle field 'ghost' that identifies ghost vs owned particles. I think it needs options OWNED, OVERLAP, and GHOST This means, locally, I need to allocate Nlocal + ghost particles (duplicated) for my model? I would do it another way. I would allocate the particles with no overlap and set them up. Then I would identify the halo particles, mark them as OVERLAP, call DMSwarmMigrate(), and mark the migrated particles as GHOST, then unmark the OVERLAP particles. Shoot! That marking will not work since we cannot tell the difference between particles we received and particles we sent. Okay, instead of the `ghost` field we need an `owner rank` field. So then we 1) Setup the non-overlapping particles 2) Identify the halo particles 3) Change the `rank`, but not the `owner rank` 4) Call DMSwarmMigrate() Now we can identify ghost particles by the `owner rank` If that so, how to do the communication between the ghost particles living in the rank i and their ?real? counterpart in the rank j. Algo, as an alternative, what about: 1) Use an IS tag which contains, for each rank, a list of the global index of the neighbors particles outside of the rank. 2) Use VecCreateGhost to create a new vector which contains extra local space for the ghost components of the vector. 3) Use VecScatterCreate, VecScatterBegin, and VecScatterEnd to do the transference of data between a vector obtained with DMSwarmCreateGlobalVectorFromField 4) Do necessary computations using the vectors created with VecCreateGhost. This is essentially what Migrate() does. I was trying to reuse the code. Thanks, Matt Thanks, Miguel On Aug 2, 2024, at 8:58?AM, Matthew Knepley > wrote: On Thu, Aug 1, 2024 at 4:40?PM MIGUEL MOLINOS PEREZ > wrote: This Message Is From an External Sender This message came from outside your organization. Dear all, I am implementing a Molecular Dynamics (MD) code using the DMSWARM interface. In the MD simulations we evaluate on each particle (atoms) some kind of scalar functional using data from the neighbouring atoms. My problem lies in the parallel implementation of the model, because sometimes, some of these neighbours lie on a different processor. This is usually solved by using ghost particles. A similar approach (with nodes instead) is already implemented for other PETSc mesh structures like DMPlexConstructGhostCells. Unfortunately, I don't see this kind of constructs for DMSWARM. Am I missing something? I this could be done by applying a buffer region by exploiting the background DMDA mesh that I already use to do domain decomposition. Then using the buffer region of each cell to locate the ghost particles and finally using VecCreateGhost. Is this feasible? Or is there an easier approach using other PETSc functions. This is feasible, but it would be good to develop a set of best practices, since we have been mainly focused on the case of non-redundant particles. Here is how I think I would do what you want. 1) Add a particle field 'ghost' that identifies ghost vs owned particles. I think it needs options OWNED, OVERLAP, and GHOST 2) At some interval identify particles that should be sent to other processes as ghosts. I would call these "overlap particles". The determination seems application specific, so I would leave this determination to the user right now. We do two things to these particles a) Mark chosen particles as OVERLAP b) Change rank to process we are sending to 3) Call DMSwarmMigrate with PETSC_FALSE for the particle deletion flag 4) Mark OVERLAP particles as GHOST when they arrive There is one problem in the above algorithm. It does not allow sending particles to multiple ranks. We would have to do this in phases right now, or make a small adjustment to the interface allowing replication of particles when a set of ranks is specified. THanks, Matt Thank you, Miguel -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cf8sHFJMm5zy0bZUF7SbP0eWYkhzgoC3ir3qwyj-sXsONCncKGF8kTYtKjs-yQRJxH2mInp4TCnA8ow6kWrA9w$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cf8sHFJMm5zy0bZUF7SbP0eWYkhzgoC3ir3qwyj-sXsONCncKGF8kTYtKjs-yQRJxH2mInp4TCnA8ow6kWrA9w$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cf8sHFJMm5zy0bZUF7SbP0eWYkhzgoC3ir3qwyj-sXsONCncKGF8kTYtKjs-yQRJxH2mInp4TCnA8ow6kWrA9w$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cf8sHFJMm5zy0bZUF7SbP0eWYkhzgoC3ir3qwyj-sXsONCncKGF8kTYtKjs-yQRJxH2mInp4TCnA8ow6kWrA9w$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cf8sHFJMm5zy0bZUF7SbP0eWYkhzgoC3ir3qwyj-sXsONCncKGF8kTYtKjs-yQRJxH2mInp4TCnA8ow6kWrA9w$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Sep 26 20:44:39 2024 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 26 Sep 2024 21:44:39 -0400 Subject: [petsc-users] Ghost particles for DMSWARM (or similar) In-Reply-To: References: <8FBAC7A5-B6AE-4B21-8FEB-52BE1C04A265@us.es> <1B9B1277-9566-444C-9DA8-7ED17684FE01@us.es> <24337E11-33D2-4FFC-89E2-12520AD487FF@us.es> <681E96BD-62A4-4566-A13E-E034B2F19D54@us.es> Message-ID: On Thu, Sep 26, 2024 at 7:18?PM MIGUEL MOLINOS PEREZ wrote: > I see, you mean: > > Create the ghost particles at the local cell with the same properties as > particle 1 (duplicate the original particle) but different value > DMSwarmField_rank. Then, call DMSwarmMigrate(*,PETSC_FALSE) so we do the > migration and delete the local copies of the particle 1. Right? > Yep. I think it will work, from what I know about BASIC. Thanks, Matt > Thanks, > Miguel > > On Sep 26, 2024, at 11:09?PM, Matthew Knepley wrote: > > On Thu, Sep 26, 2024 at 11:20?AM MIGUEL MOLINOS PEREZ > wrote: > >> Thank you Matt. >> >> Okey, let me have a careful look to the DMSwarmMigrate_Push_Basic implementation >> to see if there is some workaround. >> >> The idea of adding new particles is interesting. However, in that case, >> we need to initialize the new (ghost) particles using the fields of the >> ?real? particle, right? This can be done using something like: >> >> VecGhostUpdateBegin (Vec globalout,InsertMode ADD_VALUES , ScatterMode SCATTER_REVERSE );VecGhostUpdateEnd (Vec globalout,InsertMode ADD_VALUES , ScatterMode SCATTER_REVERSE ); >> >> for the particle fields (?). >> > > I think we can just copy from the local particle. For example, suppose I > decide that particle 1 should go to rank 5, 12, and 27. Then > I first set p1.rank = 5, then I add two new particles with the same values > as particle 1, but with rank = 12 and 27. Then when I call migrate, it will > move these three particles to the correct processes, and delete the > original particles and the copies from the local set. > > Thanks, > > Matt > > >> Thanks, >> Miguel >> >> >> On Sep 26, 2024, at 3:53?PM, Matthew Knepley wrote: >> >> On Thu, Sep 26, 2024 at 6:31?AM MIGUEL MOLINOS PEREZ >> wrote: >> >>> Hi Matt et al, >>> >>> I?ve been working on the scheme that you proposed to create ghost >>> particles (atoms in my case), and it works! With a couple of caveats: >>> -1? In general the overlap particles will be migrate from their own rank >>> to more than one neighbor rank, this is specially relevant for those >>> located close to the corners. Therefore, you'll need to call DMSwarmMigrate >>> several times (27 times for 3D cells), during the migration process. >>> >> >> That is terrible. Let's just fix DMSwarmMigrate to have a mode that sends >> the particle to all overlapping neighbors at once. It can't be that hard. >> >> >>> -2? You need to set DMSWARM_MIGRATE_BASIC. Otherwise the proposed >>> algorithm will not work at all! >>> >> >> Oh, I should have thought of that. Sorry. >> >> I can help code up that extension. Can you take a quick look at the BASIC >> code? Right now, we just use the rank attached to the particle >> to send it. We could have an arrays of ranks, but that seems crazy, and >> would blow up particle storage. How about just adding new particles >> with the other ranks right before migration? >> >> Thanks, >> >> Matt >> >> >>> Hope this helps to other folks! >>> >>> I have a follow-up question about periodic bcc on this context, should I >>> open a new thread of keep posting here? >>> >>> Thanks, >>> Miguel >>> >>> On Aug 7, 2024, at 4:22?AM, MIGUEL MOLINOS PEREZ wrote: >>> >>> Thanks Matt, I think I'll start by making a small program as a proof of >>> concept. Then, if it works I'll implement it in my code and I'll be happy >>> to share it too :-) >>> >>> Miguel >>> >>> On Aug 4, 2024, at 3:30?AM, Matthew Knepley wrote: >>> >>> On Fri, Aug 2, 2024 at 7:15?PM MIGUEL MOLINOS PEREZ >>> wrote: >>> >>>> Thanks again Matt, that makes a lot more sense !! >>>> >>>> Just to check that we are on the same page. You are saying: >>>> >>>> 1. create a field define a field called "owner rank" for each particle. >>>> >>>> 2. Identify the phantom particles and modify the internal variable >>>> defined by the DMSwarmField_rank variable. >>>> >>>> 3. Call DMSwarmMigrate(*,PETSC_FALSE), do the calculations using the >>>> new local vector including the ghost particles. >>>> >>>> 4. Then, once the calculations are done, rename the DMSwarmField_rank >>>> variable using the "owner rank" variable and call >>>> DMSwarmMigrate(*,PETSC_FALSE) once again. >>>> >>> >>> I don't think we need this last step. We can just remove those ghost >>> particles for the next step I think. >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Thank you, >>>> Miguel >>>> >>>> >>>> On Aug 2, 2024, at 5:33?PM, Matthew Knepley wrote: >>>> >>>> On Fri, Aug 2, 2024 at 11:15?AM MIGUEL MOLINOS PEREZ >>>> wrote: >>>> >>>>> Thank you Matt for your time, >>>>> >>>>> What you describe seems to me the ideal approach. >>>>> >>>>> 1) Add a particle field 'ghost' that identifies ghost vs owned >>>>> particles. I think it needs options OWNED, OVERLAP, and GHOST >>>>> >>>>> This means, locally, I need to allocate Nlocal + ghost particles >>>>> (duplicated) for my model? >>>>> >>>> >>>> I would do it another way. I would allocate the particles with no >>>> overlap and set them up. Then I would identify the halo particles, mark >>>> them as OVERLAP, call DMSwarmMigrate(), and mark the migrated particles as >>>> GHOST, then unmark the OVERLAP particles. Shoot! That marking will not work >>>> since we cannot tell the difference between particles we received and >>>> particles we sent. Okay, instead of the `ghost` field we need an `owner >>>> rank` field. So then we >>>> >>>> 1) Setup the non-overlapping particles >>>> >>>> 2) Identify the halo particles >>>> >>>> 3) Change the `rank`, but not the `owner rank` >>>> >>>> 4) Call DMSwarmMigrate() >>>> >>>> Now we can identify ghost particles by the `owner rank` >>>> >>>> >>>>> If that so, how to do the communication between the ghost particles >>>>> living in the rank i and their ?real? counterpart in the rank j. >>>>> >>>>> Algo, as an alternative, what about: >>>>> 1) Use an IS tag which contains, for each rank, a list of the global >>>>> index of the neighbors particles outside of the rank. >>>>> 2) Use VecCreateGhost to create a new vector which contains extra >>>>> local space for the ghost components of the vector. >>>>> 3) Use VecScatterCreate, VecScatterBegin, and VecScatterEnd to do the >>>>> transference of data between a vector obtained with >>>>> DMSwarmCreateGlobalVectorFromField >>>>> 4) Do necessary computations using the vectors created with >>>>> VecCreateGhost. >>>>> >>>> >>>> This is essentially what Migrate() does. I was trying to reuse the code. >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> Thanks, >>>>> Miguel >>>>> >>>>> On Aug 2, 2024, at 8:58?AM, Matthew Knepley wrote: >>>>> >>>>> On Thu, Aug 1, 2024 at 4:40?PM MIGUEL MOLINOS PEREZ >>>>> wrote: >>>>> >>>>>> This Message Is From an External Sender >>>>>> This message came from outside your organization. >>>>>> >>>>>> >>>>>> Dear all, >>>>>> >>>>>> I am implementing a Molecular Dynamics (MD) code using the DMSWARM interface. In the MD simulations we evaluate on each particle (atoms) some kind of scalar functional using data from the neighbouring atoms. My problem lies in the parallel implementation of the model, because sometimes, some of these neighbours lie on a different processor. >>>>>> >>>>>> This is usually solved by using ghost particles. A similar approach (with nodes instead) is already implemented for other PETSc mesh structures like DMPlexConstructGhostCells. Unfortunately, I don't see this kind of constructs for DMSWARM. Am I missing something? >>>>>> >>>>>> I this could be done by applying a buffer region by exploiting the background DMDA mesh that I already use to do domain decomposition. Then using the buffer region of each cell to locate the ghost particles and finally using VecCreateGhost. Is this feasible? Or is there an easier approach using other PETSc functions. >>>>>> >>>>>> >>>>> This is feasible, but it would be good to develop a set of best >>>>> practices, since we have been mainly focused on the case of non-redundant >>>>> particles. Here is how I think I would do what you want. >>>>> >>>>> 1) Add a particle field 'ghost' that identifies ghost vs owned >>>>> particles. I think it needs options OWNED, OVERLAP, and GHOST >>>>> >>>>> 2) At some interval identify particles that should be sent to other >>>>> processes as ghosts. I would call these "overlap particles". The >>>>> determination >>>>> seems application specific, so I would leave this determination to >>>>> the user right now. We do two things to these particles >>>>> >>>>> a) Mark chosen particles as OVERLAP >>>>> >>>>> b) Change rank to process we are sending to >>>>> >>>>> 3) Call DMSwarmMigrate with PETSC_FALSE for the particle deletion flag >>>>> >>>>> 4) Mark OVERLAP particles as GHOST when they arrive >>>>> >>>>> There is one problem in the above algorithm. It does not allow sending >>>>> particles to multiple ranks. We would have to do this >>>>> in phases right now, or make a small adjustment to the interface >>>>> allowing replication of particles when a set of ranks is specified. >>>>> >>>>> THanks, >>>>> >>>>> Matt >>>>> >>>>> >>>>>> Thank you, >>>>>> Miguel >>>>>> >>>>>> >>>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!emloiQzBtgzgIIVq1midB7Rd9bkkr7wwUgGr8mzcWk2AzIkn7e_2Rr5Nkxd2ysaNUXWFp1jLd98xptrlWQeo$ >>>>> >>>>> >>>>> >>>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!emloiQzBtgzgIIVq1midB7Rd9bkkr7wwUgGr8mzcWk2AzIkn7e_2Rr5Nkxd2ysaNUXWFp1jLd98xptrlWQeo$ >>>> >>>> >>>> >>>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!emloiQzBtgzgIIVq1midB7Rd9bkkr7wwUgGr8mzcWk2AzIkn7e_2Rr5Nkxd2ysaNUXWFp1jLd98xptrlWQeo$ >>> >>> >>> >>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!emloiQzBtgzgIIVq1midB7Rd9bkkr7wwUgGr8mzcWk2AzIkn7e_2Rr5Nkxd2ysaNUXWFp1jLd98xptrlWQeo$ >> >> >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!emloiQzBtgzgIIVq1midB7Rd9bkkr7wwUgGr8mzcWk2AzIkn7e_2Rr5Nkxd2ysaNUXWFp1jLd98xptrlWQeo$ > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!emloiQzBtgzgIIVq1midB7Rd9bkkr7wwUgGr8mzcWk2AzIkn7e_2Rr5Nkxd2ysaNUXWFp1jLd98xptrlWQeo$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mmolinos at us.es Fri Sep 27 03:51:06 2024 From: mmolinos at us.es (MIGUEL MOLINOS PEREZ) Date: Fri, 27 Sep 2024 08:51:06 +0000 Subject: [petsc-users] Ghost particles for DMSWARM (or similar) In-Reply-To: References: <8FBAC7A5-B6AE-4B21-8FEB-52BE1C04A265@us.es> <1B9B1277-9566-444C-9DA8-7ED17684FE01@us.es> <24337E11-33D2-4FFC-89E2-12520AD487FF@us.es> <681E96BD-62A4-4566-A13E-E034B2F19D54@us.es> Message-ID: <7698089E-0909-429F-9E89-9D1AD636ACBF@us.es> Thank you Matt, let me give it try. Miguel On Sep 27, 2024, at 3:44?AM, Matthew Knepley wrote: On Thu, Sep 26, 2024 at 7:18?PM MIGUEL MOLINOS PEREZ > wrote: I see, you mean: Create the ghost particles at the local cell with the same properties as particle 1 (duplicate the original particle) but different value DMSwarmField_rank. Then, call DMSwarmMigrate(*,PETSC_FALSE) so we do the migration and delete the local copies of the particle 1. Right? Yep. I think it will work, from what I know about BASIC. Thanks, Matt Thanks, Miguel On Sep 26, 2024, at 11:09?PM, Matthew Knepley > wrote: On Thu, Sep 26, 2024 at 11:20?AM MIGUEL MOLINOS PEREZ > wrote: Thank you Matt. Okey, let me have a careful look to the DMSwarmMigrate_Push_Basic implementation to see if there is some workaround. The idea of adding new particles is interesting. However, in that case, we need to initialize the new (ghost) particles using the fields of the ?real? particle, right? This can be done using something like: VecGhostUpdateBegin(Vec globalout,InsertMode ADD_VALUES, ScatterMode SCATTER_REVERSE); VecGhostUpdateEnd(Vec globalout,InsertMode ADD_VALUES, ScatterMode SCATTER_REVERSE); for the particle fields (?). I think we can just copy from the local particle. For example, suppose I decide that particle 1 should go to rank 5, 12, and 27. Then I first set p1.rank = 5, then I add two new particles with the same values as particle 1, but with rank = 12 and 27. Then when I call migrate, it will move these three particles to the correct processes, and delete the original particles and the copies from the local set. Thanks, Matt Thanks, Miguel On Sep 26, 2024, at 3:53?PM, Matthew Knepley > wrote: On Thu, Sep 26, 2024 at 6:31?AM MIGUEL MOLINOS PEREZ > wrote: Hi Matt et al, I?ve been working on the scheme that you proposed to create ghost particles (atoms in my case), and it works! With a couple of caveats: -1? In general the overlap particles will be migrate from their own rank to more than one neighbor rank, this is specially relevant for those located close to the corners. Therefore, you'll need to call DMSwarmMigrate several times (27 times for 3D cells), during the migration process. That is terrible. Let's just fix DMSwarmMigrate to have a mode that sends the particle to all overlapping neighbors at once. It can't be that hard. -2? You need to set DMSWARM_MIGRATE_BASIC. Otherwise the proposed algorithm will not work at all! Oh, I should have thought of that. Sorry. I can help code up that extension. Can you take a quick look at the BASIC code? Right now, we just use the rank attached to the particle to send it. We could have an arrays of ranks, but that seems crazy, and would blow up particle storage. How about just adding new particles with the other ranks right before migration? Thanks, Matt Hope this helps to other folks! I have a follow-up question about periodic bcc on this context, should I open a new thread of keep posting here? Thanks, Miguel On Aug 7, 2024, at 4:22?AM, MIGUEL MOLINOS PEREZ > wrote: Thanks Matt, I think I'll start by making a small program as a proof of concept. Then, if it works I'll implement it in my code and I'll be happy to share it too :-) Miguel On Aug 4, 2024, at 3:30?AM, Matthew Knepley > wrote: On Fri, Aug 2, 2024 at 7:15?PM MIGUEL MOLINOS PEREZ > wrote: Thanks again Matt, that makes a lot more sense !! Just to check that we are on the same page. You are saying: 1. create a field define a field called "owner rank" for each particle. 2. Identify the phantom particles and modify the internal variable defined by the DMSwarmField_rank variable. 3. Call DMSwarmMigrate(*,PETSC_FALSE), do the calculations using the new local vector including the ghost particles. 4. Then, once the calculations are done, rename the DMSwarmField_rank variable using the "owner rank" variable and call DMSwarmMigrate(*,PETSC_FALSE) once again. I don't think we need this last step. We can just remove those ghost particles for the next step I think. Thanks, Matt Thank you, Miguel On Aug 2, 2024, at 5:33?PM, Matthew Knepley > wrote: On Fri, Aug 2, 2024 at 11:15?AM MIGUEL MOLINOS PEREZ > wrote: Thank you Matt for your time, What you describe seems to me the ideal approach. 1) Add a particle field 'ghost' that identifies ghost vs owned particles. I think it needs options OWNED, OVERLAP, and GHOST This means, locally, I need to allocate Nlocal + ghost particles (duplicated) for my model? I would do it another way. I would allocate the particles with no overlap and set them up. Then I would identify the halo particles, mark them as OVERLAP, call DMSwarmMigrate(), and mark the migrated particles as GHOST, then unmark the OVERLAP particles. Shoot! That marking will not work since we cannot tell the difference between particles we received and particles we sent. Okay, instead of the `ghost` field we need an `owner rank` field. So then we 1) Setup the non-overlapping particles 2) Identify the halo particles 3) Change the `rank`, but not the `owner rank` 4) Call DMSwarmMigrate() Now we can identify ghost particles by the `owner rank` If that so, how to do the communication between the ghost particles living in the rank i and their ?real? counterpart in the rank j. Algo, as an alternative, what about: 1) Use an IS tag which contains, for each rank, a list of the global index of the neighbors particles outside of the rank. 2) Use VecCreateGhost to create a new vector which contains extra local space for the ghost components of the vector. 3) Use VecScatterCreate, VecScatterBegin, and VecScatterEnd to do the transference of data between a vector obtained with DMSwarmCreateGlobalVectorFromField 4) Do necessary computations using the vectors created with VecCreateGhost. This is essentially what Migrate() does. I was trying to reuse the code. Thanks, Matt Thanks, Miguel On Aug 2, 2024, at 8:58?AM, Matthew Knepley > wrote: On Thu, Aug 1, 2024 at 4:40?PM MIGUEL MOLINOS PEREZ > wrote: This Message Is From an External Sender This message came from outside your organization. Dear all, I am implementing a Molecular Dynamics (MD) code using the DMSWARM interface. In the MD simulations we evaluate on each particle (atoms) some kind of scalar functional using data from the neighbouring atoms. My problem lies in the parallel implementation of the model, because sometimes, some of these neighbours lie on a different processor. This is usually solved by using ghost particles. A similar approach (with nodes instead) is already implemented for other PETSc mesh structures like DMPlexConstructGhostCells. Unfortunately, I don't see this kind of constructs for DMSWARM. Am I missing something? I this could be done by applying a buffer region by exploiting the background DMDA mesh that I already use to do domain decomposition. Then using the buffer region of each cell to locate the ghost particles and finally using VecCreateGhost. Is this feasible? Or is there an easier approach using other PETSc functions. This is feasible, but it would be good to develop a set of best practices, since we have been mainly focused on the case of non-redundant particles. Here is how I think I would do what you want. 1) Add a particle field 'ghost' that identifies ghost vs owned particles. I think it needs options OWNED, OVERLAP, and GHOST 2) At some interval identify particles that should be sent to other processes as ghosts. I would call these "overlap particles". The determination seems application specific, so I would leave this determination to the user right now. We do two things to these particles a) Mark chosen particles as OVERLAP b) Change rank to process we are sending to 3) Call DMSwarmMigrate with PETSC_FALSE for the particle deletion flag 4) Mark OVERLAP particles as GHOST when they arrive There is one problem in the above algorithm. It does not allow sending particles to multiple ranks. We would have to do this in phases right now, or make a small adjustment to the interface allowing replication of particles when a set of ranks is specified. THanks, Matt Thank you, Miguel -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!dHNd2Wiqx529QMIoMUJ2N9KCSssD66koKRtx2-UcBvPo__o_PDSsijzmBTWcrfapO5ug6AnIaoy6lJrGfMiBWg$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!dHNd2Wiqx529QMIoMUJ2N9KCSssD66koKRtx2-UcBvPo__o_PDSsijzmBTWcrfapO5ug6AnIaoy6lJrGfMiBWg$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!dHNd2Wiqx529QMIoMUJ2N9KCSssD66koKRtx2-UcBvPo__o_PDSsijzmBTWcrfapO5ug6AnIaoy6lJrGfMiBWg$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!dHNd2Wiqx529QMIoMUJ2N9KCSssD66koKRtx2-UcBvPo__o_PDSsijzmBTWcrfapO5ug6AnIaoy6lJrGfMiBWg$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!dHNd2Wiqx529QMIoMUJ2N9KCSssD66koKRtx2-UcBvPo__o_PDSsijzmBTWcrfapO5ug6AnIaoy6lJrGfMiBWg$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!dHNd2Wiqx529QMIoMUJ2N9KCSssD66koKRtx2-UcBvPo__o_PDSsijzmBTWcrfapO5ug6AnIaoy6lJrGfMiBWg$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From boris.voinov at intel.com Fri Sep 27 06:24:01 2024 From: boris.voinov at intel.com (Voinov, Boris) Date: Fri, 27 Sep 2024 11:24:01 +0000 Subject: [petsc-users] Ask for help in tracking down bugs - setting #threads for pardiso block LU solver Message-ID: Hello, Could you please help me to figure out what's wrong with the way I'm trying to make mkl pardiso use the number of threads more than one while setting it as a sub PC solver for ASM preconditioner. Here is how I do this Loop over block PCs { PCSetType(subPc, PCLU); set subPC type PCFactorSetMatSolverType(subPc, MATSOLVERMKL_PARDISO); set solver for LU SetPardisoParams("pardiso"); set mkl related options including -mat_mkl_pardiso_65 equal to the desired #threads; return code after PetscOptionsSetValue is ok, so I presume the option is set correctly. PCSetFromOptions(subPc); } Then what I see under the debugger PCSetUp after a series of calls gets to MatLUFactorSymbolic_AIJMKL_PARDISO which in turn calls MatSetFromOptions_MKL_PARDISO(F, A) and there in these lines it seems to ignore what I've set for -mat_mkl_pardiso_65 and sets #threads to the default 1 PetscCall(PetscOptionsInt("-mat_mkl_pardiso_65", "Suggested number of threads to use within PARDISO", "None", threads, &threads, &flg)); if (flg) PetscSetMKL_PARDISOThreads((int)threads); This is about petsc-3.20. PS when I do this in a standalone code which reads a matrix and solves the linear system and set -mat_mkl_pardiso_65 in the command line it's all right but I need the number of threads to be set in other way in my code. Thank you and best regards, Boris ------------------------------------------------------------- Intel Ireland Limited (Branch) Collinstown Industrial Park, Leixlip, County Kildare, Ireland Registered Number: E902934 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Fri Sep 27 09:43:46 2024 From: dave.mayhem23 at gmail.com (Dave May) Date: Fri, 27 Sep 2024 07:43:46 -0700 Subject: [petsc-users] Ghost particles for DMSWARM (or similar) In-Reply-To: <7698089E-0909-429F-9E89-9D1AD636ACBF@us.es> References: <8FBAC7A5-B6AE-4B21-8FEB-52BE1C04A265@us.es> <1B9B1277-9566-444C-9DA8-7ED17684FE01@us.es> <24337E11-33D2-4FFC-89E2-12520AD487FF@us.es> <681E96BD-62A4-4566-A13E-E034B2F19D54@us.es> <7698089E-0909-429F-9E89-9D1AD636ACBF@us.es> Message-ID: Hi all, Sorry to be very late coming to this party. I had always intended for swarm to support ghost particles. The way I wanted to do this was via a new migrate type (DMSWARM_MIGRATE_GHOST) (or a better name - suggestions welcome) and a new migrate function. The new migrate function would call a user defined function, with a signature DMSwarmMigrateGhostSelect(DM swarm, void *user_context, PetscInt *nghost, PetscMPIInt *ranks[] PetscInt *pointids[]) DMSwarmMigrateGhostSelect() returns an array of all points indices which are to be ghosted, and the ranks where these ghost points should be sent. Then in the ghost migrate function one would use ranks[] to define the scatter pattern and then pack swarm points found in pointids[] into buffers and then send em. Removal of points (although unlikely required) could be achieved by sorting and removing dups from pointids[] and deleting indices from largest to smallest. Duplicate indices will likely appear in pointids[] in most ghost scenarios (eg eval an SPH kernel) and thus need to be handled if one is to remove sent points. Lastly, I would have a mechanism to "reset" the DMSwarm post ghost scatter to remove the scattered ghost points. As ghosted points would appear at the back of the array a reset would just mean setting the length of the valid points back to the value it had prior to the ghosting operation. I think it?s fairly straightforward to add support for this. Let me know if want to tackle it, or if you want me to code it up. Cheers, Dave On Fri, 27 Sept 2024 at 01:51, MIGUEL MOLINOS PEREZ wrote: > Thank you Matt, let me give it try. > > Miguel > > On Sep 27, 2024, at 3:44?AM, Matthew Knepley wrote: > > On Thu, Sep 26, 2024 at 7:18?PM MIGUEL MOLINOS PEREZ > wrote: > >> I see, you mean: >> >> Create the ghost particles at the local cell with the same properties as >> particle 1 (duplicate the original particle) but different value >> DMSwarmField_rank. Then, call DMSwarmMigrate(*,PETSC_FALSE) so we do the >> migration and delete the local copies of the particle 1. Right? >> > > Yep. I think it will work, from what I know about BASIC. > > Thanks, > > Matt > > >> Thanks, >> Miguel >> >> On Sep 26, 2024, at 11:09?PM, Matthew Knepley wrote: >> >> On Thu, Sep 26, 2024 at 11:20?AM MIGUEL MOLINOS PEREZ >> wrote: >> >>> Thank you Matt. >>> >>> Okey, let me have a careful look to the DMSwarmMigrate_Push_Basic implementation >>> to see if there is some workaround. >>> >>> The idea of adding new particles is interesting. However, in that case, >>> we need to initialize the new (ghost) particles using the fields of the >>> ?real? particle, right? This can be done using something like: >>> >>> VecGhostUpdateBegin (Vec globalout,InsertMode ADD_VALUES , ScatterMode SCATTER_REVERSE );VecGhostUpdateEnd (Vec globalout,InsertMode ADD_VALUES , ScatterMode SCATTER_REVERSE ); >>> >>> for the particle fields (?). >>> >> >> I think we can just copy from the local particle. For example, suppose I >> decide that particle 1 should go to rank 5, 12, and 27. Then >> I first set p1.rank = 5, then I add two new particles with the same >> values as particle 1, but with rank = 12 and 27. Then when I call migrate, >> it will move these three particles to the correct processes, and delete the >> original particles and the copies from the local set. >> >> Thanks, >> >> Matt >> >> >>> Thanks, >>> Miguel >>> >>> >>> On Sep 26, 2024, at 3:53?PM, Matthew Knepley wrote: >>> >>> On Thu, Sep 26, 2024 at 6:31?AM MIGUEL MOLINOS PEREZ >>> wrote: >>> >>>> Hi Matt et al, >>>> >>>> I?ve been working on the scheme that you proposed to create ghost >>>> particles (atoms in my case), and it works! With a couple of caveats: >>>> -1? In general the overlap particles will be migrate from their own >>>> rank to more than one neighbor rank, this is specially relevant for those >>>> located close to the corners. Therefore, you'll need to call DMSwarmMigrate >>>> several times (27 times for 3D cells), during the migration process. >>>> >>> >>> That is terrible. Let's just fix DMSwarmMigrate to have a mode that >>> sends the particle to all overlapping neighbors at once. It can't be that >>> hard. >>> >>> >>>> -2? You need to set DMSWARM_MIGRATE_BASIC. Otherwise the proposed >>>> algorithm will not work at all! >>>> >>> >>> Oh, I should have thought of that. Sorry. >>> >>> I can help code up that extension. Can you take a quick look at the >>> BASIC code? Right now, we just use the rank attached to the particle >>> to send it. We could have an arrays of ranks, but that seems crazy, and >>> would blow up particle storage. How about just adding new particles >>> with the other ranks right before migration? >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Hope this helps to other folks! >>>> >>>> I have a follow-up question about periodic bcc on this context, should >>>> I open a new thread of keep posting here? >>>> >>>> Thanks, >>>> Miguel >>>> >>>> On Aug 7, 2024, at 4:22?AM, MIGUEL MOLINOS PEREZ >>>> wrote: >>>> >>>> Thanks Matt, I think I'll start by making a small program as a proof of >>>> concept. Then, if it works I'll implement it in my code and I'll be happy >>>> to share it too :-) >>>> >>>> Miguel >>>> >>>> On Aug 4, 2024, at 3:30?AM, Matthew Knepley wrote: >>>> >>>> On Fri, Aug 2, 2024 at 7:15?PM MIGUEL MOLINOS PEREZ >>>> wrote: >>>> >>>>> Thanks again Matt, that makes a lot more sense !! >>>>> >>>>> Just to check that we are on the same page. You are saying: >>>>> >>>>> 1. create a field define a field called "owner rank" for each >>>>> particle. >>>>> >>>>> 2. Identify the phantom particles and modify the internal variable >>>>> defined by the DMSwarmField_rank variable. >>>>> >>>>> 3. Call DMSwarmMigrate(*,PETSC_FALSE), do the calculations using the >>>>> new local vector including the ghost particles. >>>>> >>>>> 4. Then, once the calculations are done, rename the DMSwarmField_rank >>>>> variable using the "owner rank" variable and call >>>>> DMSwarmMigrate(*,PETSC_FALSE) once again. >>>>> >>>> >>>> I don't think we need this last step. We can just remove those ghost >>>> particles for the next step I think. >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> Thank you, >>>>> Miguel >>>>> >>>>> >>>>> On Aug 2, 2024, at 5:33?PM, Matthew Knepley wrote: >>>>> >>>>> On Fri, Aug 2, 2024 at 11:15?AM MIGUEL MOLINOS PEREZ >>>>> wrote: >>>>> >>>>>> Thank you Matt for your time, >>>>>> >>>>>> What you describe seems to me the ideal approach. >>>>>> >>>>>> 1) Add a particle field 'ghost' that identifies ghost vs owned >>>>>> particles. I think it needs options OWNED, OVERLAP, and GHOST >>>>>> >>>>>> This means, locally, I need to allocate Nlocal + ghost particles >>>>>> (duplicated) for my model? >>>>>> >>>>> >>>>> I would do it another way. I would allocate the particles with no >>>>> overlap and set them up. Then I would identify the halo particles, mark >>>>> them as OVERLAP, call DMSwarmMigrate(), and mark the migrated particles as >>>>> GHOST, then unmark the OVERLAP particles. Shoot! That marking will not work >>>>> since we cannot tell the difference between particles we received and >>>>> particles we sent. Okay, instead of the `ghost` field we need an `owner >>>>> rank` field. So then we >>>>> >>>>> 1) Setup the non-overlapping particles >>>>> >>>>> 2) Identify the halo particles >>>>> >>>>> 3) Change the `rank`, but not the `owner rank` >>>>> >>>>> 4) Call DMSwarmMigrate() >>>>> >>>>> Now we can identify ghost particles by the `owner rank` >>>>> >>>>> >>>>>> If that so, how to do the communication between the ghost particles >>>>>> living in the rank i and their ?real? counterpart in the rank j. >>>>>> >>>>>> Algo, as an alternative, what about: >>>>>> 1) Use an IS tag which contains, for each rank, a list of the global >>>>>> index of the neighbors particles outside of the rank. >>>>>> 2) Use VecCreateGhost to create a new vector which contains extra >>>>>> local space for the ghost components of the vector. >>>>>> 3) Use VecScatterCreate, VecScatterBegin, and VecScatterEnd to do >>>>>> the transference of data between a vector obtained with >>>>>> DMSwarmCreateGlobalVectorFromField >>>>>> 4) Do necessary computations using the vectors created with >>>>>> VecCreateGhost. >>>>>> >>>>> >>>>> This is essentially what Migrate() does. I was trying to reuse the >>>>> code. >>>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>> >>>>>> Thanks, >>>>>> Miguel >>>>>> >>>>>> On Aug 2, 2024, at 8:58?AM, Matthew Knepley >>>>>> wrote: >>>>>> >>>>>> On Thu, Aug 1, 2024 at 4:40?PM MIGUEL MOLINOS PEREZ >>>>>> wrote: >>>>>> >>>>>>> This Message Is From an External Sender >>>>>>> This message came from outside your organization. >>>>>>> >>>>>>> >>>>>>> Dear all, >>>>>>> >>>>>>> I am implementing a Molecular Dynamics (MD) code using the DMSWARM interface. In the MD simulations we evaluate on each particle (atoms) some kind of scalar functional using data from the neighbouring atoms. My problem lies in the parallel implementation of the model, because sometimes, some of these neighbours lie on a different processor. >>>>>>> >>>>>>> This is usually solved by using ghost particles. A similar approach (with nodes instead) is already implemented for other PETSc mesh structures like DMPlexConstructGhostCells. Unfortunately, I don't see this kind of constructs for DMSWARM. Am I missing something? >>>>>>> >>>>>>> I this could be done by applying a buffer region by exploiting the background DMDA mesh that I already use to do domain decomposition. Then using the buffer region of each cell to locate the ghost particles and finally using VecCreateGhost. Is this feasible? Or is there an easier approach using other PETSc functions. >>>>>>> >>>>>>> >>>>>> This is feasible, but it would be good to develop a set of best >>>>>> practices, since we have been mainly focused on the case of non-redundant >>>>>> particles. Here is how I think I would do what you want. >>>>>> >>>>>> 1) Add a particle field 'ghost' that identifies ghost vs owned >>>>>> particles. I think it needs options OWNED, OVERLAP, and GHOST >>>>>> >>>>>> 2) At some interval identify particles that should be sent to other >>>>>> processes as ghosts. I would call these "overlap particles". The >>>>>> determination >>>>>> seems application specific, so I would leave this determination >>>>>> to the user right now. We do two things to these particles >>>>>> >>>>>> a) Mark chosen particles as OVERLAP >>>>>> >>>>>> b) Change rank to process we are sending to >>>>>> >>>>>> 3) Call DMSwarmMigrate with PETSC_FALSE for the particle deletion flag >>>>>> >>>>>> 4) Mark OVERLAP particles as GHOST when they arrive >>>>>> >>>>>> There is one problem in the above algorithm. It does not allow >>>>>> sending particles to multiple ranks. We would have to do this >>>>>> in phases right now, or make a small adjustment to the interface >>>>>> allowing replication of particles when a set of ranks is specified. >>>>>> >>>>>> THanks, >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>>> Thank you, >>>>>>> Miguel >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!ahalwdSnSASVIbituzwm7DPQIeQg567RGPKU8GED5W9YvP3djRN246zQb4w1L4k4HjAxjZvg9BpJJ5PRKd-J4uXmsEqA$ >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!ahalwdSnSASVIbituzwm7DPQIeQg567RGPKU8GED5W9YvP3djRN246zQb4w1L4k4HjAxjZvg9BpJJ5PRKd-J4uXmsEqA$ >>>>> >>>>> >>>>> >>>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!ahalwdSnSASVIbituzwm7DPQIeQg567RGPKU8GED5W9YvP3djRN246zQb4w1L4k4HjAxjZvg9BpJJ5PRKd-J4uXmsEqA$ >>>> >>>> >>>> >>>> >>>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!ahalwdSnSASVIbituzwm7DPQIeQg567RGPKU8GED5W9YvP3djRN246zQb4w1L4k4HjAxjZvg9BpJJ5PRKd-J4uXmsEqA$ >>> >>> >>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!ahalwdSnSASVIbituzwm7DPQIeQg567RGPKU8GED5W9YvP3djRN246zQb4w1L4k4HjAxjZvg9BpJJ5PRKd-J4uXmsEqA$ >> >> >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!ahalwdSnSASVIbituzwm7DPQIeQg567RGPKU8GED5W9YvP3djRN246zQb4w1L4k4HjAxjZvg9BpJJ5PRKd-J4uXmsEqA$ > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Fri Sep 27 10:20:34 2024 From: bsmith at petsc.dev (Barry Smith) Date: Fri, 27 Sep 2024 11:20:34 -0400 Subject: [petsc-users] Ask for help in tracking down bugs - setting #threads for pardiso block LU solver In-Reply-To: References: Message-ID: <2F21B31F-7F62-466A-A960-A97F9B07998A@petsc.dev> Boris, There is nothing obviously wrong with what you outline, so let's dive down into the code with the debugger and see what is happening. PetscSetMKL_PARDISOThreads() set from options is called when the command line option is provided, and this calls PETSC_EXTERN void PetscSetMKL_PARDISOThreads(int threads) { mkl_domain_set_num_threads(threads, MKL_DOMAIN_PARDISO); } so in the debugger can you check that mkl_domain_set_num_threads() gets called with your requested number of threads? BTW: Since you are hardwiring the use of Pardiso in the code with function calls you couldsimply call PetscSetMKL_PARDISOThreads() directly after your line of code PCFactorSetMatSolverType(subPc, MATSOLVERMKL_PARDISO); set solver for LU instead of feeding it in through the options database. Let us know how this turns out Barry > On Sep 27, 2024, at 7:24?AM, Voinov, Boris wrote: > > Hello, > > Could you please help me to figure out what?s wrong with the way I?m trying to make mkl pardiso use the number of threads more than one while setting it as a sub PC solver for ASM preconditioner. > Here is how I do this > Loop over block PCs { > PCSetType(subPc, PCLU); set subPC type > PCFactorSetMatSolverType(subPc, MATSOLVERMKL_PARDISO); set solver for LU > SetPardisoParams("pardiso"); set mkl related options including -mat_mkl_pardiso_65 equal to the desired #threads; return code after PetscOptionsSetValue is ok, so I presume the option is set correctly. > PCSetFromOptions(subPc); > } > > Then what I see under the debugger > PCSetUp after a series of calls gets to MatLUFactorSymbolic_AIJMKL_PARDISO which in turn calls MatSetFromOptions_MKL_PARDISO(F, A) and there > in these lines it seems to ignore what I?ve set for -mat_mkl_pardiso_65 and sets #threads to the default 1 > PetscCall(PetscOptionsInt("-mat_mkl_pardiso_65", "Suggested number of threads to use within PARDISO", "None", threads, &threads, &flg)); > if (flg) PetscSetMKL_PARDISOThreads((int)threads); > > This is about petsc-3.20. > PS when I do this in a standalone code which reads a matrix and solves the linear system and set -mat_mkl_pardiso_65 in the command line it?s all right but I need the number of threads to be set in other way in my code. > > Thank you and best regards, > Boris > > ------------------------------------------------------------- > Intel Ireland Limited (Branch) > Collinstown Industrial Park, Leixlip, County Kildare, Ireland > Registered Number: E902934 > > This e-mail and any attachments may contain confidential material for > the sole use of the intended recipient(s). Any review or distribution > by others is strictly prohibited. If you are not the intended > recipient, please contact the sender and delete all copies. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mmolinos at us.es Fri Sep 27 10:46:59 2024 From: mmolinos at us.es (MIGUEL MOLINOS PEREZ) Date: Fri, 27 Sep 2024 15:46:59 +0000 Subject: [petsc-users] Ghost particles for DMSWARM (or similar) In-Reply-To: References: <8FBAC7A5-B6AE-4B21-8FEB-52BE1C04A265@us.es> <1B9B1277-9566-444C-9DA8-7ED17684FE01@us.es> <24337E11-33D2-4FFC-89E2-12520AD487FF@us.es> <681E96BD-62A4-4566-A13E-E034B2F19D54@us.es> <7698089E-0909-429F-9E89-9D1AD636ACBF@us.es> Message-ID: Hi Dave: On Sep 27, 2024, at 4:43?PM, Dave May wrote: Hi all, Sorry to be very late coming to this party. Welcome to the party :) I had always intended for swarm to support ghost particles. The way I wanted to do this was via a new migrate type (DMSWARM_MIGRATE_GHOST) (or a better name - suggestions welcome) and a new migrate function. The new migrate function would call a user defined function, with a signature DMSwarmMigrateGhostSelect(DM swarm, void *user_context, PetscInt *nghost, PetscMPIInt *ranks[] PetscInt *pointids[]) DMSwarmMigrateGhostSelect() returns an array of all points indices which are to be ghosted, and the ranks where these ghost points should be sent. I coded myself something similar (see create_ghost_atoms in the attached file) . I use the background DMDA mesh to create the paddles (3D case) for the ghost particles, and DMDAGetNeighbors to get a list of the neighbor ranks. Then in the ghost migrate function one would use ranks[] to define the scatter pattern and then pack swarm points found in pointids[] into buffers and then send em. You mean do something like this lines in DMSwarmMigrate_Push_Basic: for (p = 0; p < npoints; p++) { nrank = rankval[p]; if (nrank != rank) { /* copy point into buffer */ PetscCall(DMSwarmDataBucketFillPackedArray(swarm->db, p, point_buffer)); /* insert point buffer into DMSwarmDataExchanger */ PetscCall(DMSwarmDataExPackData(de, nrank, 1, point_buffer)); } } ??? Removal of points (although unlikely required) could be achieved by sorting and removing dups from pointids[] and deleting indices from largest to smallest. Duplicate indices will likely appear in pointids[] in most ghost scenarios (eg eval an SPH kernel) and thus need to be handled if one is to remove sent points Lastly, I would have a mechanism to "reset" the DMSwarm post ghost scatter to remove the scattered ghost points. As ghosted points would appear at the back of the array a reset would just mean setting the length of the valid points back to the value it had prior to the ghosting operation. Yep, I implemented something similar (I think) in destroy_ghost_atoms inside Ghosts.cpp. I think it?s fairly straightforward to add support for this. Let me know if want to tackle it, or if you want me to code it up. I would be happy to discuss further about it! Let me know how. An other interesting question to address (which I am facing right now) would be the communication between ghosts and real particles during a SNES solver. For instance, something like this: VecScatterBegin(VecScatter scatter,Vec globalin,Vec localin,InsertMode INSERT_VALUES, ScatterMode SCATTER_FORWARD); VecScatterEnd(VecScatter scatter,Vec globalin,Vec localin,InsertMode INSERT_VALUES,ScatterMode SCATTER_FORWARD); /* For example, do local calculations from localin to localout */ VecScatterBegin(VecScatter scatter,Vec localout,Vec globalout,InsertMode ADD_VALUES,ScatterMode SCATTER_REVERSE); VecScatterEnd(VecScatter scatter,Vec localout,Vec globalout,InsertMode ADD_VALUES,ScatterMode SCATTER_REVERSE); Thanks, Miguel Cheers, Dave On Fri, 27 Sept 2024 at 01:51, MIGUEL MOLINOS PEREZ > wrote: Thank you Matt, let me give it try. Miguel On Sep 27, 2024, at 3:44?AM, Matthew Knepley > wrote: On Thu, Sep 26, 2024 at 7:18?PM MIGUEL MOLINOS PEREZ > wrote: I see, you mean: Create the ghost particles at the local cell with the same properties as particle 1 (duplicate the original particle) but different value DMSwarmField_rank. Then, call DMSwarmMigrate(*,PETSC_FALSE) so we do the migration and delete the local copies of the particle 1. Right? Yep. I think it will work, from what I know about BASIC. Thanks, Matt Thanks, Miguel On Sep 26, 2024, at 11:09?PM, Matthew Knepley > wrote: On Thu, Sep 26, 2024 at 11:20?AM MIGUEL MOLINOS PEREZ > wrote: Thank you Matt. Okey, let me have a careful look to the DMSwarmMigrate_Push_Basic implementation to see if there is some workaround. The idea of adding new particles is interesting. However, in that case, we need to initialize the new (ghost) particles using the fields of the ?real? particle, right? This can be done using something like: VecGhostUpdateBegin(Vec globalout,InsertMode ADD_VALUES, ScatterMode SCATTER_REVERSE); VecGhostUpdateEnd(Vec globalout,InsertMode ADD_VALUES, ScatterMode SCATTER_REVERSE); for the particle fields (?). I think we can just copy from the local particle. For example, suppose I decide that particle 1 should go to rank 5, 12, and 27. Then I first set p1.rank = 5, then I add two new particles with the same values as particle 1, but with rank = 12 and 27. Then when I call migrate, it will move these three particles to the correct processes, and delete the original particles and the copies from the local set. Thanks, Matt Thanks, Miguel On Sep 26, 2024, at 3:53?PM, Matthew Knepley > wrote: On Thu, Sep 26, 2024 at 6:31?AM MIGUEL MOLINOS PEREZ > wrote: Hi Matt et al, I?ve been working on the scheme that you proposed to create ghost particles (atoms in my case), and it works! With a couple of caveats: -1? In general the overlap particles will be migrate from their own rank to more than one neighbor rank, this is specially relevant for those located close to the corners. Therefore, you'll need to call DMSwarmMigrate several times (27 times for 3D cells), during the migration process. That is terrible. Let's just fix DMSwarmMigrate to have a mode that sends the particle to all overlapping neighbors at once. It can't be that hard. -2? You need to set DMSWARM_MIGRATE_BASIC. Otherwise the proposed algorithm will not work at all! Oh, I should have thought of that. Sorry. I can help code up that extension. Can you take a quick look at the BASIC code? Right now, we just use the rank attached to the particle to send it. We could have an arrays of ranks, but that seems crazy, and would blow up particle storage. How about just adding new particles with the other ranks right before migration? Thanks, Matt Hope this helps to other folks! I have a follow-up question about periodic bcc on this context, should I open a new thread of keep posting here? Thanks, Miguel On Aug 7, 2024, at 4:22?AM, MIGUEL MOLINOS PEREZ > wrote: Thanks Matt, I think I'll start by making a small program as a proof of concept. Then, if it works I'll implement it in my code and I'll be happy to share it too :-) Miguel On Aug 4, 2024, at 3:30?AM, Matthew Knepley > wrote: On Fri, Aug 2, 2024 at 7:15?PM MIGUEL MOLINOS PEREZ > wrote: Thanks again Matt, that makes a lot more sense !! Just to check that we are on the same page. You are saying: 1. create a field define a field called "owner rank" for each particle. 2. Identify the phantom particles and modify the internal variable defined by the DMSwarmField_rank variable. 3. Call DMSwarmMigrate(*,PETSC_FALSE), do the calculations using the new local vector including the ghost particles. 4. Then, once the calculations are done, rename the DMSwarmField_rank variable using the "owner rank" variable and call DMSwarmMigrate(*,PETSC_FALSE) once again. I don't think we need this last step. We can just remove those ghost particles for the next step I think. Thanks, Matt Thank you, Miguel On Aug 2, 2024, at 5:33?PM, Matthew Knepley > wrote: On Fri, Aug 2, 2024 at 11:15?AM MIGUEL MOLINOS PEREZ > wrote: Thank you Matt for your time, What you describe seems to me the ideal approach. 1) Add a particle field 'ghost' that identifies ghost vs owned particles. I think it needs options OWNED, OVERLAP, and GHOST This means, locally, I need to allocate Nlocal + ghost particles (duplicated) for my model? I would do it another way. I would allocate the particles with no overlap and set them up. Then I would identify the halo particles, mark them as OVERLAP, call DMSwarmMigrate(), and mark the migrated particles as GHOST, then unmark the OVERLAP particles. Shoot! That marking will not work since we cannot tell the difference between particles we received and particles we sent. Okay, instead of the `ghost` field we need an `owner rank` field. So then we 1) Setup the non-overlapping particles 2) Identify the halo particles 3) Change the `rank`, but not the `owner rank` 4) Call DMSwarmMigrate() Now we can identify ghost particles by the `owner rank` If that so, how to do the communication between the ghost particles living in the rank i and their ?real? counterpart in the rank j. Algo, as an alternative, what about: 1) Use an IS tag which contains, for each rank, a list of the global index of the neighbors particles outside of the rank. 2) Use VecCreateGhost to create a new vector which contains extra local space for the ghost components of the vector. 3) Use VecScatterCreate, VecScatterBegin, and VecScatterEnd to do the transference of data between a vector obtained with DMSwarmCreateGlobalVectorFromField 4) Do necessary computations using the vectors created with VecCreateGhost. This is essentially what Migrate() does. I was trying to reuse the code. Thanks, Matt Thanks, Miguel On Aug 2, 2024, at 8:58?AM, Matthew Knepley > wrote: On Thu, Aug 1, 2024 at 4:40?PM MIGUEL MOLINOS PEREZ > wrote: This Message Is From an External Sender This message came from outside your organization. Dear all, I am implementing a Molecular Dynamics (MD) code using the DMSWARM interface. In the MD simulations we evaluate on each particle (atoms) some kind of scalar functional using data from the neighbouring atoms. My problem lies in the parallel implementation of the model, because sometimes, some of these neighbours lie on a different processor. This is usually solved by using ghost particles. A similar approach (with nodes instead) is already implemented for other PETSc mesh structures like DMPlexConstructGhostCells. Unfortunately, I don't see this kind of constructs for DMSWARM. Am I missing something? I this could be done by applying a buffer region by exploiting the background DMDA mesh that I already use to do domain decomposition. Then using the buffer region of each cell to locate the ghost particles and finally using VecCreateGhost. Is this feasible? Or is there an easier approach using other PETSc functions. This is feasible, but it would be good to develop a set of best practices, since we have been mainly focused on the case of non-redundant particles. Here is how I think I would do what you want. 1) Add a particle field 'ghost' that identifies ghost vs owned particles. I think it needs options OWNED, OVERLAP, and GHOST 2) At some interval identify particles that should be sent to other processes as ghosts. I would call these "overlap particles". The determination seems application specific, so I would leave this determination to the user right now. We do two things to these particles a) Mark chosen particles as OVERLAP b) Change rank to process we are sending to 3) Call DMSwarmMigrate with PETSC_FALSE for the particle deletion flag 4) Mark OVERLAP particles as GHOST when they arrive There is one problem in the above algorithm. It does not allow sending particles to multiple ranks. We would have to do this in phases right now, or make a small adjustment to the interface allowing replication of particles when a set of ranks is specified. THanks, Matt Thank you, Miguel -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cX94fmTqJ55KnwieFlOh-KbZs8VSO8aXyABaa4NGLkLwZxIihilCvTlckfwFcT3v5r6xM9adgEGW0XrDp4qUDA$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cX94fmTqJ55KnwieFlOh-KbZs8VSO8aXyABaa4NGLkLwZxIihilCvTlckfwFcT3v5r6xM9adgEGW0XrDp4qUDA$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cX94fmTqJ55KnwieFlOh-KbZs8VSO8aXyABaa4NGLkLwZxIihilCvTlckfwFcT3v5r6xM9adgEGW0XrDp4qUDA$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cX94fmTqJ55KnwieFlOh-KbZs8VSO8aXyABaa4NGLkLwZxIihilCvTlckfwFcT3v5r6xM9adgEGW0XrDp4qUDA$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cX94fmTqJ55KnwieFlOh-KbZs8VSO8aXyABaa4NGLkLwZxIihilCvTlckfwFcT3v5r6xM9adgEGW0XrDp4qUDA$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cX94fmTqJ55KnwieFlOh-KbZs8VSO8aXyABaa4NGLkLwZxIihilCvTlckfwFcT3v5r6xM9adgEGW0XrDp4qUDA$ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Ghosts.cpp Type: application/octet-stream Size: 50945 bytes Desc: Ghosts.cpp URL: From bsmith at petsc.dev Fri Sep 27 12:26:57 2024 From: bsmith at petsc.dev (Barry Smith) Date: Fri, 27 Sep 2024 13:26:57 -0400 Subject: [petsc-users] Ask for help in tracking down bugs - setting #threads for pardiso block LU solver In-Reply-To: References: <2F21B31F-7F62-466A-A960-A97F9B07998A@petsc.dev> Message-ID: <4C679E8E-9B97-40F7-8356-8BFCD1DE3FF2@petsc.dev> I should have read your initial email more carefully: ---- Loop over block PCs { PCSetType(subPc, PCLU); set subPC type PCFactorSetMatSolverType(subPc, MATSOLVERMKL_PARDISO); set solver for LU SetPardisoParams("pardiso"); set mkl related options including -mat_mkl_pardiso_65 equal to the desired #threads; return code after PetscOptionsSetValue is ok, so I presume the option is set correctly. PCSetFromOptions(subPc); } Then what I see under the debugger PCSetUp after a series of calls gets to MatLUFactorSymbolic_AIJMKL_PARDISO which in turn calls MatSetFromOptions_MKL_PARDISO(F, A) and there in these lines it seems to ignore what I?ve set for -mat_mkl_pardiso_65 and sets #threads to the default 1 PetscCall(PetscOptionsInt("-mat_mkl_pardiso_65", "Suggested number of threads to use within PARDISO", "None", threads, &threads, &flg)); if (flg) PetscSetMKL_PARDISOThreads((int)threads); ----- PCSetFromOptions() actually does not call MatSetFromOptions() on the factored matrix. There is no call to MatSetFromOptions() from PCSetFromOptions() or PetscSetFromOptions_Factor() because the factored matrix F does not yet exist (it is created in PCSetUp()). This is why your call to PCSetFromOptions(subPc); does not process any of your mat mkl options. Since MatSetFromOptions() cannot be called from PCSetFromOptions(), MatSetFromOptions_MKL_PARDISO(F, A) is called in MatFactorSymbolic_AIJMKL_PARDISO_Private() as you noted when the PC is being set up. You say "here in these lines it seems to ignore what I?ve set for -mat_mkl_pardiso_65 and sets #threads to the default 1" This is the part a do not understand, if your values have been inserted in the database earlier in the code and remain in the database they should get processed inside the MatSetFromOptions_MKL_PARDISO(F, A) called by MatFactorSymbolic_AIJMKL_PARDISO_Private(). Can you please put a break point at MatSetFromOptions_MKL_PARDISO(F, A) and then when it breaks put call PetscOptionsView(0,0) it should print out the contents of the options database so you can verify if the options you added are there. You can then step through the code in MatSetFromOptions_MKL_PARDISO to see what is happening as it tries to acess the database to see why it skips PetscCall(PetscOptionsInt("-mat_mkl_pardiso_65", "Suggested number of threads to use within MKL PARDISO", "None", threads, &threads, &flg)); if (flg) PetscSetMKL_PARDISOThreads((int)threads); > On Sep 27, 2024, at 11:58?AM, Voinov, Boris wrote: > > Thanks a lot Barry, > > It looks like something wrong with my own settings though. It doesn?t make an effect. I mean this call > SetPardisoParams("pardiso"); > followed by > PCSetFromOptions(subPc); > doesn?t end up with a call of PetscSetMKL_PARDISOThreads() > This function is called later when PCSetUp is invoked. > > Eventually what I see in the end is > Option left: name:-mat_mkl_pardiso_65 value: 8 source: code listed among unused database options > That is what I?ve set with my own settings, meaning it?s in the database, I guess. > > Could it be that it hits a ?not proper? section in the options database and is not being used by PCSetUp? > > Best regards, > Boris > > From: Barry Smith > > Sent: Friday, September 27, 2024 4:21 PM > To: Voinov, Boris > > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Ask for help in tracking down bugs - setting #threads for pardiso block LU solver > > > Boris, > > There is nothing obviously wrong with what you outline, so let's dive down into the code with the debugger and see what is happening. > > PetscSetMKL_PARDISOThreads() set from options is called when the command line option is provided, and this calls > > PETSC_EXTERN void PetscSetMKL_PARDISOThreads(int threads) > { > mkl_domain_set_num_threads(threads, MKL_DOMAIN_PARDISO); > } > > so in the debugger can you check that mkl_domain_set_num_threads() gets called with your requested number of threads? > > BTW: Since you are hardwiring the use of Pardiso in the code with function calls you couldsimply call PetscSetMKL_PARDISOThreads() directly > after your line of code > > PCFactorSetMatSolverType(subPc, MATSOLVERMKL_PARDISO); set solver for LU > > instead of feeding it in through the options database. > > Let us know how this turns out > > Barry > > > > > > > On Sep 27, 2024, at 7:24?AM, Voinov, Boris > wrote: > > Hello, > > Could you please help me to figure out what?s wrong with the way I?m trying to make mkl pardiso use the number of threads more than one while setting it as a sub PC solver for ASM preconditioner. > Here is how I do this > Loop over block PCs { > PCSetType(subPc, PCLU); set subPC type > PCFactorSetMatSolverType(subPc, MATSOLVERMKL_PARDISO); set solver for LU > SetPardisoParams("pardiso"); set mkl related options including -mat_mkl_pardiso_65 equal to the desired #threads; return code after PetscOptionsSetValue is ok, so I presume the option is set correctly. > PCSetFromOptions(subPc); > } > > Then what I see under the debugger > PCSetUp after a series of calls gets to MatLUFactorSymbolic_AIJMKL_PARDISO which in turn calls MatSetFromOptions_MKL_PARDISO(F, A) and there > in these lines it seems to ignore what I?ve set for -mat_mkl_pardiso_65 and sets #threads to the default 1 > PetscCall(PetscOptionsInt("-mat_mkl_pardiso_65", "Suggested number of threads to use within PARDISO", "None", threads, &threads, &flg)); > if (flg) PetscSetMKL_PARDISOThreads((int)threads); > > This is about petsc-3.20. > PS when I do this in a standalone code which reads a matrix and solves the linear system and set -mat_mkl_pardiso_65 in the command line it?s all right but I need the number of threads to be set in other way in my code. > > Thank you and best regards, > Boris > > ------------------------------------------------------------- > Intel Ireland Limited (Branch) > Collinstown Industrial Park, Leixlip, County Kildare, Ireland > Registered Number: E902934 > > This e-mail and any attachments may contain confidential material for > the sole use of the intended recipient(s). Any review or distribution > by others is strictly prohibited. If you are not the intended > recipient, please contact the sender and delete all copies. > > > ------------------------------------------------------------- > Intel Ireland Limited (Branch) > Collinstown Industrial Park, Leixlip, County Kildare, Ireland > Registered Number: E902934 > > This e-mail and any attachments may contain confidential material for > the sole use of the intended recipient(s). Any review or distribution > by others is strictly prohibited. If you are not the intended > recipient, please contact the sender and delete all copies. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From boris.voinov at intel.com Fri Sep 27 12:58:20 2024 From: boris.voinov at intel.com (Voinov, Boris) Date: Fri, 27 Sep 2024 17:58:20 +0000 Subject: [petsc-users] Ask for help in tracking down bugs - setting #threads for pardiso block LU solver In-Reply-To: <4C679E8E-9B97-40F7-8356-8BFCD1DE3FF2@petsc.dev> References: <2F21B31F-7F62-466A-A960-A97F9B07998A@petsc.dev> <4C679E8E-9B97-40F7-8356-8BFCD1DE3FF2@petsc.dev> Message-ID: Thanks Barry, That makes sense and explains the first part of my problem. I will track it down and let you know what happens inside MatSetFromOptions_MKL_PARDISO. I?m going to have to rebuild the code without the optimization flags, otherwise it?s not easy to follow precisely the execution order. Have a great weekend! Best regards, Boris From: Barry Smith Sent: Friday, September 27, 2024 6:27 PM To: Voinov, Boris Cc: petsc-users Subject: Re: [petsc-users] Ask for help in tracking down bugs - setting #threads for pardiso block LU solver I should have read your initial email more carefully: ---- Loop over block PCs { PCSetType(subPc, PCLU); set subPC type PCFactorSetMatSolverType(subPc, MATSOLVERMKL_PARDISO); set solver for LU SetPardisoParams("pardiso"); set mkl related options including -mat_mkl_pardiso_65 equal to the desired #threads; return code after PetscOptionsSetValue is ok, so I presume the option is set correctly. PCSetFromOptions(subPc); } Then what I see under the debugger PCSetUp after a series of calls gets to MatLUFactorSymbolic_AIJMKL_PARDISO which in turn calls MatSetFromOptions_MKL_PARDISO(F, A) and there in these lines it seems to ignore what I?ve set for -mat_mkl_pardiso_65 and sets #threads to the default 1 PetscCall(PetscOptionsInt("-mat_mkl_pardiso_65", "Suggested number of threads to use within PARDISO", "None", threads, &threads, &flg)); if (flg) PetscSetMKL_PARDISOThreads((int)threads); ----- PCSetFromOptions() actually does not call MatSetFromOptions() on the factored matrix. There is no call to MatSetFromOptions() from PCSetFromOptions() or PetscSetFromOptions_Factor() because the factored matrix F does not yet exist (it is created in PCSetUp()). This is why your call to PCSetFromOptions(subPc); does not process any of your mat mkl options. Since MatSetFromOptions() cannot be called from PCSetFromOptions(), MatSetFromOptions_MKL_PARDISO(F, A) is called in MatFactorSymbolic_AIJMKL_PARDISO_Private() as you noted when the PC is being set up. You say "here in these lines it seems to ignore what I?ve set for -mat_mkl_pardiso_65 and sets #threads to the default 1" This is the part a do not understand, if your values have been inserted in the database earlier in the code and remain in the database they should get processed inside the MatSetFromOptions_MKL_PARDISO(F, A) called by MatFactorSymbolic_AIJMKL_PARDISO_Private(). Can you please put a break point at MatSetFromOptions_MKL_PARDISO(F, A) and then when it breaks put call PetscOptionsView(0,0) it should print out the contents of the options database so you can verify if the options you added are there. You can then step through the code in MatSetFromOptions_MKL_PARDISO to see what is happening as it tries to acess the database to see why it skips PetscCall(PetscOptionsInt("-mat_mkl_pardiso_65", "Suggested number of threads to use within MKL PARDISO", "None", threads, &threads, &flg)); if (flg) PetscSetMKL_PARDISOThreads((int)threads); On Sep 27, 2024, at 11:58?AM, Voinov, Boris > wrote: Thanks a lot Barry, It looks like something wrong with my own settings though. It doesn?t make an effect. I mean this call SetPardisoParams("pardiso"); followed by PCSetFromOptions(subPc); doesn?t end up with a call of PetscSetMKL_PARDISOThreads() This function is called later when PCSetUp is invoked. Eventually what I see in the end is Option left: name:-mat_mkl_pardiso_65 value: 8 source: code listed among unused database options That is what I?ve set with my own settings, meaning it?s in the database, I guess. Could it be that it hits a ?not proper? section in the options database and is not being used by PCSetUp? Best regards, Boris From: Barry Smith > Sent: Friday, September 27, 2024 4:21 PM To: Voinov, Boris > Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Ask for help in tracking down bugs - setting #threads for pardiso block LU solver Boris, There is nothing obviously wrong with what you outline, so let's dive down into the code with the debugger and see what is happening. PetscSetMKL_PARDISOThreads() set from options is called when the command line option is provided, and this calls PETSC_EXTERN void PetscSetMKL_PARDISOThreads(int threads) { mkl_domain_set_num_threads(threads, MKL_DOMAIN_PARDISO); } so in the debugger can you check that mkl_domain_set_num_threads() gets called with your requested number of threads? BTW: Since you are hardwiring the use of Pardiso in the code with function calls you couldsimply call PetscSetMKL_PARDISOThreads() directly after your line of code PCFactorSetMatSolverType(subPc, MATSOLVERMKL_PARDISO); set solver for LU instead of feeding it in through the options database. Let us know how this turns out Barry On Sep 27, 2024, at 7:24?AM, Voinov, Boris > wrote: Hello, Could you please help me to figure out what?s wrong with the way I?m trying to make mkl pardiso use the number of threads more than one while setting it as a sub PC solver for ASM preconditioner. Here is how I do this Loop over block PCs { PCSetType(subPc, PCLU); set subPC type PCFactorSetMatSolverType(subPc, MATSOLVERMKL_PARDISO); set solver for LU SetPardisoParams("pardiso"); set mkl related options including -mat_mkl_pardiso_65 equal to the desired #threads; return code after PetscOptionsSetValue is ok, so I presume the option is set correctly. PCSetFromOptions(subPc); } Then what I see under the debugger PCSetUp after a series of calls gets to MatLUFactorSymbolic_AIJMKL_PARDISO which in turn calls MatSetFromOptions_MKL_PARDISO(F, A) and there in these lines it seems to ignore what I?ve set for -mat_mkl_pardiso_65 and sets #threads to the default 1 PetscCall(PetscOptionsInt("-mat_mkl_pardiso_65", "Suggested number of threads to use within PARDISO", "None", threads, &threads, &flg)); if (flg) PetscSetMKL_PARDISOThreads((int)threads); This is about petsc-3.20. PS when I do this in a standalone code which reads a matrix and solves the linear system and set -mat_mkl_pardiso_65 in the command line it?s all right but I need the number of threads to be set in other way in my code. Thank you and best regards, Boris ------------------------------------------------------------- Intel Ireland Limited (Branch) Collinstown Industrial Park, Leixlip, County Kildare, Ireland Registered Number: E902934 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. ------------------------------------------------------------- Intel Ireland Limited (Branch) Collinstown Industrial Park, Leixlip, County Kildare, Ireland Registered Number: E902934 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. ------------------------------------------------------------- Intel Ireland Limited (Branch) Collinstown Industrial Park, Leixlip, County Kildare, Ireland Registered Number: E902934 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. -------------- next part -------------- An HTML attachment was scrubbed... URL: From karthikeyan.chockalingam at stfc.ac.uk Mon Sep 30 05:49:55 2024 From: karthikeyan.chockalingam at stfc.ac.uk (Karthikeyan Chockalingam - STFC UKRI) Date: Mon, 30 Sep 2024 10:49:55 +0000 Subject: [petsc-users] Questions DMPlex Message-ID: Hi, We have been using PETSc?s block version of the AIJ matrix format to implement fully coupled multiphysics problems using finite elements. Are there specific advantages to moving toward DMPlex for finite element-based coupled multi-physics implementation? Can you still access Hypre? Does the format support running on GPUs? Thank you. Kind regards, Karthik. -- Karthik Chockalingam, Ph.D. Senior Research Software Engineer High Performance Systems Engineering Group Hartree Centre | Science and Technology Facilities Council karthikeyan.chockalingam at stfc.ac.uk [signature_3970890138] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 18270 bytes Desc: image001.png URL: From boris.voinov at intel.com Mon Sep 30 10:36:28 2024 From: boris.voinov at intel.com (Voinov, Boris) Date: Mon, 30 Sep 2024 15:36:28 +0000 Subject: [petsc-users] Ask for help in tracking down bugs - setting #threads for pardiso block LU solver In-Reply-To: <4C679E8E-9B97-40F7-8356-8BFCD1DE3FF2@petsc.dev> References: <2F21B31F-7F62-466A-A960-A97F9B07998A@petsc.dev> <4C679E8E-9B97-40F7-8356-8BFCD1DE3FF2@petsc.dev> Message-ID: Hi Barry, Here are a bit more details. Under the debugger, when I break just before PetscCall(PetscOptionsInt("-mat_mkl_pardiso_65", "Suggested number of threads to use within PARDISO", "None", threads, &threads, &flg)); what I see for option ?-mat_mkl_pardiso_65? in the defaultoptions is Expression: *(defaultoptions->names+16) Address: 0x17d361e0 Type: $string * Window Status: Title: *(defaultoptions->names+16) - floops.linux64.1 - 1.1 Language: C++ Valid in Scope: (Global) Compiled in Scope: PetscOptionsBegin_Private (Subroutine) aoptions.c (File) libpetsc.so.3.20 (Image) UPC Phase: UPC Addr: Examine Count: 0 Column Count: 0 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Value 0x17d35b00 -> "mat_mkl_pardiso_65" And Expression: *(defaultoptions->values+16) Address: 0x17d36330 Type: $string * Window Status: Title: *(defaultoptions->values+16) - floops.linux64.1 - 1.1 Language: C++ Valid in Scope: (Global) Compiled in Scope: PetscOptionsBegin_Private (Subroutine) aoptions.c (File) libpetsc.so.3.20 (Image) UPC Phase: UPC Addr: Examine Count: 0 Column Count: 0 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Value 0x17d35b20 -> "8" I.e. exactly what I set in the loop for subPC and all other options as well. So it looks ok. Now when I jump into (PetscOptionsInt) I turn out here: PetscErrorCode PetscOptionsInt_Private(PetscOptionItems *PetscOptionsObject, const char opt[], const char text[], const char man[], PetscInt currentvalue, PetscInt *value, PetscBool *set, PetscInt lb, PetscInt ub) { const char *prefix = PetscOptionsObject->prefix; const PetscOptions options = PetscOptionsObject->options; PetscBool wasset; PetscOptionsObject is like this: Expression: *PetscOptionsObject Address: 0x7fffffff6d80 Type: PetscOptionItems Window Status: Title: *PetscOptionsObject - PetscOptionsInt_Private - 1.1 Language: C++ Valid in Scope: PetscOptionsInt_Private (Subroutine) aoptions.c (File) libpetsc.so.3.20 (Image) Compiled in Scope: PetscOptionsInt_Private (Subroutine) aoptions.c (File) libpetsc.so.3.20 (Image) UPC Phase: UPC Addr: Examine Count: 0 Column Count: 0 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Field Type Value count PetscInt 0x00000001 (1) next PetscOptionItem 0x00000000 prefix $string * 0x00000000 pprefix $string * 0x00000000 title $string * 0x17d39140 -> "MKL_PARDISO Options" comm _p_PetscObject::MPI_Comm 0x84000002 (-2080374782) printhelp PetscBool PETSC_FALSE (0) changedmethod PetscBool PETSC_FALSE (0) alreadyprinted PetscBool PETSC_FALSE (0) object PetscObject 0x00000000 options _p_PetscObject::PetscOptions 0x00000000 - options are empty And then, I guess, it doesn?t see my settings and overwrites the number of threads with the default 1. It seems like the settings I made were not associated with PetscOptionsObject named ?MKL_PARDISO Options? Best regards, Boris From: Barry Smith Sent: Friday, September 27, 2024 6:27 PM To: Voinov, Boris Cc: petsc-users Subject: Re: [petsc-users] Ask for help in tracking down bugs - setting #threads for pardiso block LU solver I should have read your initial email more carefully: ---- Loop over block PCs { PCSetType(subPc, PCLU); set subPC type PCFactorSetMatSolverType(subPc, MATSOLVERMKL_PARDISO); set solver for LU SetPardisoParams("pardiso"); set mkl related options including -mat_mkl_pardiso_65 equal to the desired #threads; return code after PetscOptionsSetValue is ok, so I presume the option is set correctly. PCSetFromOptions(subPc); } Then what I see under the debugger PCSetUp after a series of calls gets to MatLUFactorSymbolic_AIJMKL_PARDISO which in turn calls MatSetFromOptions_MKL_PARDISO(F, A) and there in these lines it seems to ignore what I?ve set for -mat_mkl_pardiso_65 and sets #threads to the default 1 PetscCall(PetscOptionsInt("-mat_mkl_pardiso_65", "Suggested number of threads to use within PARDISO", "None", threads, &threads, &flg)); if (flg) PetscSetMKL_PARDISOThreads((int)threads); ----- PCSetFromOptions() actually does not call MatSetFromOptions() on the factored matrix. There is no call to MatSetFromOptions() from PCSetFromOptions() or PetscSetFromOptions_Factor() because the factored matrix F does not yet exist (it is created in PCSetUp()). This is why your call to PCSetFromOptions(subPc); does not process any of your mat mkl options. Since MatSetFromOptions() cannot be called from PCSetFromOptions(), MatSetFromOptions_MKL_PARDISO(F, A) is called in MatFactorSymbolic_AIJMKL_PARDISO_Private() as you noted when the PC is being set up. You say "here in these lines it seems to ignore what I?ve set for -mat_mkl_pardiso_65 and sets #threads to the default 1" This is the part a do not understand, if your values have been inserted in the database earlier in the code and remain in the database they should get processed inside the MatSetFromOptions_MKL_PARDISO(F, A) called by MatFactorSymbolic_AIJMKL_PARDISO_Private(). Can you please put a break point at MatSetFromOptions_MKL_PARDISO(F, A) and then when it breaks put call PetscOptionsView(0,0) it should print out the contents of the options database so you can verify if the options you added are there. You can then step through the code in MatSetFromOptions_MKL_PARDISO to see what is happening as it tries to acess the database to see why it skips PetscCall(PetscOptionsInt("-mat_mkl_pardiso_65", "Suggested number of threads to use within MKL PARDISO", "None", threads, &threads, &flg)); if (flg) PetscSetMKL_PARDISOThreads((int)threads); On Sep 27, 2024, at 11:58?AM, Voinov, Boris > wrote: Thanks a lot Barry, It looks like something wrong with my own settings though. It doesn?t make an effect. I mean this call SetPardisoParams("pardiso"); followed by PCSetFromOptions(subPc); doesn?t end up with a call of PetscSetMKL_PARDISOThreads() This function is called later when PCSetUp is invoked. Eventually what I see in the end is Option left: name:-mat_mkl_pardiso_65 value: 8 source: code listed among unused database options That is what I?ve set with my own settings, meaning it?s in the database, I guess. Could it be that it hits a ?not proper? section in the options database and is not being used by PCSetUp? Best regards, Boris From: Barry Smith > Sent: Friday, September 27, 2024 4:21 PM To: Voinov, Boris > Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Ask for help in tracking down bugs - setting #threads for pardiso block LU solver Boris, There is nothing obviously wrong with what you outline, so let's dive down into the code with the debugger and see what is happening. PetscSetMKL_PARDISOThreads() set from options is called when the command line option is provided, and this calls PETSC_EXTERN void PetscSetMKL_PARDISOThreads(int threads) { mkl_domain_set_num_threads(threads, MKL_DOMAIN_PARDISO); } so in the debugger can you check that mkl_domain_set_num_threads() gets called with your requested number of threads? BTW: Since you are hardwiring the use of Pardiso in the code with function calls you couldsimply call PetscSetMKL_PARDISOThreads() directly after your line of code PCFactorSetMatSolverType(subPc, MATSOLVERMKL_PARDISO); set solver for LU instead of feeding it in through the options database. Let us know how this turns out Barry On Sep 27, 2024, at 7:24?AM, Voinov, Boris > wrote: Hello, Could you please help me to figure out what?s wrong with the way I?m trying to make mkl pardiso use the number of threads more than one while setting it as a sub PC solver for ASM preconditioner. Here is how I do this Loop over block PCs { PCSetType(subPc, PCLU); set subPC type PCFactorSetMatSolverType(subPc, MATSOLVERMKL_PARDISO); set solver for LU SetPardisoParams("pardiso"); set mkl related options including -mat_mkl_pardiso_65 equal to the desired #threads; return code after PetscOptionsSetValue is ok, so I presume the option is set correctly. PCSetFromOptions(subPc); } Then what I see under the debugger PCSetUp after a series of calls gets to MatLUFactorSymbolic_AIJMKL_PARDISO which in turn calls MatSetFromOptions_MKL_PARDISO(F, A) and there in these lines it seems to ignore what I?ve set for -mat_mkl_pardiso_65 and sets #threads to the default 1 PetscCall(PetscOptionsInt("-mat_mkl_pardiso_65", "Suggested number of threads to use within PARDISO", "None", threads, &threads, &flg)); if (flg) PetscSetMKL_PARDISOThreads((int)threads); This is about petsc-3.20. PS when I do this in a standalone code which reads a matrix and solves the linear system and set -mat_mkl_pardiso_65 in the command line it?s all right but I need the number of threads to be set in other way in my code. Thank you and best regards, Boris ------------------------------------------------------------- Intel Ireland Limited (Branch) Collinstown Industrial Park, Leixlip, County Kildare, Ireland Registered Number: E902934 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. ------------------------------------------------------------- Intel Ireland Limited (Branch) Collinstown Industrial Park, Leixlip, County Kildare, Ireland Registered Number: E902934 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. ------------------------------------------------------------- Intel Ireland Limited (Branch) Collinstown Industrial Park, Leixlip, County Kildare, Ireland Registered Number: E902934 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. -------------- next part -------------- An HTML attachment was scrubbed... URL: From boris.voinov at intel.com Mon Sep 30 10:57:24 2024 From: boris.voinov at intel.com (Voinov, Boris) Date: Mon, 30 Sep 2024 15:57:24 +0000 Subject: [petsc-users] Ask for help in tracking down bugs - setting #threads for pardiso block LU solver In-Reply-To: References: <2F21B31F-7F62-466A-A960-A97F9B07998A@petsc.dev> <4C679E8E-9B97-40F7-8356-8BFCD1DE3FF2@petsc.dev> Message-ID: Hi Barry, Sorry for messing it up. Please disregard my message. It actually was all true but down in PetscOptionsInt_Private PetscOptionsGetInt(options, prefix, opt, value, &wasset) returns *value = 8 and wasset = true, i.e. it?s all right. Regards, Boris From: Voinov, Boris Sent: Monday, September 30, 2024 4:36 PM To: Barry Smith Cc: petsc-users Subject: RE: [petsc-users] Ask for help in tracking down bugs - setting #threads for pardiso block LU solver Hi Barry, Here are a bit more details. Under the debugger, when I break just before PetscCall(PetscOptionsInt("-mat_mkl_pardiso_65", "Suggested number of threads to use within PARDISO", "None", threads, &threads, &flg)); what I see for option ?-mat_mkl_pardiso_65? in the defaultoptions is Expression: *(defaultoptions->names+16) Address: 0x17d361e0 Type: $string * Window Status: Title: *(defaultoptions->names+16) - floops.linux64.1 - 1.1 Language: C++ Valid in Scope: (Global) Compiled in Scope: PetscOptionsBegin_Private (Subroutine) aoptions.c (File) libpetsc.so.3.20 (Image) UPC Phase: UPC Addr: Examine Count: 0 Column Count: 0 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Value 0x17d35b00 -> "mat_mkl_pardiso_65" And Expression: *(defaultoptions->values+16) Address: 0x17d36330 Type: $string * Window Status: Title: *(defaultoptions->values+16) - floops.linux64.1 - 1.1 Language: C++ Valid in Scope: (Global) Compiled in Scope: PetscOptionsBegin_Private (Subroutine) aoptions.c (File) libpetsc.so.3.20 (Image) UPC Phase: UPC Addr: Examine Count: 0 Column Count: 0 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Value 0x17d35b20 -> "8" I.e. exactly what I set in the loop for subPC and all other options as well. So it looks ok. Now when I jump into (PetscOptionsInt) I turn out here: PetscErrorCode PetscOptionsInt_Private(PetscOptionItems *PetscOptionsObject, const char opt[], const char text[], const char man[], PetscInt currentvalue, PetscInt *value, PetscBool *set, PetscInt lb, PetscInt ub) { const char *prefix = PetscOptionsObject->prefix; const PetscOptions options = PetscOptionsObject->options; PetscBool wasset; PetscOptionsObject is like this: Expression: *PetscOptionsObject Address: 0x7fffffff6d80 Type: PetscOptionItems Window Status: Title: *PetscOptionsObject - PetscOptionsInt_Private - 1.1 Language: C++ Valid in Scope: PetscOptionsInt_Private (Subroutine) aoptions.c (File) libpetsc.so.3.20 (Image) Compiled in Scope: PetscOptionsInt_Private (Subroutine) aoptions.c (File) libpetsc.so.3.20 (Image) UPC Phase: UPC Addr: Examine Count: 0 Column Count: 0 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Field Type Value count PetscInt 0x00000001 (1) next PetscOptionItem 0x00000000 prefix $string * 0x00000000 pprefix $string * 0x00000000 title $string * 0x17d39140 -> "MKL_PARDISO Options" comm _p_PetscObject::MPI_Comm 0x84000002 (-2080374782) printhelp PetscBool PETSC_FALSE (0) changedmethod PetscBool PETSC_FALSE (0) alreadyprinted PetscBool PETSC_FALSE (0) object PetscObject 0x00000000 options _p_PetscObject::PetscOptions 0x00000000 - options are empty And then, I guess, it doesn?t see my settings and overwrites the number of threads with the default 1. It seems like the settings I made were not associated with PetscOptionsObject named ?MKL_PARDISO Options? Best regards, Boris From: Barry Smith > Sent: Friday, September 27, 2024 6:27 PM To: Voinov, Boris > Cc: petsc-users > Subject: Re: [petsc-users] Ask for help in tracking down bugs - setting #threads for pardiso block LU solver I should have read your initial email more carefully: ---- Loop over block PCs { PCSetType(subPc, PCLU); set subPC type PCFactorSetMatSolverType(subPc, MATSOLVERMKL_PARDISO); set solver for LU SetPardisoParams("pardiso"); set mkl related options including -mat_mkl_pardiso_65 equal to the desired #threads; return code after PetscOptionsSetValue is ok, so I presume the option is set correctly. PCSetFromOptions(subPc); } Then what I see under the debugger PCSetUp after a series of calls gets to MatLUFactorSymbolic_AIJMKL_PARDISO which in turn calls MatSetFromOptions_MKL_PARDISO(F, A) and there in these lines it seems to ignore what I?ve set for -mat_mkl_pardiso_65 and sets #threads to the default 1 PetscCall(PetscOptionsInt("-mat_mkl_pardiso_65", "Suggested number of threads to use within PARDISO", "None", threads, &threads, &flg)); if (flg) PetscSetMKL_PARDISOThreads((int)threads); ----- PCSetFromOptions() actually does not call MatSetFromOptions() on the factored matrix. There is no call to MatSetFromOptions() from PCSetFromOptions() or PetscSetFromOptions_Factor() because the factored matrix F does not yet exist (it is created in PCSetUp()). This is why your call to PCSetFromOptions(subPc); does not process any of your mat mkl options. Since MatSetFromOptions() cannot be called from PCSetFromOptions(), MatSetFromOptions_MKL_PARDISO(F, A) is called in MatFactorSymbolic_AIJMKL_PARDISO_Private() as you noted when the PC is being set up. You say "here in these lines it seems to ignore what I?ve set for -mat_mkl_pardiso_65 and sets #threads to the default 1" This is the part a do not understand, if your values have been inserted in the database earlier in the code and remain in the database they should get processed inside the MatSetFromOptions_MKL_PARDISO(F, A) called by MatFactorSymbolic_AIJMKL_PARDISO_Private(). Can you please put a break point at MatSetFromOptions_MKL_PARDISO(F, A) and then when it breaks put call PetscOptionsView(0,0) it should print out the contents of the options database so you can verify if the options you added are there. You can then step through the code in MatSetFromOptions_MKL_PARDISO to see what is happening as it tries to acess the database to see why it skips PetscCall(PetscOptionsInt("-mat_mkl_pardiso_65", "Suggested number of threads to use within MKL PARDISO", "None", threads, &threads, &flg)); if (flg) PetscSetMKL_PARDISOThreads((int)threads); On Sep 27, 2024, at 11:58?AM, Voinov, Boris > wrote: Thanks a lot Barry, It looks like something wrong with my own settings though. It doesn?t make an effect. I mean this call SetPardisoParams("pardiso"); followed by PCSetFromOptions(subPc); doesn?t end up with a call of PetscSetMKL_PARDISOThreads() This function is called later when PCSetUp is invoked. Eventually what I see in the end is Option left: name:-mat_mkl_pardiso_65 value: 8 source: code listed among unused database options That is what I?ve set with my own settings, meaning it?s in the database, I guess. Could it be that it hits a ?not proper? section in the options database and is not being used by PCSetUp? Best regards, Boris From: Barry Smith > Sent: Friday, September 27, 2024 4:21 PM To: Voinov, Boris > Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Ask for help in tracking down bugs - setting #threads for pardiso block LU solver Boris, There is nothing obviously wrong with what you outline, so let's dive down into the code with the debugger and see what is happening. PetscSetMKL_PARDISOThreads() set from options is called when the command line option is provided, and this calls PETSC_EXTERN void PetscSetMKL_PARDISOThreads(int threads) { mkl_domain_set_num_threads(threads, MKL_DOMAIN_PARDISO); } so in the debugger can you check that mkl_domain_set_num_threads() gets called with your requested number of threads? BTW: Since you are hardwiring the use of Pardiso in the code with function calls you couldsimply call PetscSetMKL_PARDISOThreads() directly after your line of code PCFactorSetMatSolverType(subPc, MATSOLVERMKL_PARDISO); set solver for LU instead of feeding it in through the options database. Let us know how this turns out Barry On Sep 27, 2024, at 7:24?AM, Voinov, Boris > wrote: Hello, Could you please help me to figure out what?s wrong with the way I?m trying to make mkl pardiso use the number of threads more than one while setting it as a sub PC solver for ASM preconditioner. Here is how I do this Loop over block PCs { PCSetType(subPc, PCLU); set subPC type PCFactorSetMatSolverType(subPc, MATSOLVERMKL_PARDISO); set solver for LU SetPardisoParams("pardiso"); set mkl related options including -mat_mkl_pardiso_65 equal to the desired #threads; return code after PetscOptionsSetValue is ok, so I presume the option is set correctly. PCSetFromOptions(subPc); } Then what I see under the debugger PCSetUp after a series of calls gets to MatLUFactorSymbolic_AIJMKL_PARDISO which in turn calls MatSetFromOptions_MKL_PARDISO(F, A) and there in these lines it seems to ignore what I?ve set for -mat_mkl_pardiso_65 and sets #threads to the default 1 PetscCall(PetscOptionsInt("-mat_mkl_pardiso_65", "Suggested number of threads to use within PARDISO", "None", threads, &threads, &flg)); if (flg) PetscSetMKL_PARDISOThreads((int)threads); This is about petsc-3.20. PS when I do this in a standalone code which reads a matrix and solves the linear system and set -mat_mkl_pardiso_65 in the command line it?s all right but I need the number of threads to be set in other way in my code. Thank you and best regards, Boris ------------------------------------------------------------- Intel Ireland Limited (Branch) Collinstown Industrial Park, Leixlip, County Kildare, Ireland Registered Number: E902934 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. ------------------------------------------------------------- Intel Ireland Limited (Branch) Collinstown Industrial Park, Leixlip, County Kildare, Ireland Registered Number: E902934 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. ------------------------------------------------------------- Intel Ireland Limited (Branch) Collinstown Industrial Park, Leixlip, County Kildare, Ireland Registered Number: E902934 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Sep 30 11:16:40 2024 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 30 Sep 2024 12:16:40 -0400 Subject: [petsc-users] Questions DMPlex In-Reply-To: References: Message-ID: On Mon, Sep 30, 2024 at 6:50?AM Karthikeyan Chockalingam - STFC UKRI via petsc-users wrote: > Hi, > > > > We have been using PETSc?s block version of the AIJ matrix format to > implement fully coupled multiphysics problems using finite elements. > > > > Are there specific advantages to moving toward DMPlex for finite > element-based coupled multi-physics implementation? > DMPlex is intended to help manage unstructured grids. It can read/write meshes and functions over them, layout data over the grid in parallel, compute local-to-global maps, modify meshes, and also provides some tools for assembling functions and operators. > Can you still access Hypre? > Yes. Hypre interacts with the solvers. DMPlex is only there to help in assembly. > Does the format support running on GPUs? > Running the solver on GPUs is unchanged. However, for assembling on GPUs, there are at least two options: 1) Do everything yourself using the information provided by DMPlex. This is what Mark does in dmplexland.c for assembling the Landau operator. It is hard, but some people can do it. 2) Use a library. LibCEED is a library for assembling on the GPU, and DMPlex provides support for interacting with it. There are DMPlex examples in the LibCEED distribution. This is what I do for assembling on GPUs. Thanks, Matt > Thank you. > > > > Kind regards, > > Karthik. > > > > > > > > -- > > *Karthik Chockalingam, Ph.D.* > > Senior Research Software Engineer > > High Performance Systems Engineering Group > > Hartree Centre | Science and Technology Facilities Council > > karthikeyan.chockalingam at stfc.ac.uk > > > > [image: signature_3970890138] > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bEvuhZjmSUukrYMa5jGF9VlNNxVETTLlIXH3aoOO8d6G9TdvzgX4lzosRefIY55K1fOFTKHxQQ4s_dvd0Yga$ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 18270 bytes Desc: not available URL: From a.croucher at auckland.ac.nz Mon Sep 30 18:03:05 2024 From: a.croucher at auckland.ac.nz (Adrian Croucher) Date: Tue, 1 Oct 2024 12:03:05 +1300 Subject: [petsc-users] PetscSortInt() in PETSc 3.22 Message-ID: <451e5d3c-77d3-4703-adf2-4b3b349b761d@auckland.ac.nz> hi I just pulled PETSc 3.22 and tried to build my (Fortran) code with it. I'm getting compilation errors when it gets to calls to PetscSortInt(): ? 567 |????????? call PetscSortInt(na, asort, ierr); CHKERRQ(ierr) ????? |??????????????????????????????????????????? 1 Error: Type mismatch in argument ?a? at (1); passed INTEGER(4) to INTEGER(8) Here na is PetscInt, asort an array of PetscInt and ierr is PetscErrorCode. Do I need to make changes to get this to work again? - Adrian -- Dr Adrian Croucher Senior Research Fellow Department of Engineering Science Waipapa Taumata Rau / University of Auckland, New Zealand email: a.croucher at auckland.ac.nz tel: +64 (0)9 923 4611 From balay.anl at fastmail.org Mon Sep 30 18:11:55 2024 From: balay.anl at fastmail.org (Satish Balay) Date: Mon, 30 Sep 2024 18:11:55 -0500 (CDT) Subject: [petsc-users] PetscSortInt() in PETSc 3.22 In-Reply-To: <451e5d3c-77d3-4703-adf2-4b3b349b761d@auckland.ac.nz> References: <451e5d3c-77d3-4703-adf2-4b3b349b761d@auckland.ac.nz> Message-ID: <2e0bd03e-3092-b507-a469-bfa3b562e96a@fastmail.org> https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/PetscSortInt/__;!!G_uCfscf7eWS!abfR-2RYdZsOXFaja4qPl17LNED2IYBaty9XZrlDUTRk3PGyPVqfaHKUMl9w130BTQi21XlV1oOhUbxVmyOhOUI1Y9A$ 'na' should be PetscCount Satish On Tue, 1 Oct 2024, Adrian Croucher wrote: > hi > > I just pulled PETSc 3.22 and tried to build my (Fortran) code with it. I'm > getting compilation errors when it gets to calls to PetscSortInt(): > > ? 567 |????????? call PetscSortInt(na, asort, ierr); CHKERRQ(ierr) > ????? |??????????????????????????????????????????? 1 > Error: Type mismatch in argument ?a? at (1); passed INTEGER(4) to INTEGER(8) > > Here na is PetscInt, asort an array of PetscInt and ierr is PetscErrorCode. > > Do I need to make changes to get this to work again? > > - Adrian > > From a.croucher at auckland.ac.nz Mon Sep 30 21:14:38 2024 From: a.croucher at auckland.ac.nz (Adrian Croucher) Date: Tue, 1 Oct 2024 15:14:38 +1300 Subject: [petsc-users] ISView() in PETSc 3.22 Message-ID: hi, I am testing my (Fortran) code on PETSc 3.22 and have got it to build. However I am getting some unusual new behaviour when I write an IS to an HDF5 file using ISView(). The attached minimal example shows the issue. It creates a simple 10-element IS and writes it to HDF5. With previous versions of PETSc this would give me a 10x1 dataset containing the values 0 - 9, as expected. When I run it with PETSc 3.22 (in serial), I again get the expected values written on stdout, so it looks like the IS itself is correct. But in the HDF5 file I get a 1x3 dataset containing the values (10,1,0). Has something changed here? - Adrian -- Dr Adrian Croucher Senior Research Fellow Department of Engineering Science Waipapa Taumata Rau / University of Auckland, New Zealand email: a.croucher at auckland.ac.nz tel: +64 (0)9 923 4611 -------------- next part -------------- A non-text attachment was scrubbed... Name: test_is.F90 Type: text/x-fortran Size: 817 bytes Desc: not available URL: