From stefano.zampini at gmail.com Wed Jun 2 14:43:43 2021 From: stefano.zampini at gmail.com (Stefano Zampini) Date: Wed, 2 Jun 2021 22:43:43 +0300 Subject: [petsc-users] SLEPc: non-real singular vectors from SVD of real matrix In-Reply-To: <4d664d256ddb4d24bd0ec9cf58f6d32a@mek.dtu.dk> References: <6129ddcc3238490e805cfb33019b7ecf@mek.dtu.dk> <4d664d256ddb4d24bd0ec9cf58f6d32a@mek.dtu.dk> Message-ID: <407C3235-DEA8-4A9D-BDE8-C746D7FEFBB5@gmail.com> Peder We have a fix for the hdf5 complex reader here if you want to give it a try https://gitlab.com/petsc/petsc/-/merge_requests/4044 Sorry it took so long and thank you for reporting the bug > On Apr 29, 2021, at 2:55 PM, Peder J?rgensgaard Olesen via petsc-users wrote: > > Thank you for your advice, Jose. > > I tried using a single column of the data matrix as a basis for left singular vector space, and a row for the right one - and lo and behold, all singular vectors become real. > > Somehow it did not occur to me that I was I was looking at was simply a complex phase on U and V which would cancel upon computing U?V*. In other words, rather than an incorrect result it was a slightly inconvenient representation of a correct result. > > Best regards > Peder > Fra: Jose E. Roman > Sendt: 29. april 2021 13:00:03 > Til: Peder J?rgensgaard Olesen > Cc: petsc-users at mcs.anl.gov > Emne: Re: [petsc-users] SLEPc: non-real singular vectors from SVD of real matrix > > In complex scalars there is no way of knowing if the user-provided matrix is real or not. If there was an option of MatSetOption() we could use it. > > If you set a real initial vector with SVDSetInitialSpaces(), then most probably the computed singular vectors will be real. But still rounding errors could introduce a nonzero imaginary part. > > In any case, you could normalize the computed singular vectors, as is done for instance in FixSign() in this examplehttps://slepc.upv.es/documentation/current/src/nep/tutorials/ex20.c.html > > Jose > > > > El 29 abr 2021, a las 12:42, Peder J?rgensgaard Olesen via petsc-users escribi?: > > > > Hello > > > > I've noticed that doing a singular value decomposition of a real matrix appears to result in non-real singular vectors. This should not be the case - singular vectors of a real matrix must be real-valued. > > > > In the example attached I read a matrix from a binary file (also attached), perform the SVD, and write singular vectors to an HDF5 file which I subsequently inspect using h5dump, revealing non-zero imaginary parts of vector elements as highlighted below: > > [pjool at svol mwes]$ h5dump -d U0_sample_svd -c "5,2" sample_svecs.h5 > > HDF5 "sample_svecs.h5" { > > DATASET "U0_sample_svd" { > > DATATYPE H5T_IEEE_F64LE > > DATASPACE SIMPLE { ( 882, 2 ) / ( 882, 2 ) } > > SUBSET { > > START ( 0, 0 ); > > STRIDE ( 1, 1 ); > > COUNT ( 5, 2 ); > > BLOCK ( 1, 1 ); > > DATA { > > (0,0): 0.0226108, 0.0299595, > > (1,0): 0.035414, 0.0469237, > > (2,0): 0.0276317, 0.0366122, > > (3,0): 0.0145344, 0.0192581, > > (4,0): 0.0110376, 0.0146249 > > } > > } > > ATTRIBUTE "complex" { > > DATATYPE H5T_STD_I32LE > > DATASPACE SCALAR > > DATA { > > (0): 1 > > } > > } > > } > > } > > I also extract the imaginary part of the input matrix and print its norm to ensure that the matrix is indeed real. > > > > I'm running v3.14, but I don't believe that alone should cause the issue, since it what I'm trying to do appears like a rather common and basic task. > > > > What might be the reason behind this behavior, and what can be done to resolve it? > > > > > > Med venlig hilsen / Best Regards > > > > Peder J?rgensgaard Olesen > > PhD Student, Turbulence Research Lab > > Dept. of Mechanical Engineering > > Technical University of Denmark > > Niels Koppels All? > > Bygning 403, Rum 105 > > DK-2800 Kgs. Lyngby > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pjool at mek.dtu.dk Wed Jun 2 16:11:03 2021 From: pjool at mek.dtu.dk (=?utf-8?B?UGVkZXIgSsO4cmdlbnNnYWFyZCBPbGVzZW4=?=) Date: Wed, 2 Jun 2021 21:11:03 +0000 Subject: [petsc-users] SLEPc: non-real singular vectors from SVD of real matrix In-Reply-To: <407C3235-DEA8-4A9D-BDE8-C746D7FEFBB5@gmail.com> References: <6129ddcc3238490e805cfb33019b7ecf@mek.dtu.dk> <4d664d256ddb4d24bd0ec9cf58f6d32a@mek.dtu.dk>, <407C3235-DEA8-4A9D-BDE8-C746D7FEFBB5@gmail.com> Message-ID: Hello Stefano I believe this was an issue that was discussed in another thread opened by me (this one I think: https://lists.mcs.anl.gov/pipermail/petsc-users/2021-April/043629.html). In any case thanks for fixing it, and glad to be of service! - Peder ________________________________ Fra: Stefano Zampini Sendt: 2. juni 2021 21:43:43 Til: Peder J?rgensgaard Olesen Cc: Jose E. Roman; petsc-users at mcs.anl.gov Emne: Re: [petsc-users] SLEPc: non-real singular vectors from SVD of real matrix Peder We have a fix for the hdf5 complex reader here if you want to give it a try https://gitlab.com/petsc/petsc/-/merge_requests/4044 [https://assets.gitlab-static.net/uploads/-/system/project/avatar/13882401/PETSc_RBG-logo.png] fix MatLoad_Dense_HDF5 (!4044) ? Merge requests ? PETSc / petsc ? GitLab gitlab.com follow-up !3903 add src/mat/tests/ex84.c... Sorry it took so long and thank you for reporting the bug On Apr 29, 2021, at 2:55 PM, Peder J?rgensgaard Olesen via petsc-users > wrote: Thank you for your advice, Jose. I tried using a single column of the data matrix as a basis for left singular vector space, and a row for the right one - and lo and behold, all singular vectors become real. Somehow it did not occur to me that I was I was looking at was simply a complex phase on U and V which would cancel upon computing U?V*. In other words, rather than an incorrect result it was a slightly inconvenient representation of a correct result. Best regards Peder ________________________________ Fra: Jose E. Roman > Sendt: 29. april 2021 13:00:03 Til: Peder J?rgensgaard Olesen Cc: petsc-users at mcs.anl.gov Emne: Re: [petsc-users] SLEPc: non-real singular vectors from SVD of real matrix In complex scalars there is no way of knowing if the user-provided matrix is real or not. If there was an option of MatSetOption() we could use it. If you set a real initial vector with SVDSetInitialSpaces(), then most probably the computed singular vectors will be real. But still rounding errors could introduce a nonzero imaginary part. In any case, you could normalize the computed singular vectors, as is done for instance in FixSign() in this examplehttps://slepc.upv.es/documentation/current/src/nep/tutorials/ex20.c.html Jose > El 29 abr 2021, a las 12:42, Peder J?rgensgaard Olesen via petsc-users > escribi?: > > Hello > > I've noticed that doing a singular value decomposition of a real matrix appears to result in non-real singular vectors. This should not be the case - singular vectors of a real matrix must be real-valued. > > In the example attached I read a matrix from a binary file (also attached), perform the SVD, and write singular vectors to an HDF5 file which I subsequently inspect using h5dump, revealing non-zero imaginary parts of vector elements as highlighted below: > [pjool at svol mwes]$ h5dump -d U0_sample_svd -c "5,2" sample_svecs.h5 > HDF5 "sample_svecs.h5" { > DATASET "U0_sample_svd" { > DATATYPE H5T_IEEE_F64LE > DATASPACE SIMPLE { ( 882, 2 ) / ( 882, 2 ) } > SUBSET { > START ( 0, 0 ); > STRIDE ( 1, 1 ); > COUNT ( 5, 2 ); > BLOCK ( 1, 1 ); > DATA { > (0,0): 0.0226108, 0.0299595, > (1,0): 0.035414, 0.0469237, > (2,0): 0.0276317, 0.0366122, > (3,0): 0.0145344, 0.0192581, > (4,0): 0.0110376, 0.0146249 > } > } > ATTRIBUTE "complex" { > DATATYPE H5T_STD_I32LE > DATASPACE SCALAR > DATA { > (0): 1 > } > } > } > } > I also extract the imaginary part of the input matrix and print its norm to ensure that the matrix is indeed real. > > I'm running v3.14, but I don't believe that alone should cause the issue, since it what I'm trying to do appears like a rather common and basic task. > > What might be the reason behind this behavior, and what can be done to resolve it? > > > Med venlig hilsen / Best Regards > > Peder J?rgensgaard Olesen > PhD Student, Turbulence Research Lab > Dept. of Mechanical Engineering > Technical University of Denmark > Niels Koppels All? > Bygning 403, Rum 105 > DK-2800 Kgs. Lyngby > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vaclav.hapla at erdw.ethz.ch Thu Jun 3 05:04:23 2021 From: vaclav.hapla at erdw.ethz.ch (Hapla Vaclav) Date: Thu, 3 Jun 2021 10:04:23 +0000 Subject: [petsc-users] Rather different matrix product results on multiple processes In-Reply-To: <21ea5d3641634372aafd2fecea936c0d@mek.dtu.dk> References: <6a896a1c5a6148948a1e3d2dddeeb2a3@mek.dtu.dk> <41f4e3f95d504668ae9518693aaf202a@mek.dtu.dk> <21ea5d3641634372aafd2fecea936c0d@mek.dtu.dk> Message-ID: Dear Peder The problem with HDF5 MATDENSE loader should be fixed now in the main branch. Your datafile is now stored at https://gitlab.com/petsc/datafiles/-/blob/master/matrices/hdf5/sample_data.h5 if you are ok with that, and is used in the src/mat/tests/ex84.c test. Let me note that 1) PETSc uses "Fortran storage convention" (= column-major) for dense matrices. However, HDF5 uses "C storage convention" (= row-major), assuming that the last listed dimension is the fastest-changing dimension and the first-listed dimension is the slowest changing. [https://support.hdfgroup.org/HDF5/doc/UG/HDF5_Users_Guide-Responsive%20HTML5/index.html#t=HDF5_Users_Guide%2FDataspaces%2FHDF5_Dataspaces_and_Partial_I_O.htm%3Frhhlterm%3D%2522last%2520listed%2520dimension%2522%26rhsyns%3D%2520] Hence, we decide to store dense matrices "transposed", i.e. dimension 0 is columns and dimension 1 is rows. So a matrix whose h5dump is DATASET "B" { DATATYPE H5T_IEEE_F64LE DATASPACE SIMPLE { ( 3, 4 ) / ( 3, 4 ) } DATA { (0,0): 1, 4, 7, 10, (1,0): 2, 5, 8, 11, (2,0): 3, 6, 9, 12 } will be loaded as [ 1 2 3 4 5 6 7 8 9 10 11 12] Another reason for this is to simplify compatibility with MATLAB. Real/imaginary part of complex numbers is the last dimension in any case. Please check whether your dataset should be transposed. 2) There was a bug - wrong interpretation of dimensions if "MATLAB_class" attribute was not present - resolved in the merge request 4044. 3) Complex numbers were not really supported which is now fixed in the same MR. 4) An unfortunate thing is there is currently no MatView() implementation for MATDENSE and HDF5 which would easily show how the datafile should look like. I hope to fix this soon as well. Sorry for the delay and thanks for reporting. Vaclav Hapla On 22 Apr 2021, at 11:05, Peder J?rgensgaard Olesen via petsc-users > wrote: Dear Stefano and Jose Thank you for your replies. Using SVD works like a charm. I'll try to do some trickery to work around the HDF5 reader bug. Best regards Peder ________________________________ Fra: Jose E. Roman > Sendt: 21. april 2021 14:24:38 Til: Peder J?rgensgaard Olesen Cc: petsc-users at mcs.anl.gov; Stefano Zampini Emne: Re: [petsc-users] Rather different matrix product results on multiple processes Independently of the bug mentioned by Stefano, you may want to consider using SLEPc's SVD instead of EPS. Left singular vectors of D are equal to eigenvectors of D*D', see chapter 4 of SLEPc's users manual. The default solver 'cross' gives you flexibility to compute the product D*D' explicitly or not, and build the transpose explicitly or not. Jose > El 21 abr 2021, a las 12:54, Stefano Zampini > escribi?: > > Here you have, https://gitlab.com/petsc/petsc/-/merge_requests/3903. We can discuss the issue on gitlab. > > Thanks > Stefano > > Il giorno mer 21 apr 2021 alle ore 13:39 Stefano Zampini > ha scritto: > Peder > > I have slightly modified your code and I confirm the bug. > The bug is not with the MatMatTranspose operation; it is within the HDF5 reader. I will soon open an MR with the code and discussing the issues. > > Thanks for reporting the issue > Stefano > > Il giorno mer 21 apr 2021 alle ore 12:22 Peder J?rgensgaard Olesen via petsc-users > ha scritto: > Dear Hong > > > > Thank your for your reply. > > > > I have a hunch that the issue goes beyond the minor differences that might arise from floating-point computation order, however. > > > > Writing the product matrix to a binary file using MatView() and inspecting the output shows very different entries depending on the number of processes. Here are the first three rows and columns of the product matrix obtained in a sequential run: > > 2.58348 1.68202 1.66302 > > 1.68202 4.27506 1.91897 > > 1.66302 1.91897 2.70028 > > > > - and the corresponding part of the product matrix obtained on one node (40 processes): > > 4.43536 2.17261 0.16430 > > 2.17261 4.53224 2.53210 > > 0.16430 2.53210 4.73234 > > > > The parallel result is not even close to the sequential one. Trying different numbers of processes produces yet different results. > > > > Also, the eigenvectors that I subsequently determine using a SLEPC solver do not form a proper basis for the column space of the data matrix as they must, which is hardly a surprise given the variability of results indicated above - except when the code is run on just a single process. Forming such a basis central to the intended application, and given that it would need to work on rather large data sets, running on a single process is hardly a viable solution. > > > > Best regards > > Peder > > Fra: Zhang, Hong > > Sendt: 19. april 2021 18:34:31 > Til: petsc-users at mcs.anl.gov; Peder J?rgensgaard Olesen > Emne: Re: Rather different matrix product results on multiple processes > > Peder, > I tested your code on a linux machine. I got > $ ./acorr_mwe > Data matrix norm: 5.0538e+01 > Autocorrelation matrix norm: 1.0473e+03 > > mpiexec -n 40 ./acorr_mwe -matmattransmult_mpidense_mpidense_via allgatherv (default) > Data matrix norm: 5.0538e+01 > Autocorrelation matrix norm: 1.0363e+03 > > mpiexec -n 20 ./acorr_mwe > Data matrix norm: 5.0538e+01 > Autocorrelation matrix norm: 1.0897e+03 > > mpiexec -n 40 ./acorr_mwe -matmattransmult_mpidense_mpidense_via cyclic > Data matrix norm: 5.0538e+01 > Autocorrelation matrix norm: 1.0363e+03 > > I use petsc 'main' branch (same as the latest release). You can remove MatAssemblyBegin/End calls after MatMatTransposeMult(): > MatMatTransposeMult(data_mat, data_mat, MAT_INITIAL_MATRIX, PETSC_DEFAULT, &corr_mat); > //ierr = MatAssemblyBegin(corr_mat, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); > //ierr = MatAssemblyEnd(corr_mat, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); > > The communication patterns of parallel implementation led to different order of floating-point computation, thus slightly different matrix norm of R. > Hong > > From: petsc-users > on behalf of Peder J?rgensgaard Olesen via petsc-users > > Sent: Monday, April 19, 2021 7:57 AM > To: petsc-users at mcs.anl.gov > > Subject: [petsc-users] Rather different matrix product results on multiple processes > > Hello, > > When computing a matrix product of the type R = D.DT using MatMatTransposeMult() I find I get rather different results depending on the number of processes. In one example using a data set that is small compared to the application I get Frobenius norms |R| = 1.047e3 on a single process, 1.0363e3 on a single HPC node (40 cores), and 9.7307e2 on two nodes. > > I have ascertained that the single process result is indeed the correct one (i.e., eigenvectors of R form a proper basis for the columns of D), so naturally I'd love to be able to reproduce this result across different parallel setups. How might I achieve this? > > I'm attaching MWE code and the data set used for the example. > > Thanks in advance! > > Best Regards > > Peder J?rgensgaard Olesen > PhD Student, Turbulence Research Lab > Dept. of Mechanical Engineering > Technical University of Denmark > Niels Koppels All? > Bygning 403, Rum 105 > DK-2800 Kgs. Lyngby > > > -- > Stefano > > > -- > Stefano -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Thu Jun 3 08:30:42 2021 From: mfadams at lbl.gov (Mark Adams) Date: Thu, 3 Jun 2021 09:30:42 -0400 Subject: [petsc-users] ex5k.kokkos compile error Message-ID: I am getting this error: 09:22 adams/landau-mass-opt= /gpfs/alpine/csc314/scratch/adams/petsc/src/mat/tutorials$ make PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc PETSC_ARCH=arch-summit-opt-gnu-kokkos-notpl-cuda10 ex5k.kokkos mpicxx -fPIC -g -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 -O0 -fPIC -g -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 -O0 -fPIC -I/gpfs/alpine/csc314/scratch/adams/petsc/include -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/include -I/sw/summit/cuda/10.1.243/include ex5k.kokkos.cxx -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib -L/autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/netlib-lapack-3.8.0-wcabdyqhdi5rooxbkqa6x5d7hxyxwdkm/lib64 -Wl,-rpath,/sw/summit/cuda/10.1.243/lib64 -L/sw/summit/cuda/10.1.243/lib64 -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lp4est -lsc -lblas -llapack -ltriangle -lm -lz -lcudart -lcufft -lcublas -lcusparse -lcusolver -lcurand -lstdc++ -ldl -o ex5k.kokkos In file included from /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/include/KokkosCore_Config_SetupBackend.hpp:47:0, from /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/include/Kokkos_Macros.hpp:109, from /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/include/Kokkos_Core_fwd.hpp:52, from /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/include/Kokkos_Core.hpp:51, from ex5k.kokkos.cxx:10: /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/include/setup/Kokkos_Setup_Cuda.hpp:54:2: error: #error "KOKKOS_ENABLE_CUDA defined but the compiler is not defining the __CUDACC__ macro as expected" #error \ ^~~~~ In file included from /sw/summit/cuda/10.1.243/include/crt/common_functions.h:267:0, from /sw/summit/cuda/10.1.243/include/cuda_runtime.h:115, from /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/include/setup/Kokkos_Setup_Cuda.hpp:68, from /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/include/KokkosCore_Config_SetupBackend.hpp:47, from /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/include/Kokkos_Macros.hpp:109, from /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/include/Kokkos_Core_fwd.hpp:52, from /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/include/Kokkos_Core.hpp:51, from ex5k.kokkos.cxx:10: /sw/summit/cuda/10.1.243/include/crt/math_functions.h:8944:72: error: 'int std::isnan(double)' conflicts with a previous declaration __DEVICE_FUNCTIONS_DECL__ __cudart_builtin__ int isnan(double x) throw(); ^ Here is my config file: #!/usr/bin/env python # -lnvToolsExt if __name__ == '__main__': import sys import os sys.path.insert(0, os.path.abspath('config')) import configure configure_options = [ '--with-fc=0', '--CFLAGS=-fPIC -g -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4', '--CXXFLAGS=-fPIC -g -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4', '--COPTFLAGS=-O0', '--CXXOPTFLAGS=-O0', '--CUDAOPTFLAGS=-O0', # 'LIBS= -L'+os.environ['CUDAPATH']+'/lib64 -lnvToolsExt', '--CUDAFLAGS=-Xcompiler -rdynamic -lineinfo -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4', '--with-ssl=0', '--with-batch=0', '--with-cxx=mpicxx', '--with-mpiexec=jsrun -g1 ', '--with-openmp=0', # '--with-threadsafety=1', # '--with-log=0', '--with-cuda=1', '--with-cuda-gencodearch=70', '--with-cudac=nvcc', '--download-p4est=1', '--download-zlib', # '--download-hdf5=1', # '--download-metis', # '--download-superlu_dist', # '--download-superlu_dist-commit=HEAD', # '--download-hypre-configure-arguments=HYPRE_CUDA_SM=70', #'--with-hwloc=0', # '--download-parmetis', #'--download-hypre', '--download-triangle', #'--download-amgx', #'--download-fblaslapack', '--with-blaslapack-lib=-L' + os.environ['OLCF_NETLIB_LAPACK_ROOT'] + '/lib64 -lblas -llapack', #'--download-openblas', '--with-cc=mpicc', #'--with-fc=mpif90', '--with-shared-libraries=1', # '--known-mpi-shared-libraries=1', '--with-x=0', '--with-64-bit-indices=0', '--with-debugging=1', '--download-kokkos', '--with-ctable=0', '--with-make-np=8', '--download-kokkos-kernels', # '--with-kokkos-cuda-arch=VOLTA70', '--with-kokkos-kernels-tpl=0', '--download-kokkos-cmake-arguments=-DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON -DKokkos_ARCH_POWER9=ON', # '--download-kokkos-commit=origin/develop', # '--download-kokkos-kernels-commit=origin/develop', 'PETSC_ARCH=arch-summit-opt-gnu-kokkos-notpl-cuda10', ] configure.petsc_configure(configure_options) ~ ~ ~ ~ ~ ~ ~ -------------- next part -------------- An HTML attachment was scrubbed... URL: From pjool at mek.dtu.dk Fri Jun 4 06:10:12 2021 From: pjool at mek.dtu.dk (=?iso-8859-1?Q?Peder_J=F8rgensgaard_Olesen?=) Date: Fri, 4 Jun 2021 11:10:12 +0000 Subject: [petsc-users] SLEPc: HDF5 support for SVD/EPSValuesView? Message-ID: <56981a2017944684a158b8736a6c5023@mek.dtu.dk> Hello In SLEPc one may write singular vectors to an HDF5 file using SVDVectorsView(), but a similar option doesn't seem to work for singular values using SVDValuesView(). The values are viewed correctly using other viewers, but nothing is seemingly produced when using an HDF5 viewer. Looking at the source code seems to confirm this, and suggests that there is a similar situation with EPS. Is this correct, and if so, might there be plans to include such functionality in future releases? Med venlig hilsen / Best Regards Peder J?rgensgaard Olesen PhD Student, Turbulence Research Lab Dept. of Mechanical Engineering Technical University of Denmark Koppels All? Bygning 403, Rum 105 DK-2800 Kgs. Lyngby -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Fri Jun 4 08:30:18 2021 From: jroman at dsic.upv.es (Jose E. Roman) Date: Fri, 4 Jun 2021 15:30:18 +0200 Subject: [petsc-users] SLEPc: HDF5 support for SVD/EPSValuesView? In-Reply-To: <56981a2017944684a158b8736a6c5023@mek.dtu.dk> References: <56981a2017944684a158b8736a6c5023@mek.dtu.dk> Message-ID: I could try and adapt the HDF5 code in PETSc for this, but I am no HDF5 expert. Furthermore I am busy at the moment, so it is faster if you could prepare a merge request with this addition. Jose > El 4 jun 2021, a las 13:10, Peder J?rgensgaard Olesen via petsc-users escribi?: > > Hello > > In SLEPc one may write singular vectors to an HDF5 file using SVDVectorsView(), but a similar option doesn't seem to work for singular values using SVDValuesView(). The values are viewed correctly using other viewers, but nothing is seemingly produced when using an HDF5 viewer. Looking at the source code seems to confirm this, and suggests that there is a similar situation with EPS. > > Is this correct, and if so, might there be plans to include such functionality in future releases? > > Med venlig hilsen / Best Regards > > Peder J?rgensgaard Olesen > PhD Student, Turbulence Research Lab > Dept. of Mechanical Engineering > Technical University of Denmark > Koppels All? > Bygning 403, Rum 105 > DK-2800 Kgs. Lyngby From knepley at gmail.com Fri Jun 4 08:56:06 2021 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 4 Jun 2021 09:56:06 -0400 Subject: [petsc-users] SLEPc: HDF5 support for SVD/EPSValuesView? In-Reply-To: References: <56981a2017944684a158b8736a6c5023@mek.dtu.dk> Message-ID: On Fri, Jun 4, 2021 at 9:31 AM Jose E. Roman wrote: > I could try and adapt the HDF5 code in PETSc for this, but I am no HDF5 > expert. Furthermore I am busy at the moment, so it is faster if you could > prepare a merge request with this addition. > I think the easiest thing is to wrap a Vec around the values and just VecView() it into the same HDF5 file. THanks, Matt > Jose > > > > El 4 jun 2021, a las 13:10, Peder J?rgensgaard Olesen via petsc-users < > petsc-users at mcs.anl.gov> escribi?: > > > > Hello > > > > In SLEPc one may write singular vectors to an HDF5 file using > SVDVectorsView(), but a similar option doesn't seem to work for singular > values using SVDValuesView(). The values are viewed correctly using other > viewers, but nothing is seemingly produced when using an HDF5 viewer. > Looking at the source code seems to confirm this, and suggests that there > is a similar situation with EPS. > > > > Is this correct, and if so, might there be plans to include such > functionality in future releases? > > > > Med venlig hilsen / Best Regards > > > > Peder J?rgensgaard Olesen > > PhD Student, Turbulence Research Lab > > Dept. of Mechanical Engineering > > Technical University of Denmark > > Koppels All? > > Bygning 403, Rum 105 > > DK-2800 Kgs. Lyngby > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Fri Jun 4 09:01:04 2021 From: jroman at dsic.upv.es (Jose E. Roman) Date: Fri, 4 Jun 2021 16:01:04 +0200 Subject: [petsc-users] SLEPc: HDF5 support for SVD/EPSValuesView? In-Reply-To: References: <56981a2017944684a158b8736a6c5023@mek.dtu.dk> Message-ID: <1A99ADA9-DBF2-428A-BA95-999E222D531B@dsic.upv.es> The problem is that here I am writing PetscReal's not PetscScalar's, and in EPS I am writing PetscComplex even in real scalars. That is why I did not implement it as you are suggesting. Jose > El 4 jun 2021, a las 15:56, Matthew Knepley escribi?: > > On Fri, Jun 4, 2021 at 9:31 AM Jose E. Roman wrote: > I could try and adapt the HDF5 code in PETSc for this, but I am no HDF5 expert. Furthermore I am busy at the moment, so it is faster if you could prepare a merge request with this addition. > > I think the easiest thing is to wrap a Vec around the values and just VecView() it into the same HDF5 file. > > THanks, > > Matt > > Jose > > > > El 4 jun 2021, a las 13:10, Peder J?rgensgaard Olesen via petsc-users escribi?: > > > > Hello > > > > In SLEPc one may write singular vectors to an HDF5 file using SVDVectorsView(), but a similar option doesn't seem to work for singular values using SVDValuesView(). The values are viewed correctly using other viewers, but nothing is seemingly produced when using an HDF5 viewer. Looking at the source code seems to confirm this, and suggests that there is a similar situation with EPS. > > > > Is this correct, and if so, might there be plans to include such functionality in future releases? > > > > Med venlig hilsen / Best Regards > > > > Peder J?rgensgaard Olesen > > PhD Student, Turbulence Research Lab > > Dept. of Mechanical Engineering > > Technical University of Denmark > > Koppels All? > > Bygning 403, Rum 105 > > DK-2800 Kgs. Lyngby > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ From pjool at mek.dtu.dk Fri Jun 4 09:28:56 2021 From: pjool at mek.dtu.dk (=?iso-8859-1?Q?Peder_J=F8rgensgaard_Olesen?=) Date: Fri, 4 Jun 2021 14:28:56 +0000 Subject: [petsc-users] SLEPc: HDF5 support for SVD/EPSValuesView? In-Reply-To: <1A99ADA9-DBF2-428A-BA95-999E222D531B@dsic.upv.es> References: <56981a2017944684a158b8736a6c5023@mek.dtu.dk> , <1A99ADA9-DBF2-428A-BA95-999E222D531B@dsic.upv.es> Message-ID: <95f1ad68b5114231bd2e1239d4936a5f@mek.dtu.dk> What Matt suggests could perhaps be used as a workaround, though it appears neither elegant nor in my experience efficient. I may be wrong on this. My attempted solution was to use a binary viewer instead, but I can't find a clear way to retrieve values thus stored with PETSc, as these do not lend themselves to be read using VecLoad. Again one might go about that by wrapping them inside a vector and send that to the viewer, which just brings us back around to the original problem. - Peder ________________________________ Fra: Jose E. Roman Sendt: 4. juni 2021 16:01:04 Til: Matthew Knepley Cc: Peder J?rgensgaard Olesen; petsc-users at mcs.anl.gov Emne: Re: [petsc-users] SLEPc: HDF5 support for SVD/EPSValuesView? The problem is that here I am writing PetscReal's not PetscScalar's, and in EPS I am writing PetscComplex even in real scalars. That is why I did not implement it as you are suggesting. Jose > El 4 jun 2021, a las 15:56, Matthew Knepley escribi?: > > On Fri, Jun 4, 2021 at 9:31 AM Jose E. Roman wrote: > I could try and adapt the HDF5 code in PETSc for this, but I am no HDF5 expert. Furthermore I am busy at the moment, so it is faster if you could prepare a merge request with this addition. > > I think the easiest thing is to wrap a Vec around the values and just VecView() it into the same HDF5 file. > > THanks, > > Matt > > Jose > > > > El 4 jun 2021, a las 13:10, Peder J?rgensgaard Olesen via petsc-users escribi?: > > > > Hello > > > > In SLEPc one may write singular vectors to an HDF5 file using SVDVectorsView(), but a similar option doesn't seem to work for singular values using SVDValuesView(). The values are viewed correctly using other viewers, but nothing is seemingly produced when using an HDF5 viewer. Looking at the source code seems to confirm this, and suggests that there is a similar situation with EPS. > > > > Is this correct, and if so, might there be plans to include such functionality in future releases? > > > > Med venlig hilsen / Best Regards > > > > Peder J?rgensgaard Olesen > > PhD Student, Turbulence Research Lab > > Dept. of Mechanical Engineering > > Technical University of Denmark > > Koppels All? > > Bygning 403, Rum 105 > > DK-2800 Kgs. Lyngby > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Jun 4 09:48:05 2021 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 4 Jun 2021 10:48:05 -0400 Subject: [petsc-users] SLEPc: HDF5 support for SVD/EPSValuesView? In-Reply-To: <95f1ad68b5114231bd2e1239d4936a5f@mek.dtu.dk> References: <56981a2017944684a158b8736a6c5023@mek.dtu.dk> <1A99ADA9-DBF2-428A-BA95-999E222D531B@dsic.upv.es> <95f1ad68b5114231bd2e1239d4936a5f@mek.dtu.dk> Message-ID: On Fri, Jun 4, 2021 at 10:28 AM Peder J?rgensgaard Olesen wrote: > What Matt suggests could perhaps be used as a workaround, though it > appears neither elegant nor in my experience efficient. I may be wrong on > this. > Why is it not efficient? Thanks, Matt > My attempted solution was to use a binary viewer instead, but I can't > find a clear way to retrieve values thus stored with PETSc, as these do not > lend themselves to be read using VecLoad. Again one might go about that > by wrapping them inside a vector and send that to the viewer, which just > brings us back around to the original problem. > > > - Peder > ------------------------------ > *Fra:* Jose E. Roman > *Sendt:* 4. juni 2021 16:01:04 > *Til:* Matthew Knepley > *Cc:* Peder J?rgensgaard Olesen; petsc-users at mcs.anl.gov > *Emne:* Re: [petsc-users] SLEPc: HDF5 support for SVD/EPSValuesView? > > The problem is that here I am writing PetscReal's not PetscScalar's, and > in EPS I am writing PetscComplex even in real scalars. That is why I did > not implement it as you are suggesting. > > Jose > > > > El 4 jun 2021, a las 15:56, Matthew Knepley > escribi?: > > > > On Fri, Jun 4, 2021 at 9:31 AM Jose E. Roman wrote: > > I could try and adapt the HDF5 code in PETSc for this, but I am no HDF5 > expert. Furthermore I am busy at the moment, so it is faster if you could > prepare a merge request with this addition. > > > > I think the easiest thing is to wrap a Vec around the values and just > VecView() it into the same HDF5 file. > > > > THanks, > > > > Matt > > > > Jose > > > > > > > El 4 jun 2021, a las 13:10, Peder J?rgensgaard Olesen via petsc-users < > petsc-users at mcs.anl.gov> escribi?: > > > > > > Hello > > > > > > In SLEPc one may write singular vectors to an HDF5 file using > SVDVectorsView(), but a similar option doesn't seem to work for singular > values using SVDValuesView(). The values are viewed correctly using other > viewers, but nothing is seemingly produced when using an HDF5 viewer. > Looking at the source code seems to confirm this, and suggests that there > is a similar situation with EPS. > > > > > > Is this correct, and if so, might there be plans to include such > functionality in future releases? > > > > > > Med venlig hilsen / Best Regards > > > > > > Peder J?rgensgaard Olesen > > > PhD Student, Turbulence Research Lab > > > Dept. of Mechanical Engineering > > > Technical University of Denmark > > > Koppels All? > > > Bygning 403, Rum 105 > > > DK-2800 Kgs. Lyngby > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From pjool at mek.dtu.dk Fri Jun 4 09:59:26 2021 From: pjool at mek.dtu.dk (=?iso-8859-1?Q?Peder_J=F8rgensgaard_Olesen?=) Date: Fri, 4 Jun 2021 14:59:26 +0000 Subject: [petsc-users] SLEPc: HDF5 support for SVD/EPSValuesView? In-Reply-To: References: <56981a2017944684a158b8736a6c5023@mek.dtu.dk> <1A99ADA9-DBF2-428A-BA95-999E222D531B@dsic.upv.es> <95f1ad68b5114231bd2e1239d4936a5f@mek.dtu.dk>, Message-ID: Excellent question. It may well be some quirk of my own setup that somehow makes VecView run rather sluggishly, making it hardly relevant to the general case. - Peder ________________________________ Fra: Matthew Knepley Sendt: 4. juni 2021 16:48:05 Til: Peder J?rgensgaard Olesen Cc: Jose E. Roman; petsc-users at mcs.anl.gov Emne: Re: [petsc-users] SLEPc: HDF5 support for SVD/EPSValuesView? On Fri, Jun 4, 2021 at 10:28 AM Peder J?rgensgaard Olesen > wrote: What Matt suggests could perhaps be used as a workaround, though it appears neither elegant nor in my experience efficient. I may be wrong on this. Why is it not efficient? Thanks, Matt My attempted solution was to use a binary viewer instead, but I can't find a clear way to retrieve values thus stored with PETSc, as these do not lend themselves to be read using VecLoad. Again one might go about that by wrapping them inside a vector and send that to the viewer, which just brings us back around to the original problem. - Peder ________________________________ Fra: Jose E. Roman > Sendt: 4. juni 2021 16:01:04 Til: Matthew Knepley Cc: Peder J?rgensgaard Olesen; petsc-users at mcs.anl.gov Emne: Re: [petsc-users] SLEPc: HDF5 support for SVD/EPSValuesView? The problem is that here I am writing PetscReal's not PetscScalar's, and in EPS I am writing PetscComplex even in real scalars. That is why I did not implement it as you are suggesting. Jose > El 4 jun 2021, a las 15:56, Matthew Knepley > escribi?: > > On Fri, Jun 4, 2021 at 9:31 AM Jose E. Roman > wrote: > I could try and adapt the HDF5 code in PETSc for this, but I am no HDF5 expert. Furthermore I am busy at the moment, so it is faster if you could prepare a merge request with this addition. > > I think the easiest thing is to wrap a Vec around the values and just VecView() it into the same HDF5 file. > > THanks, > > Matt > > Jose > > > > El 4 jun 2021, a las 13:10, Peder J?rgensgaard Olesen via petsc-users > escribi?: > > > > Hello > > > > In SLEPc one may write singular vectors to an HDF5 file using SVDVectorsView(), but a similar option doesn't seem to work for singular values using SVDValuesView(). The values are viewed correctly using other viewers, but nothing is seemingly produced when using an HDF5 viewer. Looking at the source code seems to confirm this, and suggests that there is a similar situation with EPS. > > > > Is this correct, and if so, might there be plans to include such functionality in future releases? > > > > Med venlig hilsen / Best Regards > > > > Peder J?rgensgaard Olesen > > PhD Student, Turbulence Research Lab > > Dept. of Mechanical Engineering > > Technical University of Denmark > > Koppels All? > > Bygning 403, Rum 105 > > DK-2800 Kgs. Lyngby > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Jun 4 10:38:39 2021 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 4 Jun 2021 11:38:39 -0400 Subject: [petsc-users] SLEPc: HDF5 support for SVD/EPSValuesView? In-Reply-To: References: <56981a2017944684a158b8736a6c5023@mek.dtu.dk> <1A99ADA9-DBF2-428A-BA95-999E222D531B@dsic.upv.es> <95f1ad68b5114231bd2e1239d4936a5f@mek.dtu.dk> Message-ID: On Fri, Jun 4, 2021 at 10:59 AM Peder J?rgensgaard Olesen wrote: > Excellent question. It may well be some quirk of my own setup that > somehow makes VecView run rather sluggishly, making it > hardly relevant to the general case. > Maybe I can help speed things up for you. Is this in serial or parallel? Is the vector big or small? Is it possible to send the output of -log_view? Thanks Matt > - Peder > ------------------------------ > *Fra:* Matthew Knepley > *Sendt:* 4. juni 2021 16:48:05 > *Til:* Peder J?rgensgaard Olesen > *Cc:* Jose E. Roman; petsc-users at mcs.anl.gov > *Emne:* Re: [petsc-users] SLEPc: HDF5 support for SVD/EPSValuesView? > > On Fri, Jun 4, 2021 at 10:28 AM Peder J?rgensgaard Olesen < > pjool at mek.dtu.dk> wrote: > >> What Matt suggests could perhaps be used as a workaround, though it >> appears neither elegant nor in my experience efficient. I may be wrong on >> this. >> > Why is it not efficient? > > Thanks, > > Matt > >> My attempted solution was to use a binary viewer instead, but I can't >> find a clear way to retrieve values thus stored with PETSc, as these do not >> lend themselves to be read using VecLoad. Again one might go about that >> by wrapping them inside a vector and send that to the viewer, which just >> brings us back around to the original problem. >> >> >> - Peder >> ------------------------------ >> *Fra:* Jose E. Roman >> *Sendt:* 4. juni 2021 16:01:04 >> *Til:* Matthew Knepley >> *Cc:* Peder J?rgensgaard Olesen; petsc-users at mcs.anl.gov >> *Emne:* Re: [petsc-users] SLEPc: HDF5 support for SVD/EPSValuesView? >> >> The problem is that here I am writing PetscReal's not PetscScalar's, and >> in EPS I am writing PetscComplex even in real scalars. That is why I did >> not implement it as you are suggesting. >> >> Jose >> >> >> > El 4 jun 2021, a las 15:56, Matthew Knepley >> escribi?: >> > >> > On Fri, Jun 4, 2021 at 9:31 AM Jose E. Roman >> wrote: >> > I could try and adapt the HDF5 code in PETSc for this, but I am no HDF5 >> expert. Furthermore I am busy at the moment, so it is faster if you could >> prepare a merge request with this addition. >> > >> > I think the easiest thing is to wrap a Vec around the values and just >> VecView() it into the same HDF5 file. >> > >> > THanks, >> > >> > Matt >> > >> > Jose >> > >> > >> > > El 4 jun 2021, a las 13:10, Peder J?rgensgaard Olesen via petsc-users >> escribi?: >> > > >> > > Hello >> > > >> > > In SLEPc one may write singular vectors to an HDF5 file using >> SVDVectorsView(), but a similar option doesn't seem to work for singular >> values using SVDValuesView(). The values are viewed correctly using other >> viewers, but nothing is seemingly produced when using an HDF5 viewer. >> Looking at the source code seems to confirm this, and suggests that there >> is a similar situation with EPS. >> > > >> > > Is this correct, and if so, might there be plans to include such >> functionality in future releases? >> > > >> > > Med venlig hilsen / Best Regards >> > > >> > > Peder J?rgensgaard Olesen >> > > PhD Student, Turbulence Research Lab >> > > Dept. of Mechanical Engineering >> > > Technical University of Denmark >> > > Koppels All? >> > > Bygning 403, Rum 105 >> > > DK-2800 Kgs. Lyngby >> > >> > >> > >> > -- >> > What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> > -- Norbert Wiener >> > >> > https://www.cse.buffalo.edu/~knepley/ >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rlmackie862 at gmail.com Mon Jun 7 16:43:38 2021 From: rlmackie862 at gmail.com (Randall Mackie) Date: Mon, 7 Jun 2021 14:43:38 -0700 Subject: [petsc-users] val grind suppression file Message-ID: <09828A79-C6A1-4E6F-8CC3-6EC92F0A0FBB@gmail.com> It seems that starting with petsc-3.15.0 the command petscmpiexec, when invoking Valgrind, now looks for a Valgrind suppression file, but there is no such file (at least in petsc-3.15.0.tar.gz from which I compiled the latest version). I did see this mention of a Valgrind suppression file: https://gitlab.com/petsc/petsc/-/commit/8d40a81ba4aa829c74edde321b419e6416763279 Is this the correct file to use? Thanks, Randy M. -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Mon Jun 7 16:50:12 2021 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 7 Jun 2021 16:50:12 -0500 Subject: [petsc-users] val grind suppression file In-Reply-To: <09828A79-C6A1-4E6F-8CC3-6EC92F0A0FBB@gmail.com> References: <09828A79-C6A1-4E6F-8CC3-6EC92F0A0FBB@gmail.com> Message-ID: <4f2adaa2-5f7d-fdc1-994-ad8d33b8cbc@mcs.anl.gov> Hm - the file always existed in the repo - and was previously used via env variable. But 'maint' dir was not included in the tarball. But I see https://gitlab.com/petsc/petsc/-/commit/90544984b0d changed from using it via env variable to a command line option. Satish On Mon, 7 Jun 2021, Randall Mackie wrote: > It seems that starting with petsc-3.15.0 the command petscmpiexec, when invoking Valgrind, now looks for a Valgrind suppression file, but there is no such file (at least in petsc-3.15.0.tar.gz from which I compiled the latest version). > > I did see this mention of a Valgrind suppression file: > > https://gitlab.com/petsc/petsc/-/commit/8d40a81ba4aa829c74edde321b419e6416763279 > > > Is this the correct file to use? > > > Thanks, > > Randy M. From balay at mcs.anl.gov Mon Jun 7 16:54:36 2021 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 7 Jun 2021 16:54:36 -0500 Subject: [petsc-users] val grind suppression file In-Reply-To: <4f2adaa2-5f7d-fdc1-994-ad8d33b8cbc@mcs.anl.gov> References: <09828A79-C6A1-4E6F-8CC3-6EC92F0A0FBB@gmail.com> <4f2adaa2-5f7d-fdc1-994-ad8d33b8cbc@mcs.anl.gov> Message-ID: Perhaps this file should be moved to share/petsc/valgrind/ Satish On Mon, 7 Jun 2021, Satish Balay via petsc-users wrote: > Hm - the file always existed in the repo - and was previously used via env variable. But 'maint' dir was not included in the tarball. > > But I see https://gitlab.com/petsc/petsc/-/commit/90544984b0d changed from using it via env variable to a command line option. > > Satish > > On Mon, 7 Jun 2021, Randall Mackie wrote: > > > It seems that starting with petsc-3.15.0 the command petscmpiexec, when invoking Valgrind, now looks for a Valgrind suppression file, but there is no such file (at least in petsc-3.15.0.tar.gz from which I compiled the latest version). > > > > I did see this mention of a Valgrind suppression file: > > > > https://gitlab.com/petsc/petsc/-/commit/8d40a81ba4aa829c74edde321b419e6416763279 > > > > > > Is this the correct file to use? > > > > > > Thanks, > > > > Randy M. > From mfadams at lbl.gov Tue Jun 8 23:11:44 2021 From: mfadams at lbl.gov (Mark Adams) Date: Wed, 9 Jun 2021 00:11:44 -0400 Subject: [petsc-users] aijcusparse error Message-ID: I have an old code that fails with -mat_type aijcusparse This is the code that creates the matrix: ierr = PetscPrintf(PETSC_COMM_SELF,"\nPetscSolve.H MatCreate adding csr \n");CHKERRQ(ierr); ierr = MatCreate(wcomm,&m_mat);CHKERRQ(ierr); ierr = MatSetOptionsPrefix(m_mat,"csr_");CHKERRQ(ierr); // prefix is new but does not matter ierr = MatSetSizes(m_mat,NN,NN,PETSC_DECIDE,PETSC_DECIDE);CHKERRQ(ierr); ierr = MatSetBlockSize(m_mat,nc);CHKERRQ(ierr); ierr = MatSetType(m_mat,MATAIJ);CHKERRQ(ierr); ierr = MatSeqAIJSetPreallocation(m_mat,nnzrow, d_nnz);CHKERRQ(ierr); ierr = MatMPIAIJSetPreallocation(m_mat,nnzrow, d_nnz, nnzrow/2, o_nnz);CHKERRQ(ierr); ierr = MatSetOption(m_mat,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE) ;CHKERRQ(ierr); ierr = MatSetFromOptions( m_mat ); CHKERRQ(ierr); With -csr_mat_type aijcusprase I get this output. Note, this apparent recursive structure in the call stack. I have no idea where that is coming from. I can't see that from the code. Any ideas? Thanks, Mark PetscSolve.H MatCreate .H adding csr [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Object is in wrong state [0]PETSC ERROR: Must call MatXXXSetPreallocation(), MatSetUp() or the matrix has not yet been factored on argument 1 "mat" before MatSetValues() [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.15.0-734-gc2c1a54 GIT Date: 2021-06-07 02:44:02 +0000 [0]PETSC ERROR: ../viscousDriver3d.Linux.64.mpiCC.mpifort.OPT.MPI.PETSC.ex on a arch-summit-dbg-gnu-cuda named a15n03 by adams Tue Jun 8 23:39:49 2021 [0]PETSC ERROR: Configure options --with-fc=mpifort --with-cc=mpicc --with-cxx=mpiCC --CFLAGS="-fPIC -g -mcpu=native -Wno-unused-but-set-variable -Wno-long-long -Wno-sign-compare -Wno-deprecated -Wno-unused-local-typedefs -Wno-variadic-macros -Wno-unknown-pragmas" --CXXFLAGS="-fPIC -g -std=c++11 -mcpu=native -Wno-unused-but-set-variable -Wno-long-long -Wno-sign-compare -Wno-deprecated -ftemplate-depth-99 -Wno-unused-local-typedefs -Wno-literal-suffix -Wno-invalid-offsetof -Wno-variadic-macros -Wno-unknown-pragmas" --CUDAFLAGS="-g -Xcompiler -lineinfo" --FCFLAGS="-fPIC -g" --FCOPTFLAGS=-O0 --COPTFLAGS=-O0 --CXXOPTFLAGS=-O0 --with-ssl=0 --with-batch=0 --with-mpiexec="jsrun -g1" --with-cuda=1 --with-cudac=nvcc --with-cuda-gencodearch=70 --download-metis --download-parmetis --with-blaslapack-lib="-L/autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/netlib-lapack-3.8.0-wcabdyqhdi5rooxbkqa6x5d7hxyxwdkm/lib64 -lblas -llapack" --with-x=0 --with-64-bit-indices=0 --with-debugging=1 PETSC_ARCH=arch-summit-dbg-gnu-cuda --force [0]PETSC ERROR: #1 MatSetValues() at /gpfs/alpine/csc314/scratch/adams/petsc/src/mat/interface/matrix.c:1366 [0]PETSC ERROR: #2 formMatrix() at ../srcLib/EBPetscLinearSolverPoisson.cpp:126 [0]PETSC ERROR: #3 setup_solver() at ../../../Chombo/lib/src/AMRElliptic/PetscSolverI.H:283 [0]PETSC ERROR: #4 solve_private() at ../../../Chombo/lib/src/AMRElliptic/PetscSolverI.H:427 [0]PETSC ERROR: #5 KSPSolve() at /gpfs/alpine/csc314/scratch/adams/petsc/src/ksp/ksp/interface/itfunc.c:1080 [0]PETSC ERROR: #6 solve_private() at ../../../Chombo/lib/src/AMRElliptic/PetscSolverI.H:508 [0]PETSC ERROR: #7 KSPSolve() at /gpfs/alpine/csc314/scratch/adams/petsc/src/ksp/ksp/interface/itfunc.c:1080 [0]PETSC ERROR: #8 solve_private() at ../../../Chombo/lib/src/AMRElliptic/PetscSolverI.H:508 [0]PETSC ERROR: #9 KSPSolve() at /gpfs/alpine/csc314/scratch/adams/petsc/src/ksp/ksp/interface/itfunc.c:1080 [0]PETSC ERROR: #10 solve_private() at ../../../Chombo/lib/src/AMRElliptic/PetscSolverI.H:508 [0]PETSC ERROR: #11 KSPSolve() at /gpfs/alpine/csc314/scratch/adams/petsc/src/ksp/ksp/interface/itfunc.c:1080 [0]PETSC ERROR: #12 solve_private() at ../../../Chombo/lib/src/AMRElliptic/PetscSolverI.H:508 -------------- next part -------------- An HTML attachment was scrubbed... URL: From dasvyat at gmail.com Wed Jun 9 00:25:51 2021 From: dasvyat at gmail.com (Daniil Svyatsky) Date: Tue, 8 Jun 2021 23:25:51 -0600 Subject: [petsc-users] Periodic BC with AMR In-Reply-To: References: Message-ID: Dear petsc developers and users, I'm trying to set up a periodic BC on a mesh obtained by AMR technique. I have a problem accessing the real coordinates of nodes on the periodic boundary. First, I generated a box mesh and apply periodic BC: bc_cond[0] = DM_BOUNDARY_PERIODIC; bc_cond[1] = DM_BOUNDARY_PERIODIC; bc_cond[2] = DM_BOUNDARY_PERIODIC; DMPlexCreateBoxMesh(PETSC_COMM_WORLD, dim, PETSC_FALSE, num_faces, lower_cor, top_cor, bc_cond, interpolate, &(dm)); DM baseParallel; PetscPartitioner part; ierr = DMPlexGetPartitioner(dm,&part);CHKERRQ(ierr); ierr = PetscPartitionerSetFromOptions(part);CHKERRQ(ierr); ierr = DMPlexDistribute(dm,0,NULL,&baseParallel);CHKERRQ(ierr); if (baseParallel) { ierr = DMDestroy(&(dm));CHKERRQ(ierr); dm = baseParallel; } After this step I can access coordinates of boundary nodes correctly. Then I applied AMR on this mesh: DM preForest, postForest; DMLabel adaptLabel = NULL; ierr = DMCreate(PETSC_COMM_SELF, &preForest);CHKERRQ(ierr); ierr = DMSetType(preForest,(dim == 2) ? DMP4EST : DMP8EST);CHKERRQ(ierr); ierr = DMForestSetBaseDM(preForest,env->dstruct->dm);CHKERRQ(ierr); ierr = DMForestSetMinimumRefinement(preForest,0);CHKERRQ(ierr); ierr = DMForestSetInitialRefinement(preForest,0);CHKERRQ(ierr); ierr = DMSetFromOptions(preForest);CHKERRQ(ierr); ierr = DMSetUp(preForest);CHKERRQ(ierr); ierr = CreateAdaptivityLabel(env,preForest,&adaptLabel);CHKERRQ(ierr); ierr = DMForestTemplate(preForest,PETSC_COMM_SELF, &postForest);CHKERRQ(ierr); ierr = DMForestSetAdaptivityLabel(postForest,adaptLabel);CHKERRQ(ierr); ierr = DMLabelDestroy(&adaptLabel);CHKERRQ(ierr); ierr = DMSetUp(postForest);CHKERRQ(ierr); DM dmConv; ierr = DMConvert(postForest,DMPLEX,&dmConv);CHKERRQ(ierr); if (dmConv){ DMDestroy(&dm); dm = dmConv; } In my example, boundary cells were also refined. After AMR I have a problem accessing the coordinates of nodes. To access node coordinates I do the following: DMGetCoordinatesLocal(dm, &coordinates); DMGetCoordinateDM(dm, &cdm); DMGetSection(cdm, &cs); for (cell = cStart; cell < cEnd; ++cell) { PetscInt numFaces; const PetscInt *faces; ierr = DMPlexGetConeSize(dm, cell, &numFaces); ierr = DMPlexGetCone(dm, cell, &faces); for (int f = 0; f From bsmith at petsc.dev Wed Jun 9 00:40:03 2021 From: bsmith at petsc.dev (Barry Smith) Date: Wed, 9 Jun 2021 00:40:03 -0500 Subject: [petsc-users] aijcusparse error In-Reply-To: References: Message-ID: <45BF0C3E-9E1F-49A9-8161-5F8B0DEEF8FC@petsc.dev> You need to set the type before calling the preallocation routines. Otherwise when you set to a new type all the previous information is essentially removed when the new type is created. Barry A slightly more clever MatCreate_MPIAIJCUSPARSE() could check if the matrix was already of the type MPIAIJ and reuse that information but currently it does not do that check. > On Jun 8, 2021, at 11:11 PM, Mark Adams wrote: > > I have an old code that fails with -mat_type aijcusparse > This is the code that creates the matrix: > > ierr = PetscPrintf(PETSC_COMM_SELF,"\nPetscSolve.H MatCreate adding csr \n");CHKERRQ(ierr); > ierr = MatCreate(wcomm,&m_mat);CHKERRQ(ierr); > ierr = MatSetOptionsPrefix(m_mat,"csr_");CHKERRQ(ierr); // prefix is new but does not matter > ierr = MatSetSizes(m_mat,NN,NN,PETSC_DECIDE,PETSC_DECIDE);CHKERRQ(ierr); > ierr = MatSetBlockSize(m_mat,nc);CHKERRQ(ierr); > ierr = MatSetType(m_mat,MATAIJ);CHKERRQ(ierr); > ierr = MatSeqAIJSetPreallocation(m_mat,nnzrow, d_nnz);CHKERRQ(ierr); > ierr = MatMPIAIJSetPreallocation(m_mat,nnzrow, d_nnz, nnzrow/2, o_nnz);CHKERRQ(ierr); > ierr = MatSetOption(m_mat,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE) ;CHKERRQ(ierr); > ierr = MatSetFromOptions( m_mat ); CHKERRQ(ierr); > > With -csr_mat_type aijcusprase I get this output. > Note, this apparent recursive structure in the call stack. I have no idea where that is coming from. I can't see that from the code. > > Any ideas? > > Thanks, > Mark > > PetscSolve.H MatCreate .H adding csr > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Object is in wrong state > [0]PETSC ERROR: Must call MatXXXSetPreallocation(), MatSetUp() or the matrix has not yet been factored on argument 1 "mat" before MatSetValues() > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.15.0-734-gc2c1a54 GIT Date: 2021-06-07 02:44:02 +0000 > [0]PETSC ERROR: ../viscousDriver3d.Linux.64.mpiCC.mpifort.OPT.MPI.PETSC.ex on a arch-summit-dbg-gnu-cuda named a15n03 by adams Tue Jun 8 23:39:49 2021 > [0]PETSC ERROR: Configure options --with-fc=mpifort --with-cc=mpicc --with-cxx=mpiCC --CFLAGS="-fPIC -g -mcpu=native -Wno-unused-but-set-variable -Wno-long-long -Wno-sign-compare -Wno-deprecated -Wno-unused-local-typedefs -Wno-variadic-macros -Wno-unknown-pragmas" --CXXFLAGS="-fPIC -g -std=c++11 -mcpu=native -Wno-unused-but-set-variable -Wno-long-long -Wno-sign-compare -Wno-deprecated -ftemplate-depth-99 -Wno-unused-local-typedefs -Wno-literal-suffix -Wno-invalid-offsetof -Wno-variadic-macros -Wno-unknown-pragmas" --CUDAFLAGS="-g -Xcompiler -lineinfo" --FCFLAGS="-fPIC -g" --FCOPTFLAGS=-O0 --COPTFLAGS=-O0 --CXXOPTFLAGS=-O0 --with-ssl=0 --with-batch=0 --with-mpiexec="jsrun -g1" --with-cuda=1 --with-cudac=nvcc --with-cuda-gencodearch=70 --download-metis --download-parmetis --with-blaslapack-lib="-L/autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/netlib-lapack-3.8.0-wcabdyqhdi5rooxbkqa6x5d7hxyxwdkm/lib64 -lblas -llapack" --with-x=0 --with-64-bit-indices=0 --with-debugging=1 PETSC_ARCH=arch-summit-dbg-gnu-cuda --force > [0]PETSC ERROR: #1 MatSetValues() at /gpfs/alpine/csc314/scratch/adams/petsc/src/mat/interface/matrix.c:1366 > [0]PETSC ERROR: #2 formMatrix() at ../srcLib/EBPetscLinearSolverPoisson.cpp:126 > [0]PETSC ERROR: #3 setup_solver() at ../../../Chombo/lib/src/AMRElliptic/PetscSolverI.H:283 > [0]PETSC ERROR: #4 solve_private() at ../../../Chombo/lib/src/AMRElliptic/PetscSolverI.H:427 > [0]PETSC ERROR: #5 KSPSolve() at /gpfs/alpine/csc314/scratch/adams/petsc/src/ksp/ksp/interface/itfunc.c:1080 > [0]PETSC ERROR: #6 solve_private() at ../../../Chombo/lib/src/AMRElliptic/PetscSolverI.H:508 > [0]PETSC ERROR: #7 KSPSolve() at /gpfs/alpine/csc314/scratch/adams/petsc/src/ksp/ksp/interface/itfunc.c:1080 > [0]PETSC ERROR: #8 solve_private() at ../../../Chombo/lib/src/AMRElliptic/PetscSolverI.H:508 > [0]PETSC ERROR: #9 KSPSolve() at /gpfs/alpine/csc314/scratch/adams/petsc/src/ksp/ksp/interface/itfunc.c:1080 > [0]PETSC ERROR: #10 solve_private() at ../../../Chombo/lib/src/AMRElliptic/PetscSolverI.H:508 > [0]PETSC ERROR: #11 KSPSolve() at /gpfs/alpine/csc314/scratch/adams/petsc/src/ksp/ksp/interface/itfunc.c:1080 > [0]PETSC ERROR: #12 solve_private() at ../../../Chombo/lib/src/AMRElliptic/PetscSolverI.H:508 > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From julien.vignollet at safrangroup.com Wed Jun 9 08:46:26 2021 From: julien.vignollet at safrangroup.com (VIGNOLLET Julien (SAFRAN)) Date: Wed, 9 Jun 2021 13:46:26 +0000 Subject: [petsc-users] petsc-users Message-ID: Hi there, I would like to be able to post questions to the petsc-users community, would you be able to add me to the list? Best regards, _________________________________ Julien Vignollet Research Engineer Safran Tech M&S | Computational Solid Mechanics (CSM) T +33 (0)1 61 31 88 07 Rue des Jeunes Bois Ch?teaufort CS08112 78772 Magny-Les-Hameaux Cedex France www.safran-group.com ____________________________________ # " Ce courriel et les documents qui lui sont joints peuvent contenir des informations confidentielles, ?tre soumis aux r?glementations relatives au contr?le des exportations ou ayant un caract?re priv?. S'ils ne vous sont pas destin?s, nous vous signalons qu'il est strictement interdit de les divulguer, de les reproduire ou d'en utiliser de quelque mani?re que ce soit le contenu. Toute exportation ou r?exportation non autoris?e est interdite Si ce message vous a ?t? transmis par erreur, merci d'en informer l'exp?diteur et de supprimer imm?diatement de votre syst?me informatique ce courriel ainsi que tous les documents qui y sont attach?s." ****** " This e-mail and any attached documents may contain confidential or proprietary information and may be subject to export control laws and regulations. If you are not the intended recipient, you are notified that any dissemination, copying of this e-mail and any attachments thereto or use of their contents by any means whatsoever is strictly prohibited. Unauthorized export or re-export is prohibited. If you have received this e-mail in error, please advise the sender immediately and delete this e-mail and all attached documents from your computer system." # -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Wed Jun 9 09:09:22 2021 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 9 Jun 2021 09:09:22 -0500 Subject: [petsc-users] petsc-users In-Reply-To: References: Message-ID: I see you are already subscribed to the list now. The list subscribe instructions are https://www.mcs.anl.gov/petsc/miscellaneous/mailing-lists.html However one can also post to the list without subscribing [but this requires a one time approval by list-admin] Satish On Wed, 9 Jun 2021, VIGNOLLET Julien (SAFRAN) wrote: > Hi there, > > I would like to be able to post questions to the petsc-users community, would you be able to add me to the list? > > Best regards, > > _________________________________ > Julien Vignollet > Research Engineer > Safran Tech > M&S | Computational Solid Mechanics (CSM) > > T +33 (0)1 61 31 88 07 > > Rue des Jeunes Bois > Ch?teaufort CS08112 > 78772 Magny-Les-Hameaux Cedex > France > www.safran-group.com > ____________________________________ > > > # > " Ce courriel et les documents qui lui sont joints peuvent contenir des informations confidentielles, ?tre soumis aux r?glementations relatives au contr?le des exportations ou ayant un caract?re priv?. S'ils ne vous sont pas destin?s, nous vous signalons qu'il est strictement interdit de les divulguer, de les reproduire ou d'en utiliser de quelque mani?re que ce soit le contenu. Toute exportation ou r?exportation non autoris?e est interdite Si ce message vous a ?t? transmis par erreur, merci d'en informer l'exp?diteur et de supprimer imm?diatement de votre syst?me informatique ce courriel ainsi que tous les documents qui y sont attach?s." > ****** > " This e-mail and any attached documents may contain confidential or proprietary information and may be subject to export control laws and regulations. If you are not the intended recipient, you are notified that any dissemination, copying of this e-mail and any attachments thereto or use of their contents by any means whatsoever is strictly prohibited. Unauthorized export or re-export is prohibited. If you have received this e-mail in error, please advise the sender immediately and delete this e-mail and all attached documents from your computer system." > # > From mfadams at lbl.gov Wed Jun 9 09:25:33 2021 From: mfadams at lbl.gov (Mark Adams) Date: Wed, 9 Jun 2021 10:25:33 -0400 Subject: [petsc-users] aijcusparse error In-Reply-To: <45BF0C3E-9E1F-49A9-8161-5F8B0DEEF8FC@petsc.dev> References: <45BF0C3E-9E1F-49A9-8161-5F8B0DEEF8FC@petsc.dev> Message-ID: On Wed, Jun 9, 2021 at 1:40 AM Barry Smith wrote: > > You need to set the type before calling the preallocation routines. > Otherwise when you set to a new type all the previous information is > essentially removed when the new type is created. > I do * set the type b*efore calling the *preallocation routines* > > Barry > > A slightly more clever MatCreate_MPIAIJCUSPARSE() could check if the > matrix was already of the type MPIAIJ and reuse that information but > currently it does not do that check. > > > On Jun 8, 2021, at 11:11 PM, Mark Adams wrote: > > I have an old code that fails with -mat_type aijcusparse > This is the code that creates the matrix: > > ierr = PetscPrintf(PETSC_COMM_SELF,"\nPetscSolve.H MatCreate adding > csr \n");CHKERRQ(ierr); > ierr = MatCreate(wcomm,&m_mat);CHKERRQ(ierr); > ierr = MatSetOptionsPrefix(m_mat,"csr_");CHKERRQ(ierr); // prefix > is new but does not matter > ierr = > MatSetSizes(m_mat,NN,NN,PETSC_DECIDE,PETSC_DECIDE);CHKERRQ(ierr); > ierr = MatSetBlockSize(m_mat,nc);CHKERRQ(ierr); > > * ierr = MatSetType(m_mat,MATAIJ);CHKERRQ(ierr);* > * ierr = MatSeqAIJSetPreallocation(m_mat,nnzrow, > d_nnz);CHKERRQ(ierr); ierr = MatMPIAIJSetPreallocation(m_mat,nnzrow, > d_nnz, nnzrow/2, o_nnz);CHKERRQ(ierr);* > ierr = > MatSetOption(m_mat,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE) > ;CHKERRQ(ierr); > ierr = MatSetFromOptions( m_mat ); CHKERRQ(ierr); > > With -csr_mat_type aijcusprase I get this output. > Note, this apparent recursive structure in the call stack. I have no idea > where that is coming from. I can't see that from the code. > > Any ideas? > > Thanks, > Mark > > PetscSolve.H MatCreate .H adding csr > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Object is in wrong state > [0]PETSC ERROR: Must call MatXXXSetPreallocation(), MatSetUp() or the > matrix has not yet been factored on argument 1 "mat" before MatSetValues() > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.15.0-734-gc2c1a54 GIT > Date: 2021-06-07 02:44:02 +0000 > [0]PETSC ERROR: ../viscousDriver3d.Linux.64.mpiCC.mpifort.OPT.MPI.PETSC.ex > on a arch-summit-dbg-gnu-cuda named a15n03 by adams Tue Jun 8 23:39:49 2021 > [0]PETSC ERROR: Configure options --with-fc=mpifort --with-cc=mpicc > --with-cxx=mpiCC --CFLAGS="-fPIC -g -mcpu=native > -Wno-unused-but-set-variable -Wno-long-long -Wno-sign-compare > -Wno-deprecated -Wno-unused-local-typedefs -Wno-variadic-macros > -Wno-unknown-pragmas" --CXXFLAGS="-fPIC -g -std=c++11 -mcpu=native > -Wno-unused-but-set-variable -Wno-long-long -Wno-sign-compare > -Wno-deprecated -ftemplate-depth-99 -Wno-unused-local-typedefs > -Wno-literal-suffix -Wno-invalid-offsetof -Wno-variadic-macros > -Wno-unknown-pragmas" --CUDAFLAGS="-g -Xcompiler -lineinfo" > --FCFLAGS="-fPIC -g" --FCOPTFLAGS=-O0 --COPTFLAGS=-O0 --CXXOPTFLAGS=-O0 > --with-ssl=0 --with-batch=0 --with-mpiexec="jsrun -g1" --with-cuda=1 > --with-cudac=nvcc --with-cuda-gencodearch=70 --download-metis > --download-parmetis > --with-blaslapack-lib="-L/autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/netlib-lapack-3.8.0-wcabdyqhdi5rooxbkqa6x5d7hxyxwdkm/lib64 > -lblas -llapack" --with-x=0 --with-64-bit-indices=0 --with-debugging=1 > PETSC_ARCH=arch-summit-dbg-gnu-cuda --force > [0]PETSC ERROR: #1 MatSetValues() at > /gpfs/alpine/csc314/scratch/adams/petsc/src/mat/interface/matrix.c:1366 > [0]PETSC ERROR: #2 formMatrix() at > ../srcLib/EBPetscLinearSolverPoisson.cpp:126 > [0]PETSC ERROR: #3 setup_solver() at > ../../../Chombo/lib/src/AMRElliptic/PetscSolverI.H:283 > [0]PETSC ERROR: #4 solve_private() at > ../../../Chombo/lib/src/AMRElliptic/PetscSolverI.H:427 > [0]PETSC ERROR: #5 KSPSolve() at > /gpfs/alpine/csc314/scratch/adams/petsc/src/ksp/ksp/interface/itfunc.c:1080 > [0]PETSC ERROR: #6 solve_private() at > ../../../Chombo/lib/src/AMRElliptic/PetscSolverI.H:508 > [0]PETSC ERROR: #7 KSPSolve() at > /gpfs/alpine/csc314/scratch/adams/petsc/src/ksp/ksp/interface/itfunc.c:1080 > [0]PETSC ERROR: #8 solve_private() at > ../../../Chombo/lib/src/AMRElliptic/PetscSolverI.H:508 > [0]PETSC ERROR: #9 KSPSolve() at > /gpfs/alpine/csc314/scratch/adams/petsc/src/ksp/ksp/interface/itfunc.c:1080 > [0]PETSC ERROR: #10 solve_private() at > ../../../Chombo/lib/src/AMRElliptic/PetscSolverI.H:508 > [0]PETSC ERROR: #11 KSPSolve() at > /gpfs/alpine/csc314/scratch/adams/petsc/src/ksp/ksp/interface/itfunc.c:1080 > [0]PETSC ERROR: #12 solve_private() at > ../../../Chombo/lib/src/AMRElliptic/PetscSolverI.H:508 > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Wed Jun 9 10:13:00 2021 From: mfadams at lbl.gov (Mark Adams) Date: Wed, 9 Jun 2021 11:13:00 -0400 Subject: [petsc-users] aijcusparse error In-Reply-To: References: <45BF0C3E-9E1F-49A9-8161-5F8B0DEEF8FC@petsc.dev> Message-ID: Argh, My mistake. ksp/ex56 has the prealloc stuff after the GetFromOptions and it seems to work now. On Wed, Jun 9, 2021 at 10:25 AM Mark Adams wrote: > > > On Wed, Jun 9, 2021 at 1:40 AM Barry Smith wrote: > >> >> You need to set the type before calling the preallocation routines. >> Otherwise when you set to a new type all the previous information is >> essentially removed when the new type is created. >> > > I do * set the type b*efore calling the *preallocation routines* > > >> >> Barry >> >> A slightly more clever MatCreate_MPIAIJCUSPARSE() could check if the >> matrix was already of the type MPIAIJ and reuse that information but >> currently it does not do that check. >> >> >> On Jun 8, 2021, at 11:11 PM, Mark Adams wrote: >> >> I have an old code that fails with -mat_type aijcusparse >> This is the code that creates the matrix: >> >> ierr = PetscPrintf(PETSC_COMM_SELF,"\nPetscSolve.H MatCreate adding >> csr \n");CHKERRQ(ierr); >> ierr = MatCreate(wcomm,&m_mat);CHKERRQ(ierr); >> ierr = MatSetOptionsPrefix(m_mat,"csr_");CHKERRQ(ierr); // prefix >> is new but does not matter >> ierr = >> MatSetSizes(m_mat,NN,NN,PETSC_DECIDE,PETSC_DECIDE);CHKERRQ(ierr); >> ierr = MatSetBlockSize(m_mat,nc);CHKERRQ(ierr); >> >> * ierr = MatSetType(m_mat,MATAIJ);CHKERRQ(ierr);* >> * ierr = MatSeqAIJSetPreallocation(m_mat,nnzrow, >> d_nnz);CHKERRQ(ierr); ierr = MatMPIAIJSetPreallocation(m_mat,nnzrow, >> d_nnz, nnzrow/2, o_nnz);CHKERRQ(ierr);* >> ierr = >> MatSetOption(m_mat,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE) >> ;CHKERRQ(ierr); >> ierr = MatSetFromOptions( m_mat ); CHKERRQ(ierr); >> >> With -csr_mat_type aijcusprase I get this output. >> Note, this apparent recursive structure in the call stack. I have no idea >> where that is coming from. I can't see that from the code. >> >> Any ideas? >> >> Thanks, >> Mark >> >> PetscSolve.H MatCreate .H adding csr >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: Object is in wrong state >> [0]PETSC ERROR: Must call MatXXXSetPreallocation(), MatSetUp() or the >> matrix has not yet been factored on argument 1 "mat" before MatSetValues() >> [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [0]PETSC ERROR: Petsc Development GIT revision: v3.15.0-734-gc2c1a54 GIT >> Date: 2021-06-07 02:44:02 +0000 >> [0]PETSC ERROR: >> ../viscousDriver3d.Linux.64.mpiCC.mpifort.OPT.MPI.PETSC.ex on a >> arch-summit-dbg-gnu-cuda named a15n03 by adams Tue Jun 8 23:39:49 2021 >> [0]PETSC ERROR: Configure options --with-fc=mpifort --with-cc=mpicc >> --with-cxx=mpiCC --CFLAGS="-fPIC -g -mcpu=native >> -Wno-unused-but-set-variable -Wno-long-long -Wno-sign-compare >> -Wno-deprecated -Wno-unused-local-typedefs -Wno-variadic-macros >> -Wno-unknown-pragmas" --CXXFLAGS="-fPIC -g -std=c++11 -mcpu=native >> -Wno-unused-but-set-variable -Wno-long-long -Wno-sign-compare >> -Wno-deprecated -ftemplate-depth-99 -Wno-unused-local-typedefs >> -Wno-literal-suffix -Wno-invalid-offsetof -Wno-variadic-macros >> -Wno-unknown-pragmas" --CUDAFLAGS="-g -Xcompiler -lineinfo" >> --FCFLAGS="-fPIC -g" --FCOPTFLAGS=-O0 --COPTFLAGS=-O0 --CXXOPTFLAGS=-O0 >> --with-ssl=0 --with-batch=0 --with-mpiexec="jsrun -g1" --with-cuda=1 >> --with-cudac=nvcc --with-cuda-gencodearch=70 --download-metis >> --download-parmetis >> --with-blaslapack-lib="-L/autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/netlib-lapack-3.8.0-wcabdyqhdi5rooxbkqa6x5d7hxyxwdkm/lib64 >> -lblas -llapack" --with-x=0 --with-64-bit-indices=0 --with-debugging=1 >> PETSC_ARCH=arch-summit-dbg-gnu-cuda --force >> [0]PETSC ERROR: #1 MatSetValues() at >> /gpfs/alpine/csc314/scratch/adams/petsc/src/mat/interface/matrix.c:1366 >> [0]PETSC ERROR: #2 formMatrix() at >> ../srcLib/EBPetscLinearSolverPoisson.cpp:126 >> [0]PETSC ERROR: #3 setup_solver() at >> ../../../Chombo/lib/src/AMRElliptic/PetscSolverI.H:283 >> [0]PETSC ERROR: #4 solve_private() at >> ../../../Chombo/lib/src/AMRElliptic/PetscSolverI.H:427 >> [0]PETSC ERROR: #5 KSPSolve() at >> /gpfs/alpine/csc314/scratch/adams/petsc/src/ksp/ksp/interface/itfunc.c:1080 >> [0]PETSC ERROR: #6 solve_private() at >> ../../../Chombo/lib/src/AMRElliptic/PetscSolverI.H:508 >> [0]PETSC ERROR: #7 KSPSolve() at >> /gpfs/alpine/csc314/scratch/adams/petsc/src/ksp/ksp/interface/itfunc.c:1080 >> [0]PETSC ERROR: #8 solve_private() at >> ../../../Chombo/lib/src/AMRElliptic/PetscSolverI.H:508 >> [0]PETSC ERROR: #9 KSPSolve() at >> /gpfs/alpine/csc314/scratch/adams/petsc/src/ksp/ksp/interface/itfunc.c:1080 >> [0]PETSC ERROR: #10 solve_private() at >> ../../../Chombo/lib/src/AMRElliptic/PetscSolverI.H:508 >> [0]PETSC ERROR: #11 KSPSolve() at >> /gpfs/alpine/csc314/scratch/adams/petsc/src/ksp/ksp/interface/itfunc.c:1080 >> [0]PETSC ERROR: #12 solve_private() at >> ../../../Chombo/lib/src/AMRElliptic/PetscSolverI.H:508 >> >> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Wed Jun 9 14:23:14 2021 From: mfadams at lbl.gov (Mark Adams) Date: Wed, 9 Jun 2021 15:23:14 -0400 Subject: [petsc-users] Periodic BC with AMR In-Reply-To: References: Message-ID: Matt is the expert (and he is busy today) but I use: ierr = DMLocalizeCoordinates(dm);CHKERRQ(ierr); /* needed for periodic */ On Wed, Jun 9, 2021 at 1:26 AM Daniil Svyatsky wrote: > > Dear petsc developers and users, > > I'm trying to set up a periodic BC on a mesh obtained by AMR technique. > I have a problem accessing the real coordinates of nodes on the periodic > boundary. > > First, I generated a box mesh and apply periodic BC: > > bc_cond[0] = DM_BOUNDARY_PERIODIC; > > bc_cond[1] = DM_BOUNDARY_PERIODIC; > > bc_cond[2] = DM_BOUNDARY_PERIODIC; > > > DMPlexCreateBoxMesh(PETSC_COMM_WORLD, dim, PETSC_FALSE, > > num_faces, lower_cor, top_cor, bc_cond, > interpolate, &(dm)); > > DM baseParallel; > > PetscPartitioner part; > > > ierr = DMPlexGetPartitioner(dm,&part);CHKERRQ(ierr); > > ierr = PetscPartitionerSetFromOptions(part);CHKERRQ(ierr); > > ierr = DMPlexDistribute(dm,0,NULL,&baseParallel);CHKERRQ(ierr); > > if (baseParallel) { > > ierr = DMDestroy(&(dm));CHKERRQ(ierr); > > dm = baseParallel; > > } > > > > After this step I can access coordinates of boundary nodes correctly. > Then I applied AMR on this mesh: > > DM preForest, postForest; > > DMLabel adaptLabel = NULL; > > > ierr = DMCreate(PETSC_COMM_SELF, &preForest);CHKERRQ(ierr); > > ierr = DMSetType(preForest,(dim == 2) ? DMP4EST : > DMP8EST);CHKERRQ(ierr); > > ierr = DMForestSetBaseDM(preForest,env->dstruct->dm);CHKERRQ(ierr); > > ierr = DMForestSetMinimumRefinement(preForest,0);CHKERRQ(ierr); > > ierr = DMForestSetInitialRefinement(preForest,0);CHKERRQ(ierr); > > ierr = DMSetFromOptions(preForest);CHKERRQ(ierr); > > ierr = DMSetUp(preForest);CHKERRQ(ierr); > > > ierr = CreateAdaptivityLabel(env,preForest,&adaptLabel);CHKERRQ(ierr); > > ierr = DMForestTemplate(preForest,PETSC_COMM_SELF, > &postForest);CHKERRQ(ierr); > > ierr = > DMForestSetAdaptivityLabel(postForest,adaptLabel);CHKERRQ(ierr); > > ierr = DMLabelDestroy(&adaptLabel);CHKERRQ(ierr); > > ierr = DMSetUp(postForest);CHKERRQ(ierr); > > > DM dmConv; > > ierr = DMConvert(postForest,DMPLEX,&dmConv);CHKERRQ(ierr); > > > > if (dmConv){ > > DMDestroy(&dm); > > dm = dmConv; > > } > > In my example, boundary cells were also refined. After AMR I have a > problem accessing the coordinates of nodes. > To access node coordinates I do the following: > > DMGetCoordinatesLocal(dm, &coordinates); > > DMGetCoordinateDM(dm, &cdm); > > DMGetSection(cdm, &cs); > > > for (cell = cStart; cell < cEnd; ++cell) { > > PetscInt numFaces; > > const PetscInt *faces; > > ierr = DMPlexGetConeSize(dm, cell, &numFaces); > > ierr = DMPlexGetCone(dm, cell, &faces); > > for (int f = 0; f > temp_f = NULL; > > DMPlexVecGetClosure(dm, cs, coordinates, faces[f], &fsize, &temp_f); > > for (p=0; p > DMLocalizeCoordinate(dm, temp_f+p, PETSC_FALSE, coord_p) > > } > > } > > } > > > coord_p has wrong coordinates either I used PETSC_TRUE or PETSC_FALSE > in DMLocalizeCoordinate. > > What am I doing wrong? I'm using petsc-3.14.1 version. > Thank you for your help in advance. > > Best regards, > Daniil Svyatsky. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Thu Jun 10 15:27:30 2021 From: mfadams at lbl.gov (Mark Adams) Date: Thu, 10 Jun 2021 16:27:30 -0400 Subject: [petsc-users] hypre on GPUs Message-ID: If I (a user) want to try hypre with CUDA, there any special configure options that I should use like: '--download-hypre-configure-arguments=HYPRE_CUDA_SM=70', I recall using a hypre branch parameter at one point. Thanks, Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From kaushikv318 at gmail.com Thu Jun 10 15:29:08 2021 From: kaushikv318 at gmail.com (Kaushik Vijaykumar) Date: Thu, 10 Jun 2021 16:29:08 -0400 Subject: [petsc-users] Copy stiffness matrix to jacobian matrix in form jacobian Message-ID: Hi everyone, I am trying to copy the stiffness matrix that I generated in form function to the jacobian matrix jac in form jacobian. The piece of code for that: my_ctx* ptr = (my_ctx*) ctx; // cast the pointer to void into pointer to struct ierr = MatGetOwnershipRange(jac,&istart,&iend); ierr = MatGetSize(ptr->K,&m,&n); CHKERRQ(ierr); ierr = MatAssemblyBegin(ptr->K,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); ierr = MatAssemblyEnd(ptr->K,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); for(i=istart;iK, i, j, &v);CHKERRQ(ierr); ierr = MatSetValue(jac, i, j, v, INSERT_VALUES);CHKERRQ(ierr); ierr = MatSetValue(B, i, j, v, INSERT_VALUES);CHKERRQ(ierr); } } The following works for a single process run only. For a multiprocessor run the ptr->K does not get copied correctly to jac I have also tried the following commands ierr = MatDuplicate(ptr->K,MAT_COPY_VALUES,&jac);CHKERRQ(ierr); ierr = MatDuplicate(ptr->K,MAT_COPY_VALUES,&B);CHKERRQ(ierr); ierr = MatCopy(ptr->K,jac,SAME_NONZERO_PATTERN);CHKERRQ(ierr); ierr = MatCopy(ptr->K,B,SAME_NONZERO_PATTERN);CHKERRQ(ierr); I get Petsc object in a wrong state error. Could you please advise on what is most efficient way to copy the matrix from the user defined struct to jacobian. Thanks Kaushik -------------- next part -------------- An HTML attachment was scrubbed... URL: From dptrebotich at lbl.gov Thu Jun 10 15:30:15 2021 From: dptrebotich at lbl.gov (David Trebotich) Date: Thu, 10 Jun 2021 13:30:15 -0700 Subject: [petsc-users] hypre on GPUs In-Reply-To: References: Message-ID: I have no idea. I will also say that Ulrike has hypre-petsc compatibility on GPUs on her list as a possible P6 level milestone in the solver ECP project. On Thu, Jun 10, 2021 at 1:27 PM Mark Adams wrote: > If I (a user) want to try hypre with CUDA, there any special configure > options that I should use like: > > '--download-hypre-configure-arguments=HYPRE_CUDA_SM=70', > > I recall using a hypre branch parameter at one point. > > Thanks, > Mark > -- ---------------------- David Trebotich Lawrence Berkeley National Laboratory Computational Research Division Applied Numerical Algorithms Group treb at lbl.gov (510) 486-5984 office (510) 384-6868 mobile -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Thu Jun 10 17:37:20 2021 From: bsmith at petsc.dev (Barry Smith) Date: Thu, 10 Jun 2021 17:37:20 -0500 Subject: [petsc-users] Copy stiffness matrix to jacobian matrix in form jacobian In-Reply-To: References: Message-ID: > On Jun 10, 2021, at 3:29 PM, Kaushik Vijaykumar wrote: > > Hi everyone, > > I am trying to copy the stiffness matrix that I generated in form function to the jacobian matrix jac in form jacobian. The piece of code for that: > > If you have created jac and B in your main program and passed them with SNESSetJacobian() then you should be able to just use > ierr = MatCopy(ptr->K,jac,SAME_NONZERO_PATTERN);CHKERRQ(ierr); Generally jac and B are the same matrix so you don't need the second copy. > ierr = MatCopy(ptr->K,B,SAME_NONZERO_PATTERN);CHKERRQ(ierr); If they are sometimes the same and sometime not you can do if (jac != B) {ierr = MatCopy(ptr->K,B,SAME_NONZERO_PATTERN);CHKERRQ(ierr); } The following won't work because you are creating new local matrices in your form jacobian that don't exist outside of that routine. > > ierr = MatDuplicate(ptr->K,MAT_COPY_VALUES,&jac);CHKERRQ(ierr); > ierr = MatDuplicate(ptr->K,MAT_COPY_VALUES,&B);CHKERRQ(ierr); > > ierr = MatCopy(ptr->K,jac,SAME_NONZERO_PATTERN);CHKERRQ(ierr); > ierr = MatCopy(ptr->K,B,SAME_NONZERO_PATTERN);CHKERRQ(ierr); > > If the matrix you are creating in form function is the Jacobian then you don't need to use the memory and time to make copies. You can just form the matrix directly into the Jacobian you will use for SNES. You can obtain this with a call SNESGetJacobian(snes,&jac,NULL,NULL,NULL); in your form function and just put the matrix values directly into jac. Then your form Jacobian does not need to do anything but return since the jac already contains the Jacobian. On the off-change you are using the iteration A(x^n) x^{n+1} = b(x^n} and not using Newtons' method (which is why you compute A in form function, then you might be better off using SNESSetPicard() instead of SNESSetFunction(), SNESSetJacobian(). > > Thanks > Kaushik From a.croucher at auckland.ac.nz Thu Jun 10 23:44:02 2021 From: a.croucher at auckland.ac.nz (Adrian Croucher) Date: Fri, 11 Jun 2021 16:44:02 +1200 Subject: [petsc-users] flux vector Message-ID: <2e0e287c-77cf-c375-3b92-c80816c30724@auckland.ac.nz> hi I have a DMPlex in a finite volume simulation and I want to create a Vec containing flux data only on the mesh faces where fluxes are computed (e.g. not on closed boundaries). Basically these are all the faces with support size 2 (aside from some complications related to boundary conditions). To do this I clone the DM using DMClone() and create a section on it with DMPlexCreateSection(), passing in a label which has value 1 on all the required faces. The problem then consists of labelling the appropriate faces. The tricky part is that when a face exists on more than one process, you have to make sure it has the same label value on all processes, otherwise an error results when you try and create a global vector on the DM. Because I distribute the DMPlex with overlap = 1, a face can have different support sizes on different processes. So on each process, I need to label a face if it has support size 2 on *any* process. Would it be possible to use PetscSFBcast and the point SF from DMGetPointSF() to somehow broadcast the support sizes? Or is there a simpler way? - Adrian -- Dr Adrian Croucher Senior Research Fellow Department of Engineering Science University of Auckland, New Zealand email: a.croucher at auckland.ac.nz tel: +64 (0)9 923 4611 From mfadams at lbl.gov Fri Jun 11 05:29:55 2021 From: mfadams at lbl.gov (Mark Adams) Date: Fri, 11 Jun 2021 06:29:55 -0400 Subject: [petsc-users] flux vector In-Reply-To: <2e0e287c-77cf-c375-3b92-c80816c30724@auckland.ac.nz> References: <2e0e287c-77cf-c375-3b92-c80816c30724@auckland.ac.nz> Message-ID: This is a Matt question, but You can set a "Boundary" label ( https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMPLEX/DMPlexMarkBoundaryFaces.html) and that will label boundaries. Next you want the dual of that ... On Fri, Jun 11, 2021 at 12:44 AM Adrian Croucher wrote: > hi > > I have a DMPlex in a finite volume simulation and I want to create a Vec > containing flux data only on the mesh faces where fluxes are computed > (e.g. not on closed boundaries). Basically these are all the faces with > support size 2 (aside from some complications related to boundary > conditions). > > To do this I clone the DM using DMClone() and create a section on it > with DMPlexCreateSection(), passing in a label which has value 1 on all > the required faces. The problem then consists of labelling the > appropriate faces. > > The tricky part is that when a face exists on more than one process, you > have to make sure it has the same label value on all processes, > otherwise an error results when you try and create a global vector on > the DM. Because I distribute the DMPlex with overlap = 1, a face can > have different support sizes on different processes. > > So on each process, I need to label a face if it has support size 2 on > *any* process. > > Would it be possible to use PetscSFBcast and the point SF from > DMGetPointSF() to somehow broadcast the support sizes? Or is there a > simpler way? > > - Adrian > > -- > Dr Adrian Croucher > Senior Research Fellow > Department of Engineering Science > University of Auckland, New Zealand > email: a.croucher at auckland.ac.nz > tel: +64 (0)9 923 4611 > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wence at gmx.li Fri Jun 11 06:37:55 2021 From: wence at gmx.li (Lawrence Mitchell) Date: Fri, 11 Jun 2021 12:37:55 +0100 Subject: [petsc-users] flux vector In-Reply-To: References: <2e0e287c-77cf-c375-3b92-c80816c30724@auckland.ac.nz> Message-ID: > On 11 Jun 2021, at 11:29, Mark Adams wrote: > > This is a Matt question, but You can set a "Boundary" label (https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMPLEX/DMPlexMarkBoundaryFaces.html) and that will label boundaries. > Next you want the dual of that ... Note that if you want this to be the _global_ boundary you must call it before mesh distribution. You need to reduce the support size over the point SF, where you would make a contribution to the count only if the cell is in the owned part of your local cells. So (pseudocode): sf = DMGetPointSF(dm); // since you're only going to reduce over facet points you could create an embedded sf here, but meh. nroots, nleaves, ilocal, iremote = SFGetGraph(sf); cstart, cend = DMPlexGetChart(dm); count = PetscCalloc(nroots); // nroots should be == cend - cstart fstart, fend = DMPlexGetDepthStratum(dm, 1); // codim 1 entities for f = fstart; f < fend; f++ { supportsize = DMPlexGetSupportSize(dm, f); support = DMPlexGetSupport(dm, f); for c in support { if (c is owned) { // c not in ilocal count[f]++; } } } gcount = PetscMalloc(nroots); PetscSFReduceBegin(sf, count, gcount, MPI_SUM); PetscSFReduceEnd(sf, count, gcount, MPI_SUM); PetscSFBcastBegin(sf, gcount, count, MPI_REPLACE); PetscSFBcastEnd(sf, gcount, count, MPI_REPLACE); interior_faces = DMLabelCreate("interior_faces") for f = fstart; f < fend; f++ { if count[f] == 2 { DMLabelSetValue(interior_faces, f, 1); } } (Obviously) untested. Lawrence From knepley at gmail.com Fri Jun 11 07:19:30 2021 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 11 Jun 2021 08:19:30 -0400 Subject: [petsc-users] flux vector In-Reply-To: References: <2e0e287c-77cf-c375-3b92-c80816c30724@auckland.ac.nz> Message-ID: On Fri, Jun 11, 2021 at 7:38 AM Lawrence Mitchell wrote: > > > > On 11 Jun 2021, at 11:29, Mark Adams wrote: > > > > This is a Matt question, but You can set a "Boundary" label ( > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMPLEX/DMPlexMarkBoundaryFaces.html) > and that will label boundaries. > > Next you want the dual of that ... > Before doing what Lawrence suggests, I would like to talk this through. I don't think we should need communication. So, Goal: Section with dofs on each face that has support 2 (separates 2 cells) Okay, suppose we loop over our local faces and get the support size of each. Boundary faces are disqualified, which is good, but if a face is on the boundary with another process, it will also have support size 1. Thus, let us distribute with overlap = 1. Now if we loop over that same face set, we will get the "right answer" for the support size. However, using overlap = 1 put in a bunch of new faces. We do not care about the ones on the process boundary. They will be handled by the other process. We do care about faces between two ghost cells, since they will be a false positive. Luckily, these are labeled by the "ghost" label. So I think this is our algorithm: 1) DMClone(dm, &fdm); 2) DMGetLocalSection(fdm, &ls); 3) Mark each face with 2 neighbors that is not a ghost DMGetLabel(dm, "ghost", &ghostLabel); DMGetHeightStratum(fdm, 1, &fStart, &fEnd); for (f = fStartl f < fEnd; ++f) { PetscInt supportSize, ghost; DMGetSupportSize(fdm, f, &supportSize); DMLabelGetValue(ghostLabel, f, &ghost); if (ghost < 0 && supportSize == 2) PetscSectionSetDof(ls, f, 1); } 4) The global section will be created automatically when we ask for it DMGetGlobalSection(fdm, &s); Let me know if this does not work. Thanks, Matt > Note that if you want this to be the _global_ boundary you must call it > before mesh distribution. > > You need to reduce the support size over the point SF, where you would > make a contribution to the count only if the cell is in the owned part of > your local cells. > > So (pseudocode): > > sf = DMGetPointSF(dm); > > // since you're only going to reduce over facet points you could create an > embedded sf here, but meh. > > nroots, nleaves, ilocal, iremote = SFGetGraph(sf); > > cstart, cend = DMPlexGetChart(dm); > > count = PetscCalloc(nroots); // nroots should be == cend - cstart > > fstart, fend = DMPlexGetDepthStratum(dm, 1); // codim 1 entities > > for f = fstart; f < fend; f++ { > supportsize = DMPlexGetSupportSize(dm, f); > support = DMPlexGetSupport(dm, f); > for c in support { > if (c is owned) { // c not in ilocal > count[f]++; > } > } > } > > gcount = PetscMalloc(nroots); > PetscSFReduceBegin(sf, count, gcount, MPI_SUM); > PetscSFReduceEnd(sf, count, gcount, MPI_SUM); > PetscSFBcastBegin(sf, gcount, count, MPI_REPLACE); > PetscSFBcastEnd(sf, gcount, count, MPI_REPLACE); > interior_faces = DMLabelCreate("interior_faces") > for f = fstart; f < fend; f++ { > if count[f] == 2 { > DMLabelSetValue(interior_faces, f, 1); > } > } > > > (Obviously) untested. > > Lawrence > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From kaushikv318 at gmail.com Fri Jun 11 10:16:09 2021 From: kaushikv318 at gmail.com (Kaushik Vijaykumar) Date: Fri, 11 Jun 2021 11:16:09 -0400 Subject: [petsc-users] Copy stiffness matrix to jacobian matrix in form jacobian In-Reply-To: References: Message-ID: Thanks Barry for your comments. Based on your suggestion I tried the following Main() { ierr = SNESSetFunction(snes,r1,FormFunction,&this_ctx);CHKERRQ(ierr); ierr = SNESSetJacobian(snes,jac,NULL,FormJacobian,&this_ctx);CHKERRQ(ierr); ierr = SNESSetFromOptions(snes);CHKERRQ(ierr); ierr = SNESSolve(snes, NULL, u);CHKERRQ(ierr); } In formfunction I populate the jacobian "*jac*" Formfunction(snes,Vec x,Vec F,void* ctx) { ierr = SNESGetJacobian(snes,&jac,Null,NULL,NULL);CHKERRQ(ierr); // "*Populate jac*" ierr = MatAssemblyBegin(jac,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); ierr = MatAssemblyEnd(jac,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); } FormJacobian(SNES snes,Vec x,Mat jac,Mat B,void *ctx) { ierr = MatAssemblyBegin(B,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); ierr = MatAssemblyEnd(B,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); ierr = MatAssemblyBegin(jac,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); ierr = MatAssemblyEnd(jac,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); } When I execute this I get a Petsc error, object in wrong state missing diagonal entry. I am not sure what I am missing here? Thanks Kaushik On Thu, Jun 10, 2021 at 6:37 PM Barry Smith wrote: > > > > On Jun 10, 2021, at 3:29 PM, Kaushik Vijaykumar > wrote: > > > > Hi everyone, > > > > I am trying to copy the stiffness matrix that I generated in form > function to the jacobian matrix jac in form jacobian. The piece of code for > that: > > > > > If you have created jac and B in your main program and passed them with > SNESSetJacobian() then you should be able to just use > > > ierr = MatCopy(ptr->K,jac,SAME_NONZERO_PATTERN);CHKERRQ(ierr); > > Generally jac and B are the same matrix so you don't need the second copy. > > ierr = MatCopy(ptr->K,B,SAME_NONZERO_PATTERN);CHKERRQ(ierr); > > If they are sometimes the same and sometime not you can do > > if (jac != B) {ierr = > MatCopy(ptr->K,B,SAME_NONZERO_PATTERN);CHKERRQ(ierr); } > > > The following won't work because you are creating new local matrices in > your form jacobian that don't exist outside of that routine. > > > > > ierr = MatDuplicate(ptr->K,MAT_COPY_VALUES,&jac);CHKERRQ(ierr); > > ierr = MatDuplicate(ptr->K,MAT_COPY_VALUES,&B);CHKERRQ(ierr); > > > > ierr = MatCopy(ptr->K,jac,SAME_NONZERO_PATTERN);CHKERRQ(ierr); > > ierr = MatCopy(ptr->K,B,SAME_NONZERO_PATTERN);CHKERRQ(ierr); > > > > > If the matrix you are creating in form function is the Jacobian then > you don't need to use the memory and time to make copies. You can just form > the matrix directly into the Jacobian you will use for SNES. You can obtain > this with a call > > SNESGetJacobian(snes,&jac,NULL,NULL,NULL); > > in your form function and just put the matrix values directly into jac. > Then your form Jacobian does not need to do anything but return since the > jac already contains the Jacobian. > > On the off-change you are using the iteration A(x^n) x^{n+1} = b(x^n} and > not using Newtons' method (which is why you compute A in form function, > then you might be better off using SNESSetPicard() instead of > SNESSetFunction(), SNESSetJacobian(). > > > > > Thanks > > Kaushik > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Fri Jun 11 10:33:50 2021 From: bsmith at petsc.dev (Barry Smith) Date: Fri, 11 Jun 2021 10:33:50 -0500 Subject: [petsc-users] Copy stiffness matrix to jacobian matrix in form jacobian In-Reply-To: References: Message-ID: <72658765-A7DC-4C47-84F0-BD366EF22A4B@petsc.dev> > When I execute this I get a Petsc error, object in wrong state missing diagonal entry. When do you get this error? During a TSSolve? While KSP is building a preconditioner. Some parts of PETSc require that you put entries (even if they are zero) on all diagonal entries in the Jacobian. So likely you need to just make sure that your populate jac routine puts/adds 0 to all diagonal entries. Barry > On Jun 11, 2021, at 10:16 AM, Kaushik Vijaykumar wrote: > > Thanks Barry for your comments. Based on your suggestion I tried the following > > Main() > { > ierr = SNESSetFunction(snes,r1,FormFunction,&this_ctx);CHKERRQ(ierr); > ierr = SNESSetJacobian(snes,jac,NULL,FormJacobian,&this_ctx);CHKERRQ(ierr); > ierr = SNESSetFromOptions(snes);CHKERRQ(ierr); > ierr = SNESSolve(snes, NULL, u);CHKERRQ(ierr); > } > > In formfunction I populate the jacobian "jac" > > Formfunction(snes,Vec x,Vec F,void* ctx) > { > ierr = SNESGetJacobian(snes,&jac,Null,NULL,NULL);CHKERRQ(ierr); > > // "Populate jac" > ierr = MatAssemblyBegin(jac,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > ierr = MatAssemblyEnd(jac,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > > } > FormJacobian(SNES snes,Vec x,Mat jac,Mat B,void *ctx) > { > ierr = MatAssemblyBegin(B,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > ierr = MatAssemblyEnd(B,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > > ierr = MatAssemblyBegin(jac,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > ierr = MatAssemblyEnd(jac,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > } > > When I execute this I get a Petsc error, object in wrong state missing diagonal entry. > > I am not sure what I am missing here? > > Thanks > Kaushik > > > On Thu, Jun 10, 2021 at 6:37 PM Barry Smith > wrote: > > > > On Jun 10, 2021, at 3:29 PM, Kaushik Vijaykumar > wrote: > > > > Hi everyone, > > > > I am trying to copy the stiffness matrix that I generated in form function to the jacobian matrix jac in form jacobian. The piece of code for that: > > > > > If you have created jac and B in your main program and passed them with SNESSetJacobian() then you should be able to just use > > > ierr = MatCopy(ptr->K,jac,SAME_NONZERO_PATTERN);CHKERRQ(ierr); > > Generally jac and B are the same matrix so you don't need the second copy. > > ierr = MatCopy(ptr->K,B,SAME_NONZERO_PATTERN);CHKERRQ(ierr); > > If they are sometimes the same and sometime not you can do > > if (jac != B) {ierr = MatCopy(ptr->K,B,SAME_NONZERO_PATTERN);CHKERRQ(ierr); } > > > The following won't work because you are creating new local matrices in your form jacobian that don't exist outside of that routine. > > > > > ierr = MatDuplicate(ptr->K,MAT_COPY_VALUES,&jac);CHKERRQ(ierr); > > ierr = MatDuplicate(ptr->K,MAT_COPY_VALUES,&B);CHKERRQ(ierr); > > > > ierr = MatCopy(ptr->K,jac,SAME_NONZERO_PATTERN);CHKERRQ(ierr); > > ierr = MatCopy(ptr->K,B,SAME_NONZERO_PATTERN);CHKERRQ(ierr); > > > > > If the matrix you are creating in form function is the Jacobian then you don't need to use the memory and time to make copies. You can just form the matrix directly into the Jacobian you will use for SNES. You can obtain this with a call > > SNESGetJacobian(snes,&jac,NULL,NULL,NULL); > > in your form function and just put the matrix values directly into jac. Then your form Jacobian does not need to do anything but return since the jac already contains the Jacobian. > > On the off-change you are using the iteration A(x^n) x^{n+1} = b(x^n} and not using Newtons' method (which is why you compute A in form function, then you might be better off using SNESSetPicard() instead of SNESSetFunction(), SNESSetJacobian(). > > > > > Thanks > > Kaushik > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wence at gmx.li Sat Jun 12 04:25:15 2021 From: wence at gmx.li (Lawrence Mitchell) Date: Sat, 12 Jun 2021 10:25:15 +0100 Subject: [petsc-users] flux vector In-Reply-To: References: <2e0e287c-77cf-c375-3b92-c80816c30724@auckland.ac.nz> Message-ID: <084C1974-B0B3-4125-8D3E-175C0732A29E@gmx.li> > On 11 Jun 2021, at 13:19, Matthew Knepley wrote: > > Before doing what Lawrence suggests, I would like to talk this through. I don't think we should need communication. So, > > Goal: Section with dofs on each face that has support 2 (separates 2 cells) > > Okay, suppose we loop over our local faces and get the support size of each. Boundary faces are disqualified, which is good, > but if a face is on the boundary with another process, it will also have support size 1. Thus, let us distribute with overlap = 1. > Now if we loop over that same face set, we will get the "right answer" for the support size. > > However, using overlap = 1 put in a bunch of new faces. We do not care about the ones on the process boundary. They will be > handled by the other process. We do care about faces between two ghost cells, since they will be a false positive. Luckily, these > are labeled by the "ghost" label. So I think this is our algorithm: Ah, I missed that the mesh is already overlapped. > 1) DMClone(dm, &fdm); > > 2) DMGetLocalSection(fdm, &ls); > > 3) Mark each face with 2 neighbors that is not a ghost > > DMGetLabel(dm, "ghost", &ghostLabel); This label is only created if one calls DMPlexConstructGhostCells, I don't understand this code really, but it seems like if I call DMPlexDistribute(dm, overlap=1, &newdm) and then DMPlexConstructGhostCells, the "depth 2" faces/etc... will be marked with the ghost label, but the "depth 1" faces will not be. That said, I agree with Matt's algorithm, and if the ghost label isn't available or set up right you can replace DMLabelGetValue(ghostLabel, f, &ghost) by checking if the point f is in the ilocal array of the DM's pointSF, I think. > DMGetHeightStratum(fdm, 1, &fStart, &fEnd); > for (f = fStartl f < fEnd; ++f) { > PetscInt supportSize, ghost; > > DMGetSupportSize(fdm, f, &supportSize); > DMLabelGetValue(ghostLabel, f, &ghost); > if (ghost < 0 && supportSize == 2) PetscSectionSetDof(ls, f, 1); > } > > 4) The global section will be created automatically when we ask for it > > DMGetGlobalSection(fdm, &s); > > Let me know if this does not work. Lawrence From knepley at gmail.com Sat Jun 12 05:29:46 2021 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 12 Jun 2021 06:29:46 -0400 Subject: [petsc-users] flux vector In-Reply-To: <084C1974-B0B3-4125-8D3E-175C0732A29E@gmx.li> References: <2e0e287c-77cf-c375-3b92-c80816c30724@auckland.ac.nz> <084C1974-B0B3-4125-8D3E-175C0732A29E@gmx.li> Message-ID: On Sat, Jun 12, 2021 at 5:25 AM Lawrence Mitchell wrote: > > > On 11 Jun 2021, at 13:19, Matthew Knepley wrote: > > > > Before doing what Lawrence suggests, I would like to talk this through. > I don't think we should need communication. So, > > > > Goal: Section with dofs on each face that has support 2 (separates 2 > cells) > > > > Okay, suppose we loop over our local faces and get the support size of > each. Boundary faces are disqualified, which is good, > > but if a face is on the boundary with another process, it will also have > support size 1. Thus, let us distribute with overlap = 1. > > Now if we loop over that same face set, we will get the "right answer" > for the support size. > > > > However, using overlap = 1 put in a bunch of new faces. We do not care > about the ones on the process boundary. They will be > > handled by the other process. We do care about faces between two ghost > cells, since they will be a false positive. Luckily, these > > are labeled by the "ghost" label. So I think this is our algorithm: > > Ah, I missed that the mesh is already overlapped. > > > 1) DMClone(dm, &fdm); > > > > 2) DMGetLocalSection(fdm, &ls); > > > > 3) Mark each face with 2 neighbors that is not a ghost > > > > DMGetLabel(dm, "ghost", &ghostLabel); > > This label is only created if one calls DMPlexConstructGhostCells, Shoot, that is the wrong workflow. > I don't understand this code really, but it seems like if I call > DMPlexDistribute(dm, overlap=1, &newdm) and then DMPlexConstructGhostCells, > the "depth 2" faces/etc... will be marked with the ghost label, but the > "depth 1" faces will not be. > Yes, I only marked those because those are the only ones you might calculate fluxes on. > That said, I agree with Matt's algorithm, and if the ghost label isn't > available or set up right you can replace > > DMLabelGetValue(ghostLabel, f, &ghost) > > by checking if the point f is in the ilocal array of the DM's pointSF, I > think. > It is a little more complicated. The check for "ghost" is that you have 2 cells in the support, but they are both not owned, meaning both are in the SF. Thanks, Matt > > DMGetHeightStratum(fdm, 1, &fStart, &fEnd); > > for (f = fStartl f < fEnd; ++f) { > > PetscInt supportSize, ghost; > > > > DMGetSupportSize(fdm, f, &supportSize); > > DMLabelGetValue(ghostLabel, f, &ghost); > > if (ghost < 0 && supportSize == 2) PetscSectionSetDof(ls, f, 1); > > } > > > > 4) The global section will be created automatically when we ask for it > > > > DMGetGlobalSection(fdm, &s); > > > > Let me know if this does not work. > > > Lawrence -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From popov at uni-mainz.de Sun Jun 13 09:54:48 2021 From: popov at uni-mainz.de (Anton Popov) Date: Sun, 13 Jun 2021 16:54:48 +0200 Subject: [petsc-users] Change Amat in FormJacobian Message-ID: <1e340ba2-32b7-f397-282b-ef0c8289c62d@uni-mainz.de> Hi, I want a simple(?) thing. I want to decide and be able to assign the Jacobian matrix (Amat) on the fly within the FormJacobian function (i.e. during Newton iteration) to one of the following options: 1) built-in MFFD approximation 2) assembled preconditioner matrix (Pmat) I have not found this scenario demonstrated in any example, therefore I?m asking how to do that. Currently I do the following: 1) setup Amat as a shell matrix with a MATOP_MULT operation that simply retrieves a matrix object form its context and calls MatMult on it. 2) if I need MFFD, I put a matrix generated with MatCreateSNESMF in the Amat context (of course I also call MatMFFDComputeJacobian before that). 3) if I need Pmat, I simply put Pmat in the Amat context. 4) call MatAssemblyBegin/End on Amat So far so good. However, shell Amat and assembled Pmat generate a problem if Galerkin multigrid is requested as a preconditioner (I just test on 1 CPU): [0]PETSC ERROR: MatPtAP requires A, shell, to be compatible with P, seqaij (Misses composed function MatPtAP_shell_seqaij_C) [0]PETSC ERROR: #1 MatPtAP() [0]PETSC ERROR: #2 MatGalerkin() [0]PETSC ERROR: #3 PCSetUp_MG() [0]PETSC ERROR: #4 PCSetUp() [0]PETSC ERROR: #5 KSPSetUp() [0]PETSC ERROR: #6 KSPSolve() [0]PETSC ERROR: #7 SNESSolve_NEWTONLS() [0]PETSC ERROR: #8 SNESSolve() It seems like PETSc tries to apply Galerkin coarsening to the shell Amat matrix instead of the assembled Pmat. Why? Pmat is internally generated by SNES using a DMDA attached to SNES, so it has everything to define Galerkin coarsening. And it actually works if I just get rid of the shell Amat. I can get around this problem by wrapping a PC object in a shell matrix and pass it as Pmat. This is a rather ugly approach, so I wonder how to resolve the situation in a better way, if it is possible. Thanks. Anton From knepley at gmail.com Sun Jun 13 11:21:05 2021 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 13 Jun 2021 12:21:05 -0400 Subject: [petsc-users] Change Amat in FormJacobian In-Reply-To: <1e340ba2-32b7-f397-282b-ef0c8289c62d@uni-mainz.de> References: <1e340ba2-32b7-f397-282b-ef0c8289c62d@uni-mainz.de> Message-ID: On Sun, Jun 13, 2021 at 10:55 AM Anton Popov wrote: > Hi, > > I want a simple(?) thing. I want to decide and be able to assign the > Jacobian matrix (Amat) on the fly within the FormJacobian function (i.e. > during Newton iteration) to one of the following options: > > 1) built-in MFFD approximation > 2) assembled preconditioner matrix (Pmat) > > I have not found this scenario demonstrated in any example, therefore > I?m asking how to do that. > > Currently I do the following: > > 1) setup Amat as a shell matrix with a MATOP_MULT operation that simply > retrieves a matrix object form its context and calls MatMult on it. > > 2) if I need MFFD, I put a matrix generated with MatCreateSNESMF in the > Amat context (of course I also call MatMFFDComputeJacobian before that). > > 3) if I need Pmat, I simply put Pmat in the Amat context. > > 4) call MatAssemblyBegin/End on Amat > > So far so good. > > However, shell Amat and assembled Pmat generate a problem if Galerkin > multigrid is requested as a preconditioner (I just test on 1 CPU): > > [0]PETSC ERROR: MatPtAP requires A, shell, to be compatible with P, > seqaij (Misses composed function MatPtAP_shell_seqaij_C) > [0]PETSC ERROR: #1 MatPtAP() > [0]PETSC ERROR: #2 MatGalerkin() > [0]PETSC ERROR: #3 PCSetUp_MG() > [0]PETSC ERROR: #4 PCSetUp() > [0]PETSC ERROR: #5 KSPSetUp() > [0]PETSC ERROR: #6 KSPSolve() > [0]PETSC ERROR: #7 SNESSolve_NEWTONLS() > [0]PETSC ERROR: #8 SNESSolve() > > It seems like PETSc tries to apply Galerkin coarsening to the shell Amat > matrix instead of the assembled Pmat. Why? > > Pmat is internally generated by SNES using a DMDA attached to SNES, so > it has everything to define Galerkin coarsening. And it actually works > if I just get rid of the shell Amat. > > I can get around this problem by wrapping a PC object in a shell matrix > and pass it as Pmat. This is a rather ugly approach, so I wonder how to > resolve the situation in a better way, if it is possible. > Hi Anton, You are correct that there is no specific test, so I can believe that a bug might be lurking here. I believe the way it is supposed to work is that you set the type of Galerkin coarsening that you want https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCMGSetGalerkin.html So I am thinking you want 'pmat' only, and you would be in charge of making the coarse MFFD operator and telling PCMG about it. I could imagine that you might want us to automatically do that, but we would need some way to know that it is MFFD, and with the scheme above we do not have that. What is the advantage of switching representations during the Newton iteration? Thanks, Matt > Thanks. > Anton > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Sun Jun 13 13:52:05 2021 From: bsmith at petsc.dev (Barry Smith) Date: Sun, 13 Jun 2021 13:52:05 -0500 Subject: [petsc-users] Change Amat in FormJacobian In-Reply-To: References: <1e340ba2-32b7-f397-282b-ef0c8289c62d@uni-mainz.de> Message-ID: Anton, -pc_mg_galerkin pmat Though it seems simple, there is some subtly in swapping out matrices with SNES. When using multigrid with SNES there are at least five distinct uses of the Jacobian operator. Perform matrix-vector product in line search to check Wolf sufficient decrease convergence criteria Perform the matrix-vector product for the Krylov accelerator of the system Perform smoothing on the finest level of MG Perform the matrix-vector product needed on the finest level of MG to compute the residual that will be restricted to the coarser level of multigrid When using Galerkin products to compute the coarser grid operator performing the Galerkin matrix triple product when one swaps out the mat, which of these do they wish to change? The first two seem to naturally go together as do the last three. In your case I presume you want to swap for the first two, but always use pmat for the last three? To achieve this you will also need -pc_use_amat false If you run with -info and -snes_view it will print out some of the information about which operator it is using for each case, but probably not all of them. Note: if the pmat is actually an accurate computation of the Jacobian then it is likely best not to ever use a matrix-free product. It is only when pmat is approximated in some specific way that using the matrix-free product would be useful to insure the "Newton" method actually computes a Newton step. Barry > On Jun 13, 2021, at 11:21 AM, Matthew Knepley wrote: > > On Sun, Jun 13, 2021 at 10:55 AM Anton Popov > wrote: > Hi, > > I want a simple(?) thing. I want to decide and be able to assign the > Jacobian matrix (Amat) on the fly within the FormJacobian function (i.e. > during Newton iteration) to one of the following options: > > 1) built-in MFFD approximation > 2) assembled preconditioner matrix (Pmat) > > I have not found this scenario demonstrated in any example, therefore > I?m asking how to do that. > > Currently I do the following: > > 1) setup Amat as a shell matrix with a MATOP_MULT operation that simply > retrieves a matrix object form its context and calls MatMult on it. > > 2) if I need MFFD, I put a matrix generated with MatCreateSNESMF in the > Amat context (of course I also call MatMFFDComputeJacobian before that). > > 3) if I need Pmat, I simply put Pmat in the Amat context. > > 4) call MatAssemblyBegin/End on Amat > > So far so good. > > However, shell Amat and assembled Pmat generate a problem if Galerkin > multigrid is requested as a preconditioner (I just test on 1 CPU): > > [0]PETSC ERROR: MatPtAP requires A, shell, to be compatible with P, > seqaij (Misses composed function MatPtAP_shell_seqaij_C) > [0]PETSC ERROR: #1 MatPtAP() > [0]PETSC ERROR: #2 MatGalerkin() > [0]PETSC ERROR: #3 PCSetUp_MG() > [0]PETSC ERROR: #4 PCSetUp() > [0]PETSC ERROR: #5 KSPSetUp() > [0]PETSC ERROR: #6 KSPSolve() > [0]PETSC ERROR: #7 SNESSolve_NEWTONLS() > [0]PETSC ERROR: #8 SNESSolve() > > It seems like PETSc tries to apply Galerkin coarsening to the shell Amat > matrix instead of the assembled Pmat. Why? > > Pmat is internally generated by SNES using a DMDA attached to SNES, so > it has everything to define Galerkin coarsening. And it actually works > if I just get rid of the shell Amat. > > I can get around this problem by wrapping a PC object in a shell matrix > and pass it as Pmat. This is a rather ugly approach, so I wonder how to > resolve the situation in a better way, if it is possible. > > Hi Anton, > > You are correct that there is no specific test, so I can believe that a bug might be lurking here. > I believe the way it is supposed to work is that you set the type of Galerkin coarsening that you > want > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCMGSetGalerkin.html > > So I am thinking you want 'pmat' only, and you would be in charge of making the coarse MFFD operator > and telling PCMG about it. I could imagine that you might want us to automatically do that, but we would > need some way to know that it is MFFD, and with the scheme above we do not have that. > > What is the advantage of switching representations during the Newton iteration? > > Thanks, > > Matt > > Thanks. > Anton > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.croucher at auckland.ac.nz Sun Jun 13 18:47:57 2021 From: a.croucher at auckland.ac.nz (Adrian Croucher) Date: Mon, 14 Jun 2021 11:47:57 +1200 Subject: [petsc-users] flux vector In-Reply-To: References: <2e0e287c-77cf-c375-3b92-c80816c30724@auckland.ac.nz> Message-ID: <17d35df1-5ae0-5cf4-4cda-3592e56776d9@auckland.ac.nz> hi, thanks for the suggestions! On 12/06/21 12:19 am, Matthew Knepley wrote: > However, using overlap = 1 put in a bunch of new faces. We do not care > about the ones on the process boundary. They will be > handled by the other process. We do care about faces between two ghost > cells, since they will be a false positive. Luckily, these > are labeled by the "ghost" label. I think we *do* have to care about the faces on the process boundary (assuming by that you mean the faces with support size 1 on the outside edge of the partition ghost cells), because they can be ghosts of flux faces on other processes. If we ignore them they will have no dofs on the current process, but they could have dofs on another one. That inconsistency is the problem that causes the error when you create the global section. Also, I don't think we want to exclude faces between partition ghost cells. Those faces will definitely be ghosts of a flux face on another process. Again, if we exclude them the dofs will not be consistent across processes. We might not actually compute a flux on those faces locally, but there has to be a space for them anyway. I have found a simple algorithm now which seems to work in all my test cases, though I still wonder if there is a better way. The algorithm is: 1) label the global boundary faces before distribution (all faces with support size 1), using DMPlexMarkBoundaryFaces() 2) label any open boundary faces (on which Dirichlet BCs are applied) - I didn't mention these before, but they need to be included as flux faces 3) after distribution, loop over all faces on current process: ??? if face on open boundary: label face as a flux face ??? else: ????? if face not on global boundary: label face as a flux face Here the only test of support size is in step 1), which ensures that the dofs are always consistent between processes. Ultimately, the support size on the local process is not really relevant or reliable. The main thing I don't like about this algorithm is that it relies on looping over all mesh faces in serial during step 1). I would rather not add to the serial part of the code and would prefer if it could all be done in parallel, after distribution. Maybe I'm worrying unnecessarily, and just getting the support size of each face is cheap enough that this won't be a significant issue? - Adrian -- Dr Adrian Croucher Senior Research Fellow Department of Engineering Science University of Auckland, New Zealand email: a.croucher at auckland.ac.nz tel: +64 (0)9 923 4611 -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Jun 14 04:54:12 2021 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 14 Jun 2021 05:54:12 -0400 Subject: [petsc-users] flux vector In-Reply-To: <17d35df1-5ae0-5cf4-4cda-3592e56776d9@auckland.ac.nz> References: <2e0e287c-77cf-c375-3b92-c80816c30724@auckland.ac.nz> <17d35df1-5ae0-5cf4-4cda-3592e56776d9@auckland.ac.nz> Message-ID: On Sun, Jun 13, 2021 at 7:48 PM Adrian Croucher wrote: > hi, thanks for the suggestions! > On 12/06/21 12:19 am, Matthew Knepley wrote: > > However, using overlap = 1 put in a bunch of new faces. We do not care > about the ones on the process boundary. They will be > handled by the other process. We do care about faces between two ghost > cells, since they will be a false positive. Luckily, these > are labeled by the "ghost" label. > > I think we *do* have to care about the faces on the process boundary > (assuming by that you mean the faces with support size 1 on the outside > edge of the partition ghost cells), because they can be ghosts of flux > faces on other processes. If we ignore them they will have no dofs on the > current process, but they could have dofs on another one. That > inconsistency is the problem that causes the error when you create the > global section. > > Also, I don't think we want to exclude faces between partition ghost > cells. Those faces will definitely be ghosts of a flux face on another > process. Again, if we exclude them the dofs will not be consistent across > processes. We might not actually compute a flux on those faces locally, but > there has to be a space for them anyway. > > I have found a simple algorithm now which seems to work in all my test > cases, though I still wonder if there is a better way. The algorithm is: > > 1) label the global boundary faces before distribution (all faces with > support size 1), using DMPlexMarkBoundaryFaces() > > 2) label any open boundary faces (on which Dirichlet BCs are applied) - I > didn't mention these before, but they need to be included as flux faces > > 3) after distribution, loop over all faces on current process: > > if face on open boundary: label face as a flux face > > else: > > if face not on global boundary: label face as a flux face > > Here the only test of support size is in step 1), which ensures that the > dofs are always consistent between processes. Ultimately, the support size > on the local process is not really relevant or reliable. > > The main thing I don't like about this algorithm is that it relies on > looping over all mesh faces in serial during step 1). I would rather not > add to the serial part of the code and would prefer if it could all be done > in parallel, after distribution. Maybe I'm worrying unnecessarily, and just > getting the support size of each face is cheap enough that this won't be a > significant issue? > > I see that I have misunderstood something. I thought you wanted to put dofs on only the faces that you compute. It everyone only puts dofs on the computed faces, then I thought you would get a globally consistent Section. However, now I see that you want unknowns on all local faces that anyone computes, so that you can get those values from the other process. Okay, I think it is not so hard to get what you want in parallel. There are only two kinds of faces with supportSize == 1: a) Faces on the global boundary b) Faces which are "shared" It is the second set that is somewhat confusing because PetscSF does not have 2-sided information by default. However, it can make it. There is a two-step check for "shared": 1) Is the face in the PetscSF? Here you just check for it in the sorted "locals" array from PetscSFGetGraph() 2) Is the face ghosted on another process? You can get this from PetscSFGetRootRanks(). I just wrote a small function to check for "shared" points. After that, I think you can just run 1) After distribution, loop overall faces on current process If face on open boundary, label face as flux face else: if face has supportSize != 1 or (supportSize == 1 && shared), label face as flux face Thanks, Matt > - Adrian > > -- > Dr Adrian Croucher > Senior Research Fellow > Department of Engineering Science > University of Auckland, New Zealand > email: a.croucher at auckland.ac.nz > tel: +64 (0)9 923 4611 > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From kaushikv318 at gmail.com Mon Jun 14 06:20:47 2021 From: kaushikv318 at gmail.com (Kaushik Vijaykumar) Date: Mon, 14 Jun 2021 07:20:47 -0400 Subject: [petsc-users] Copy stiffness matrix to jacobian matrix in form jacobian In-Reply-To: <72658765-A7DC-4C47-84F0-BD366EF22A4B@petsc.dev> References: <72658765-A7DC-4C47-84F0-BD366EF22A4B@petsc.dev> Message-ID: Thanks for the suggestion Barry, I did do that by setting it in multiple ways first by setting Formfunction(snes,Vec x,Vec F,void* ctx) { ierr = SNESGetJacobian(snes,&jac,NULL,NULL,NULL);CHKERRQ(ierr); ierr = MatZeroEntries(jac);CHKERRQ(ierr); ..... ierr = MatGetOwnershipRange(jac,&istart,&iend); for(i=istart;i wrote: > > When I execute this I get a Petsc error, object in wrong state missing > diagonal entry. > > > When do you get this error? During a TSSolve? While KSP is building a > preconditioner. > > Some parts of PETSc require that you put entries (even if they are > zero) on all diagonal entries in the Jacobian. So likely you need to just > make sure that your populate jac routine puts/adds 0 to all diagonal > entries. > > Barry > > > > On Jun 11, 2021, at 10:16 AM, Kaushik Vijaykumar > wrote: > > Thanks Barry for your comments. Based on your suggestion I tried the > following > > Main() > { > ierr = SNESSetFunction(snes,r1,FormFunction,&this_ctx);CHKERRQ(ierr); > ierr = SNESSetJacobian(snes,jac,NULL,FormJacobian,&this_ctx);CHKERRQ(ierr); > ierr = SNESSetFromOptions(snes);CHKERRQ(ierr); > ierr = SNESSolve(snes, NULL, u);CHKERRQ(ierr); > } > > In formfunction I populate the jacobian "*jac*" > > Formfunction(snes,Vec x,Vec F,void* ctx) > { > ierr = SNESGetJacobian(snes,&jac,Null,NULL,NULL);CHKERRQ(ierr); > > // "*Populate jac*" > ierr = MatAssemblyBegin(jac,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > ierr = MatAssemblyEnd(jac,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > > } > FormJacobian(SNES snes,Vec x,Mat jac,Mat B,void *ctx) > { > ierr = MatAssemblyBegin(B,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > ierr = MatAssemblyEnd(B,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > > ierr = MatAssemblyBegin(jac,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > ierr = MatAssemblyEnd(jac,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > } > > When I execute this I get a Petsc error, object in wrong state missing > diagonal entry. > > I am not sure what I am missing here? > > Thanks > Kaushik > > > On Thu, Jun 10, 2021 at 6:37 PM Barry Smith wrote: > >> >> >> > On Jun 10, 2021, at 3:29 PM, Kaushik Vijaykumar >> wrote: >> > >> > Hi everyone, >> > >> > I am trying to copy the stiffness matrix that I generated in form >> function to the jacobian matrix jac in form jacobian. The piece of code for >> that: >> > >> > >> If you have created jac and B in your main program and passed them >> with SNESSetJacobian() then you should be able to just use >> >> > ierr = MatCopy(ptr->K,jac,SAME_NONZERO_PATTERN);CHKERRQ(ierr); >> >> Generally jac and B are the same matrix so you don't need the second >> copy. >> > ierr = MatCopy(ptr->K,B,SAME_NONZERO_PATTERN);CHKERRQ(ierr); >> >> If they are sometimes the same and sometime not you can do >> >> if (jac != B) {ierr = >> MatCopy(ptr->K,B,SAME_NONZERO_PATTERN);CHKERRQ(ierr); } >> >> >> The following won't work because you are creating new local matrices >> in your form jacobian that don't exist outside of that routine. >> >> > >> > ierr = MatDuplicate(ptr->K,MAT_COPY_VALUES,&jac);CHKERRQ(ierr); >> > ierr = MatDuplicate(ptr->K,MAT_COPY_VALUES,&B);CHKERRQ(ierr); >> > >> > ierr = MatCopy(ptr->K,jac,SAME_NONZERO_PATTERN);CHKERRQ(ierr); >> > ierr = MatCopy(ptr->K,B,SAME_NONZERO_PATTERN);CHKERRQ(ierr); >> > >> > >> If the matrix you are creating in form function is the Jacobian then >> you don't need to use the memory and time to make copies. You can just form >> the matrix directly into the Jacobian you will use for SNES. You can obtain >> this with a call >> >> SNESGetJacobian(snes,&jac,NULL,NULL,NULL); >> >> in your form function and just put the matrix values directly into jac. >> Then your form Jacobian does not need to do anything but return since the >> jac already contains the Jacobian. >> >> On the off-change you are using the iteration A(x^n) x^{n+1} = b(x^n} and >> not using Newtons' method (which is why you compute A in form function, >> then you might be better off using SNESSetPicard() instead of >> SNESSetFunction(), SNESSetJacobian(). >> >> > >> > Thanks >> > Kaushik >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Jun 14 07:27:13 2021 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 14 Jun 2021 08:27:13 -0400 Subject: [petsc-users] Copy stiffness matrix to jacobian matrix in form jacobian In-Reply-To: References: <72658765-A7DC-4C47-84F0-BD366EF22A4B@petsc.dev> Message-ID: On Mon, Jun 14, 2021 at 7:21 AM Kaushik Vijaykumar wrote: > Thanks for the suggestion Barry, I did do that by setting it in multiple > ways first by setting > > Formfunction(snes,Vec x,Vec F,void* ctx) > { > ierr = SNESGetJacobian(snes,&jac,NULL,NULL,NULL);CHKERRQ(ierr); > ierr = MatZeroEntries(jac);CHKERRQ(ierr); > > ..... > > > ierr = MatGetOwnershipRange(jac,&istart,&iend); > > for(i=istart;i { > val = 0.0; > ierr = MatSetValue(jac, i, i, val, ADD_VALUES);CHKERRQ(ierr); > } > ierr = MatAssemblyBegin(jac,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > ierr = MatAssemblyEnd(jac,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > } > I get the same error as before for a single processor run. > "[0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Object is in wrong state > [0]PETSC ERROR: Matrix is missing diagonal entry 0" > It must be that the matrix you are setting the diagonal for is not the same matrix as referenced in the error message. Thanks, Matt > Thanks > Kaushik > > On Fri, Jun 11, 2021 at 11:33 AM Barry Smith wrote: > >> >> When I execute this I get a Petsc error, object in wrong state missing >> diagonal entry. >> >> >> When do you get this error? During a TSSolve? While KSP is building a >> preconditioner. >> >> Some parts of PETSc require that you put entries (even if they are >> zero) on all diagonal entries in the Jacobian. So likely you need to just >> make sure that your populate jac routine puts/adds 0 to all diagonal >> entries. >> >> Barry >> >> >> >> On Jun 11, 2021, at 10:16 AM, Kaushik Vijaykumar >> wrote: >> >> Thanks Barry for your comments. Based on your suggestion I tried the >> following >> >> Main() >> { >> ierr = SNESSetFunction(snes,r1,FormFunction,&this_ctx);CHKERRQ(ierr); >> ierr = >> SNESSetJacobian(snes,jac,NULL,FormJacobian,&this_ctx);CHKERRQ(ierr); >> ierr = SNESSetFromOptions(snes);CHKERRQ(ierr); >> ierr = SNESSolve(snes, NULL, u);CHKERRQ(ierr); >> } >> >> In formfunction I populate the jacobian "*jac*" >> >> Formfunction(snes,Vec x,Vec F,void* ctx) >> { >> ierr = SNESGetJacobian(snes,&jac,Null,NULL,NULL);CHKERRQ(ierr); >> >> // "*Populate jac*" >> ierr = MatAssemblyBegin(jac,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); >> ierr = MatAssemblyEnd(jac,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); >> >> } >> FormJacobian(SNES snes,Vec x,Mat jac,Mat B,void *ctx) >> { >> ierr = MatAssemblyBegin(B,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); >> ierr = MatAssemblyEnd(B,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); >> >> ierr = MatAssemblyBegin(jac,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); >> ierr = MatAssemblyEnd(jac,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); >> } >> >> When I execute this I get a Petsc error, object in wrong state missing >> diagonal entry. >> >> I am not sure what I am missing here? >> >> Thanks >> Kaushik >> >> >> On Thu, Jun 10, 2021 at 6:37 PM Barry Smith wrote: >> >>> >>> >>> > On Jun 10, 2021, at 3:29 PM, Kaushik Vijaykumar >>> wrote: >>> > >>> > Hi everyone, >>> > >>> > I am trying to copy the stiffness matrix that I generated in form >>> function to the jacobian matrix jac in form jacobian. The piece of code for >>> that: >>> > >>> > >>> If you have created jac and B in your main program and passed them >>> with SNESSetJacobian() then you should be able to just use >>> >>> > ierr = MatCopy(ptr->K,jac,SAME_NONZERO_PATTERN);CHKERRQ(ierr); >>> >>> Generally jac and B are the same matrix so you don't need the second >>> copy. >>> > ierr = MatCopy(ptr->K,B,SAME_NONZERO_PATTERN);CHKERRQ(ierr); >>> >>> If they are sometimes the same and sometime not you can do >>> >>> if (jac != B) {ierr = >>> MatCopy(ptr->K,B,SAME_NONZERO_PATTERN);CHKERRQ(ierr); } >>> >>> >>> The following won't work because you are creating new local matrices >>> in your form jacobian that don't exist outside of that routine. >>> >>> > >>> > ierr = MatDuplicate(ptr->K,MAT_COPY_VALUES,&jac);CHKERRQ(ierr); >>> > ierr = MatDuplicate(ptr->K,MAT_COPY_VALUES,&B);CHKERRQ(ierr); >>> > >>> > ierr = MatCopy(ptr->K,jac,SAME_NONZERO_PATTERN);CHKERRQ(ierr); >>> > ierr = MatCopy(ptr->K,B,SAME_NONZERO_PATTERN);CHKERRQ(ierr); >>> > >>> > >>> If the matrix you are creating in form function is the Jacobian then >>> you don't need to use the memory and time to make copies. You can just form >>> the matrix directly into the Jacobian you will use for SNES. You can obtain >>> this with a call >>> >>> SNESGetJacobian(snes,&jac,NULL,NULL,NULL); >>> >>> in your form function and just put the matrix values directly into jac. >>> Then your form Jacobian does not need to do anything but return since the >>> jac already contains the Jacobian. >>> >>> On the off-change you are using the iteration A(x^n) x^{n+1} = b(x^n} >>> and not using Newtons' method (which is why you compute A in form function, >>> then you might be better off using SNESSetPicard() instead of >>> SNESSetFunction(), SNESSetJacobian(). >>> >>> > >>> > Thanks >>> > Kaushik >>> >>> >> -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From popov at uni-mainz.de Mon Jun 14 07:45:47 2021 From: popov at uni-mainz.de (Anton Popov) Date: Mon, 14 Jun 2021 14:45:47 +0200 Subject: [petsc-users] Change Amat in FormJacobian In-Reply-To: References: <1e340ba2-32b7-f397-282b-ef0c8289c62d@uni-mainz.de> Message-ID: Hi Barry & Matt, thanks for your quick response. These options were exactly what I needed and expected: -pc_mg_galerkin pmat -pc_use_amat false I just assumed that it?s a default behavior of the PC object. So to clarify my case, I don't use nonlinear multigrid. Galerkin is expected to deal with Pmat only, and it's enough if Amat implements a matrix-vector product for the Krylov accelerator. Matt, the reason for switching Amat during the iteration is a quite common Picard-Newton combination. Jacobian matrix gives accurate updates close to the solution, but is rather unstable far form the solution. Picard matrix (approximate Jacobian) is quite the opposite ? it?s kind of stable, but slow. So the idea is to begin the iteration with Picard matrix, and switch to the Jacobian later. If the assembled matrices are used, then the standard SNES interface is just perfect. I can decide how to fill the matrices. But I don?t bother with? Jacobian assembly and want to use a built-in MFFD approximation instead. I did quite a few tests previously and figured out that MFFD is practically the same as closed-from matrix-free Jacobian for the later stages of the iteration. The Picard matrix still does a good job as a preconditioner. But it is important to start the iteration with Picard and only change to MFFD later. Is my workaround with the shell matrix acceptable, or there is a better solution? Thanks, Anton On 13.06.21 20:52, Barry Smith wrote: > > ? Anton, > > ??-pc_mg_galerkin pmat > > ? Though it seems simple, there is some subtly in swapping out > matrices with SNES. > > ? When using multigrid with SNES there are at least five distinct uses > of the Jacobian operator. > > * Perform matrix-vector product in line search to check Wolf > sufficient decrease convergence criteria > * Perform the matrix-vector product for the Krylov accelerator > of the system > * Perform smoothing on the finest level of MG > * Perform the matrix-vector product needed on the finest level > of MG to compute the residual that will be restricted to the > coarser level of multigrid > * When using Galerkin products to compute the coarser grid > operator performing the Galerkin matrix triple product > > > when one swaps out the mat, which of these do they wish to change? The > first two seem to naturally go together as do the last three. In your > case I presume you want to swap for the first two, but always use pmat > for the last three? To achieve this you will also need?-pc_use_amat ?false > > If you run with -info and -snes_view it will print out some of the > information about which operator it is using for each case, but > probably not all of them. > > Note: if the pmat is actually an accurate computation of the Jacobian > then it is likely best not to ever use a matrix-free product. It is > only when pmat is approximated in some specific way that using the > matrix-free product would be useful to insure the "Newton" method > actually computes a Newton step. > > Barry > > > >> On Jun 13, 2021, at 11:21 AM, Matthew Knepley > > wrote: >> >> On Sun, Jun 13, 2021 at 10:55 AM Anton Popov > > wrote: >> >> Hi, >> >> I want a simple(?) thing. I want to decide and be able to assign the >> Jacobian matrix (Amat) on the fly within the FormJacobian >> function (i.e. >> during Newton iteration) to one of the following options: >> >> 1) built-in MFFD approximation >> 2) assembled preconditioner matrix (Pmat) >> >> I have not found this scenario demonstrated in any example, >> therefore >> I?m asking how to do that. >> >> Currently I do the following: >> >> 1) setup Amat as a shell matrix with a MATOP_MULT operation that >> simply >> retrieves a matrix object form its context and calls MatMult on it. >> >> 2) if I need MFFD, I put a matrix generated with MatCreateSNESMF >> in the >> Amat context (of course I also call MatMFFDComputeJacobian before >> that). >> >> 3) if I need Pmat, I simply put Pmat in the Amat context. >> >> 4) call MatAssemblyBegin/End on Amat >> >> So far so good. >> >> However, shell Amat and assembled Pmat generate a problem if >> Galerkin >> multigrid is requested as a preconditioner (I just test on 1 CPU): >> >> [0]PETSC ERROR: MatPtAP requires A, shell, to be compatible with P, >> seqaij (Misses composed function MatPtAP_shell_seqaij_C) >> [0]PETSC ERROR: #1 MatPtAP() >> [0]PETSC ERROR: #2 MatGalerkin() >> [0]PETSC ERROR: #3 PCSetUp_MG() >> [0]PETSC ERROR: #4 PCSetUp() >> [0]PETSC ERROR: #5 KSPSetUp() >> [0]PETSC ERROR: #6 KSPSolve() >> [0]PETSC ERROR: #7 SNESSolve_NEWTONLS() >> [0]PETSC ERROR: #8 SNESSolve() >> >> It seems like PETSc tries to apply Galerkin coarsening to the >> shell Amat >> matrix instead of the assembled Pmat. Why? >> >> Pmat is internally generated by SNES using a DMDA attached to >> SNES, so >> it has everything to define Galerkin coarsening. And it actually >> works >> if I just get rid of the shell Amat. >> >> I can get around this problem by wrapping a PC object in a shell >> matrix >> and pass it as Pmat. This is a rather ugly approach, so I wonder >> how to >> resolve the situation in a better way, if it is possible. >> >> >> Hi Anton, >> >> You are correct that there is no specific test, so I can believe that >> a bug might be lurking here. >> I believe the way it is supposed to work is that you set the type of >> Galerkin coarsening that you >> want >> >> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCMGSetGalerkin.html >> >> So I am thinking you want 'pmat' only, and you would be in charge of >> making the coarse MFFD operator >> and telling PCMG about it. I could imagine that you might want us to >> automatically do that, but we would >> need some way to know that it is MFFD, and with the scheme above we >> do not have that. >> >> What is the advantage of switching representations during the Newton >> iteration? >> >> ? Thanks, >> >> ? ? ?Matt >> >> Thanks. >> Anton >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which >> their experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Jun 14 08:02:15 2021 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 14 Jun 2021 09:02:15 -0400 Subject: [petsc-users] Change Amat in FormJacobian In-Reply-To: References: <1e340ba2-32b7-f397-282b-ef0c8289c62d@uni-mainz.de> Message-ID: On Mon, Jun 14, 2021 at 8:45 AM Anton Popov wrote: > Hi Barry & Matt, > > thanks for your quick response. These options were exactly what I needed > and expected: > > -pc_mg_galerkin pmat > -pc_use_amat false > > I just assumed that it?s a default behavior of the PC object. > > So to clarify my case, I don't use nonlinear multigrid. Galerkin is > expected to deal with Pmat only, and it's enough if Amat implements a > matrix-vector product for the Krylov accelerator. > > Matt, the reason for switching Amat during the iteration is a quite common > Picard-Newton combination. Jacobian matrix gives accurate updates close to > the solution, but is rather unstable far form the solution. Picard matrix > (approximate Jacobian) is quite the opposite ? it?s kind of stable, but > slow. So the idea is to begin the iteration with Picard matrix, and switch > to the Jacobian later. > > If the assembled matrices are used, then the standard SNES interface is > just perfect. I can decide how to fill the matrices. But I don?t bother > with Jacobian assembly and want to use a built-in MFFD approximation > instead. I did quite a few tests previously and figured out that MFFD is > practically the same as closed-from matrix-free Jacobian for the later > stages of the iteration. The Picard matrix still does a good job as a > preconditioner. But it is important to start the iteration with Picard and > only change to MFFD later. > > Is my workaround with the shell matrix acceptable, or there is a better > solution? > It is fine. However, I wonder if you have time to try out a slightly different solution. I experimented with this, and it worked as well for me as switching between Picard and Newton. I made a SNESCOMPOSITE with Picard and Newton as the two components. I made Newton take a single step, and then adjusted the Picard steps a little.The additiveoptimal variant always converged for me, and took the same or less time. I also used left preconditioning of Picard by Newton, but that is more involved, and for my simple problem was not a lot better. The advantage I see is that both solves are not plain vanilla, and you do not have any switching logic. Thanks, Matt > Thanks, > Anton > On 13.06.21 20:52, Barry Smith wrote: > > > Anton, > > -pc_mg_galerkin pmat > > Though it seems simple, there is some subtly in swapping out matrices > with SNES. > > When using multigrid with SNES there are at least five distinct uses of > the Jacobian operator. > > - Perform matrix-vector product in line search to check Wolf > sufficient decrease convergence criteria > - Perform the matrix-vector product for the Krylov accelerator of > the system > - Perform smoothing on the finest level of MG > - Perform the matrix-vector product needed on the finest level of > MG to compute the residual that will be restricted to the coarser level of > multigrid > - When using Galerkin products to compute the coarser grid operator > performing the Galerkin matrix triple product > > > when one swaps out the mat, which of these do they wish to change? The > first two seem to naturally go together as do the last three. In your case > I presume you want to swap for the first two, but always use pmat for the > last three? To achieve this you will also need -pc_use_amat false > > If you run with -info and -snes_view it will print out some of the > information about which operator it is using for each case, but probably > not all of them. > > Note: if the pmat is actually an accurate computation of the Jacobian then > it is likely best not to ever use a matrix-free product. It is only when > pmat is approximated in some specific way that using the matrix-free > product would be useful to insure the "Newton" method actually computes a > Newton step. > > Barry > > > > On Jun 13, 2021, at 11:21 AM, Matthew Knepley wrote: > > On Sun, Jun 13, 2021 at 10:55 AM Anton Popov wrote: > >> Hi, >> >> I want a simple(?) thing. I want to decide and be able to assign the >> Jacobian matrix (Amat) on the fly within the FormJacobian function (i.e. >> during Newton iteration) to one of the following options: >> >> 1) built-in MFFD approximation >> 2) assembled preconditioner matrix (Pmat) >> >> I have not found this scenario demonstrated in any example, therefore >> I?m asking how to do that. >> >> Currently I do the following: >> >> 1) setup Amat as a shell matrix with a MATOP_MULT operation that simply >> retrieves a matrix object form its context and calls MatMult on it. >> >> 2) if I need MFFD, I put a matrix generated with MatCreateSNESMF in the >> Amat context (of course I also call MatMFFDComputeJacobian before that). >> >> 3) if I need Pmat, I simply put Pmat in the Amat context. >> >> 4) call MatAssemblyBegin/End on Amat >> >> So far so good. >> >> However, shell Amat and assembled Pmat generate a problem if Galerkin >> multigrid is requested as a preconditioner (I just test on 1 CPU): >> >> [0]PETSC ERROR: MatPtAP requires A, shell, to be compatible with P, >> seqaij (Misses composed function MatPtAP_shell_seqaij_C) >> [0]PETSC ERROR: #1 MatPtAP() >> [0]PETSC ERROR: #2 MatGalerkin() >> [0]PETSC ERROR: #3 PCSetUp_MG() >> [0]PETSC ERROR: #4 PCSetUp() >> [0]PETSC ERROR: #5 KSPSetUp() >> [0]PETSC ERROR: #6 KSPSolve() >> [0]PETSC ERROR: #7 SNESSolve_NEWTONLS() >> [0]PETSC ERROR: #8 SNESSolve() >> >> It seems like PETSc tries to apply Galerkin coarsening to the shell Amat >> matrix instead of the assembled Pmat. Why? >> >> Pmat is internally generated by SNES using a DMDA attached to SNES, so >> it has everything to define Galerkin coarsening. And it actually works >> if I just get rid of the shell Amat. >> >> I can get around this problem by wrapping a PC object in a shell matrix >> and pass it as Pmat. This is a rather ugly approach, so I wonder how to >> resolve the situation in a better way, if it is possible. >> > > Hi Anton, > > You are correct that there is no specific test, so I can believe that a > bug might be lurking here. > I believe the way it is supposed to work is that you set the type of > Galerkin coarsening that you > want > > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCMGSetGalerkin.html > > So I am thinking you want 'pmat' only, and you would be in charge of > making the coarse MFFD operator > and telling PCMG about it. I could imagine that you might want us to > automatically do that, but we would > need some way to know that it is MFFD, and with the scheme above we do not > have that. > > What is the advantage of switching representations during the Newton > iteration? > > Thanks, > > Matt > > >> Thanks. >> Anton >> > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Mon Jun 14 08:04:39 2021 From: dave.mayhem23 at gmail.com (Dave May) Date: Mon, 14 Jun 2021 15:04:39 +0200 Subject: [petsc-users] Change Amat in FormJacobian In-Reply-To: References: <1e340ba2-32b7-f397-282b-ef0c8289c62d@uni-mainz.de> Message-ID: Hi Anton, On Mon, 14 Jun 2021 at 14:47, Anton Popov wrote: > Hi Barry & Matt, > > thanks for your quick response. These options were exactly what I needed > and expected: > > -pc_mg_galerkin pmat > -pc_use_amat false > > I just assumed that it?s a default behavior of the PC object. > > So to clarify my case, I don't use nonlinear multigrid. Galerkin is > expected to deal with Pmat only, and it's enough if Amat implements a > matrix-vector product for the Krylov accelerator. > > Matt, the reason for switching Amat during the iteration is a quite common > Picard-Newton combination. Jacobian matrix gives accurate updates close to > the solution, but is rather unstable far form the solution. Picard matrix > (approximate Jacobian) is quite the opposite ? it?s kind of stable, but > slow. So the idea is to begin the iteration with Picard matrix, and switch > to the Jacobian later. > > If the assembled matrices are used, then the standard SNES interface is > just perfect. I can decide how to fill the matrices. But I don?t bother > with Jacobian assembly and want to use a built-in MFFD approximation > instead. I did quite a few tests previously and figured out that MFFD is > practically the same as closed-from matrix-free Jacobian for the later > stages of the iteration. The Picard matrix still does a good job as a > preconditioner. But it is important to start the iteration with Picard and > only change to MFFD later. > > Is my workaround with the shell matrix acceptable, or there is a better > solution? > Given what you write, it sounds like you already have a good heuristic for when to switch from Picard to Newton, Hence I think the simplest option is just to use two seperate SNES objects - one for performing Picard and one for Newton. The stopping condition for the Picard object would encode your heuristic to abort earlier when the solution was deemed sufficiently accurate. Thanks, Dave > > Thanks, > Anton > On 13.06.21 20:52, Barry Smith wrote: > > > Anton, > > -pc_mg_galerkin pmat > > Though it seems simple, there is some subtly in swapping out matrices > with SNES. > > When using multigrid with SNES there are at least five distinct uses of > the Jacobian operator. > > - Perform matrix-vector product in line search to check Wolf > sufficient decrease convergence criteria > - Perform the matrix-vector product for the Krylov accelerator of > the system > - Perform smoothing on the finest level of MG > - Perform the matrix-vector product needed on the finest level of > MG to compute the residual that will be restricted to the coarser level of > multigrid > - When using Galerkin products to compute the coarser grid operator > performing the Galerkin matrix triple product > > > when one swaps out the mat, which of these do they wish to change? The > first two seem to naturally go together as do the last three. In your case > I presume you want to swap for the first two, but always use pmat for the > last three? To achieve this you will also need -pc_use_amat false > > If you run with -info and -snes_view it will print out some of the > information about which operator it is using for each case, but probably > not all of them. > > Note: if the pmat is actually an accurate computation of the Jacobian then > it is likely best not to ever use a matrix-free product. It is only when > pmat is approximated in some specific way that using the matrix-free > product would be useful to insure the "Newton" method actually computes a > Newton step. > > Barry > > > > On Jun 13, 2021, at 11:21 AM, Matthew Knepley wrote: > > On Sun, Jun 13, 2021 at 10:55 AM Anton Popov wrote: > >> Hi, >> >> I want a simple(?) thing. I want to decide and be able to assign the >> Jacobian matrix (Amat) on the fly within the FormJacobian function (i.e. >> during Newton iteration) to one of the following options: >> >> 1) built-in MFFD approximation >> 2) assembled preconditioner matrix (Pmat) >> >> I have not found this scenario demonstrated in any example, therefore >> I?m asking how to do that. >> >> Currently I do the following: >> >> 1) setup Amat as a shell matrix with a MATOP_MULT operation that simply >> retrieves a matrix object form its context and calls MatMult on it. >> >> 2) if I need MFFD, I put a matrix generated with MatCreateSNESMF in the >> Amat context (of course I also call MatMFFDComputeJacobian before that). >> >> 3) if I need Pmat, I simply put Pmat in the Amat context. >> >> 4) call MatAssemblyBegin/End on Amat >> >> So far so good. >> >> However, shell Amat and assembled Pmat generate a problem if Galerkin >> multigrid is requested as a preconditioner (I just test on 1 CPU): >> >> [0]PETSC ERROR: MatPtAP requires A, shell, to be compatible with P, >> seqaij (Misses composed function MatPtAP_shell_seqaij_C) >> [0]PETSC ERROR: #1 MatPtAP() >> [0]PETSC ERROR: #2 MatGalerkin() >> [0]PETSC ERROR: #3 PCSetUp_MG() >> [0]PETSC ERROR: #4 PCSetUp() >> [0]PETSC ERROR: #5 KSPSetUp() >> [0]PETSC ERROR: #6 KSPSolve() >> [0]PETSC ERROR: #7 SNESSolve_NEWTONLS() >> [0]PETSC ERROR: #8 SNESSolve() >> >> It seems like PETSc tries to apply Galerkin coarsening to the shell Amat >> matrix instead of the assembled Pmat. Why? >> >> Pmat is internally generated by SNES using a DMDA attached to SNES, so >> it has everything to define Galerkin coarsening. And it actually works >> if I just get rid of the shell Amat. >> >> I can get around this problem by wrapping a PC object in a shell matrix >> and pass it as Pmat. This is a rather ugly approach, so I wonder how to >> resolve the situation in a better way, if it is possible. >> > > Hi Anton, > > You are correct that there is no specific test, so I can believe that a > bug might be lurking here. > I believe the way it is supposed to work is that you set the type of > Galerkin coarsening that you > want > > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCMGSetGalerkin.html > > So I am thinking you want 'pmat' only, and you would be in charge of > making the coarse MFFD operator > and telling PCMG about it. I could imagine that you might want us to > automatically do that, but we would > need some way to know that it is MFFD, and with the scheme above we do not > have that. > > What is the advantage of switching representations during the Newton > iteration? > > Thanks, > > Matt > > >> Thanks. >> Anton >> > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From popov at uni-mainz.de Mon Jun 14 10:18:22 2021 From: popov at uni-mainz.de (Anton Popov) Date: Mon, 14 Jun 2021 17:18:22 +0200 Subject: [petsc-users] Change Amat in FormJacobian In-Reply-To: References: <1e340ba2-32b7-f397-282b-ef0c8289c62d@uni-mainz.de> Message-ID: <4dac6c96-1412-19bf-8436-492d1347fee7@uni-mainz.de> On 14.06.21 15:02, Matthew Knepley wrote: > On Mon, Jun 14, 2021 at 8:45 AM Anton Popov > wrote: > > Hi Barry & Matt, > > thanks for your quick response. These options were exactly what I > needed and expected: > > -pc_mg_galerkin pmat > -pc_use_amat false > > I just assumed that it?s a default behavior of the PC object. > > So to clarify my case, I don't use nonlinear multigrid. Galerkin > is expected to deal with Pmat only, and it's enough if Amat > implements a matrix-vector product for the Krylov accelerator. > > Matt, the reason for switching Amat during the iteration is a > quite common Picard-Newton combination. Jacobian matrix gives > accurate updates close to the solution, but is rather unstable far > form the solution. Picard matrix (approximate Jacobian) is quite > the opposite ? it?s kind of stable, but slow. So the idea is to > begin the iteration with Picard matrix, and switch to the Jacobian > later. > > If the assembled matrices are used, then the standard SNES > interface is just perfect. I can decide how to fill the matrices. > But I don?t bother with? Jacobian assembly and want to use a > built-in MFFD approximation instead. I did quite a few tests > previously and figured out that MFFD is practically the same as > closed-from matrix-free Jacobian for the later stages of the > iteration. The Picard matrix still does a good job as a > preconditioner. But it is important to start the iteration with > Picard and only change to MFFD later. > > Is my workaround with the shell matrix acceptable, or there is a > better solution? > > It is fine. However, I wonder if you have time to try out a slightly > different solution. I experimented with this, and it worked as well > for me as switching between > Picard and Newton. I made a SNESCOMPOSITE with Picard and Newton as > the two components. I made Newton take a single step, and then > adjusted the > Picard steps a little.The additiveoptimal?variant always converged for > me, and took the same or less time. I also used left preconditioning > of Picard by Newton, > but that is more involved, and for my simple problem was not a lot > better. The advantage I see is that both solves are not plain vanilla, > and you do not have any > switching logic. > hmmm, this sounds really interesting. I will definitely look into this possibility. Thanks Matt. Thanks, Anton > ? Thanks, > > ? ? Matt > > Thanks, > Anton > > On 13.06.21 20:52, Barry Smith wrote: >> >> ? Anton, >> >> ??-pc_mg_galerkin pmat >> >> ? Though it seems simple, there is some subtly in swapping out >> matrices with SNES. >> >> ? When using multigrid with SNES there are at least five distinct >> uses of the Jacobian operator. >> >> * Perform matrix-vector product in line search to check >> Wolf sufficient decrease convergence criteria >> * Perform the matrix-vector product for the Krylov >> accelerator of the system >> * Perform smoothing on the finest level of MG >> * Perform the matrix-vector product needed on the finest >> level of MG to compute the residual that will be >> restricted to the coarser level of multigrid >> * When using Galerkin products to compute the coarser grid >> operator performing the Galerkin matrix triple product >> >> >> when one swaps out the mat, which of these do they wish to >> change? The first two seem to naturally go together as do the >> last three. In your case I presume you want to swap for the first >> two, but always use pmat for the last three? To achieve this you >> will also need?-pc_use_amat ?false >> >> If you run with -info and -snes_view it will print out some of >> the information about which operator it is using for each case, >> but probably not all of them. >> >> Note: if the pmat is actually an accurate computation of the >> Jacobian then it is likely best not to ever use a matrix-free >> product. It is only when pmat is approximated in some specific >> way that using the matrix-free product would be useful to insure >> the "Newton" method actually computes a Newton step. >> >> Barry >> >> >> >>> On Jun 13, 2021, at 11:21 AM, Matthew Knepley >> > wrote: >>> >>> On Sun, Jun 13, 2021 at 10:55 AM Anton Popov >> > wrote: >>> >>> Hi, >>> >>> I want a simple(?) thing. I want to decide and be able to >>> assign the >>> Jacobian matrix (Amat) on the fly within the FormJacobian >>> function (i.e. >>> during Newton iteration) to one of the following options: >>> >>> 1) built-in MFFD approximation >>> 2) assembled preconditioner matrix (Pmat) >>> >>> I have not found this scenario demonstrated in any example, >>> therefore >>> I?m asking how to do that. >>> >>> Currently I do the following: >>> >>> 1) setup Amat as a shell matrix with a MATOP_MULT operation >>> that simply >>> retrieves a matrix object form its context and calls MatMult >>> on it. >>> >>> 2) if I need MFFD, I put a matrix generated with >>> MatCreateSNESMF in the >>> Amat context (of course I also call MatMFFDComputeJacobian >>> before that). >>> >>> 3) if I need Pmat, I simply put Pmat in the Amat context. >>> >>> 4) call MatAssemblyBegin/End on Amat >>> >>> So far so good. >>> >>> However, shell Amat and assembled Pmat generate a problem if >>> Galerkin >>> multigrid is requested as a preconditioner (I just test on 1 >>> CPU): >>> >>> [0]PETSC ERROR: MatPtAP requires A, shell, to be compatible >>> with P, >>> seqaij (Misses composed function MatPtAP_shell_seqaij_C) >>> [0]PETSC ERROR: #1 MatPtAP() >>> [0]PETSC ERROR: #2 MatGalerkin() >>> [0]PETSC ERROR: #3 PCSetUp_MG() >>> [0]PETSC ERROR: #4 PCSetUp() >>> [0]PETSC ERROR: #5 KSPSetUp() >>> [0]PETSC ERROR: #6 KSPSolve() >>> [0]PETSC ERROR: #7 SNESSolve_NEWTONLS() >>> [0]PETSC ERROR: #8 SNESSolve() >>> >>> It seems like PETSc tries to apply Galerkin coarsening to >>> the shell Amat >>> matrix instead of the assembled Pmat. Why? >>> >>> Pmat is internally generated by SNES using a DMDA attached >>> to SNES, so >>> it has everything to define Galerkin coarsening. And it >>> actually works >>> if I just get rid of the shell Amat. >>> >>> I can get around this problem by wrapping a PC object in a >>> shell matrix >>> and pass it as Pmat. This is a rather ugly approach, so I >>> wonder how to >>> resolve the situation in a better way, if it is possible. >>> >>> >>> Hi Anton, >>> >>> You are correct that there is no specific test, so I can believe >>> that a bug might be lurking here. >>> I believe the way it is supposed to work is that you set the >>> type of Galerkin coarsening that you >>> want >>> >>> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCMGSetGalerkin.html >>> >>> So I am thinking you want 'pmat' only, and you would be in >>> charge of making the coarse MFFD operator >>> and telling PCMG about it. I could imagine that you might want >>> us to automatically do that, but we would >>> need some way to know that it is MFFD, and with the scheme above >>> we do not have that. >>> >>> What is the advantage of switching representations during the >>> Newton iteration? >>> >>> ? Thanks, >>> >>> ? ? ?Matt >>> >>> Thanks. >>> Anton >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to >>> which their experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From popov at uni-mainz.de Mon Jun 14 10:27:42 2021 From: popov at uni-mainz.de (Anton Popov) Date: Mon, 14 Jun 2021 17:27:42 +0200 Subject: [petsc-users] Change Amat in FormJacobian In-Reply-To: References: <1e340ba2-32b7-f397-282b-ef0c8289c62d@uni-mainz.de> Message-ID: On 14.06.21 15:04, Dave May wrote: > > Hi Anton, Hi Dave, > > On Mon, 14 Jun 2021 at 14:47, Anton Popov > wrote: > > Hi Barry & Matt, > > thanks for your quick response. These options were exactly what I > needed and expected: > > -pc_mg_galerkin pmat > -pc_use_amat false > > I just assumed that it?s a default behavior of the PC object. > > So to clarify my case, I don't use nonlinear multigrid. Galerkin > is expected to deal with Pmat only, and it's enough if Amat > implements a matrix-vector product for the Krylov accelerator. > > Matt, the reason for switching Amat during the iteration is a > quite common Picard-Newton combination. Jacobian matrix gives > accurate updates close to the solution, but is rather unstable far > form the solution. Picard matrix (approximate Jacobian) is quite > the opposite ? it?s kind of stable, but slow. So the idea is to > begin the iteration with Picard matrix, and switch to the Jacobian > later. > > If the assembled matrices are used, then the standard SNES > interface is just perfect. I can decide how to fill the matrices. > But I don?t bother with? Jacobian assembly and want to use a > built-in MFFD approximation instead. I did quite a few tests > previously and figured out that MFFD is practically the same as > closed-from matrix-free Jacobian for the later stages of the > iteration. The Picard matrix still does a good job as a > preconditioner. But it is important to start the iteration with > Picard and only change to MFFD later. > > Is my workaround with the shell matrix acceptable, or there is a > better solution? > > > Given what you write, it sounds like you already have a good heuristic > for when to switch from Picard to Newton, > Hence I think the simplest option is just to use two seperate SNES > objects - one for performing Picard and one for Newton. Yes, I considered this option initially. Sometimes it is necessary to switch back and forth between the methods, so it becomes a bit messy to organize this in the code. But maybe if Newton fails after Picard, I should just reduce the time step and restart, instead of switching back to Picard. Thanks, Dave. Thanks, Anton > The stopping condition for the Picard object would encode your > heuristic to abort earlier when the solution was deemed sufficiently > accurate. > > Thanks, > Dave > > > Thanks, > Anton > > On 13.06.21 20:52, Barry Smith wrote: >> >> ? Anton, >> >> ??-pc_mg_galerkin pmat >> >> ? Though it seems simple, there is some subtly in swapping out >> matrices with SNES. >> >> ? When using multigrid with SNES there are at least five distinct >> uses of the Jacobian operator. >> >> * Perform matrix-vector product in line search to check >> Wolf sufficient decrease convergence criteria >> * Perform the matrix-vector product for the Krylov >> accelerator of the system >> * Perform smoothing on the finest level of MG >> * Perform the matrix-vector product needed on the finest >> level of MG to compute the residual that will be >> restricted to the coarser level of multigrid >> * When using Galerkin products to compute the coarser grid >> operator performing the Galerkin matrix triple product >> >> >> when one swaps out the mat, which of these do they wish to >> change? The first two seem to naturally go together as do the >> last three. In your case I presume you want to swap for the first >> two, but always use pmat for the last three? To achieve this you >> will also need?-pc_use_amat ?false >> >> If you run with -info and -snes_view it will print out some of >> the information about which operator it is using for each case, >> but probably not all of them. >> >> Note: if the pmat is actually an accurate computation of the >> Jacobian then it is likely best not to ever use a matrix-free >> product. It is only when pmat is approximated in some specific >> way that using the matrix-free product would be useful to insure >> the "Newton" method actually computes a Newton step. >> >> Barry >> >> >> >>> On Jun 13, 2021, at 11:21 AM, Matthew Knepley >> > wrote: >>> >>> On Sun, Jun 13, 2021 at 10:55 AM Anton Popov >> > wrote: >>> >>> Hi, >>> >>> I want a simple(?) thing. I want to decide and be able to >>> assign the >>> Jacobian matrix (Amat) on the fly within the FormJacobian >>> function (i.e. >>> during Newton iteration) to one of the following options: >>> >>> 1) built-in MFFD approximation >>> 2) assembled preconditioner matrix (Pmat) >>> >>> I have not found this scenario demonstrated in any example, >>> therefore >>> I?m asking how to do that. >>> >>> Currently I do the following: >>> >>> 1) setup Amat as a shell matrix with a MATOP_MULT operation >>> that simply >>> retrieves a matrix object form its context and calls MatMult >>> on it. >>> >>> 2) if I need MFFD, I put a matrix generated with >>> MatCreateSNESMF in the >>> Amat context (of course I also call MatMFFDComputeJacobian >>> before that). >>> >>> 3) if I need Pmat, I simply put Pmat in the Amat context. >>> >>> 4) call MatAssemblyBegin/End on Amat >>> >>> So far so good. >>> >>> However, shell Amat and assembled Pmat generate a problem if >>> Galerkin >>> multigrid is requested as a preconditioner (I just test on 1 >>> CPU): >>> >>> [0]PETSC ERROR: MatPtAP requires A, shell, to be compatible >>> with P, >>> seqaij (Misses composed function MatPtAP_shell_seqaij_C) >>> [0]PETSC ERROR: #1 MatPtAP() >>> [0]PETSC ERROR: #2 MatGalerkin() >>> [0]PETSC ERROR: #3 PCSetUp_MG() >>> [0]PETSC ERROR: #4 PCSetUp() >>> [0]PETSC ERROR: #5 KSPSetUp() >>> [0]PETSC ERROR: #6 KSPSolve() >>> [0]PETSC ERROR: #7 SNESSolve_NEWTONLS() >>> [0]PETSC ERROR: #8 SNESSolve() >>> >>> It seems like PETSc tries to apply Galerkin coarsening to >>> the shell Amat >>> matrix instead of the assembled Pmat. Why? >>> >>> Pmat is internally generated by SNES using a DMDA attached >>> to SNES, so >>> it has everything to define Galerkin coarsening. And it >>> actually works >>> if I just get rid of the shell Amat. >>> >>> I can get around this problem by wrapping a PC object in a >>> shell matrix >>> and pass it as Pmat. This is a rather ugly approach, so I >>> wonder how to >>> resolve the situation in a better way, if it is possible. >>> >>> >>> Hi Anton, >>> >>> You are correct that there is no specific test, so I can believe >>> that a bug might be lurking here. >>> I believe the way it is supposed to work is that you set the >>> type of Galerkin coarsening that you >>> want >>> >>> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCMGSetGalerkin.html >>> >>> So I am thinking you want 'pmat' only, and you would be in >>> charge of making the coarse MFFD operator >>> and telling PCMG about it. I could imagine that you might want >>> us to automatically do that, but we would >>> need some way to know that it is MFFD, and with the scheme above >>> we do not have that. >>> >>> What is the advantage of switching representations during the >>> Newton iteration? >>> >>> ? Thanks, >>> >>> ? ? ?Matt >>> >>> Thanks. >>> Anton >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to >>> which their experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Mon Jun 14 10:37:51 2021 From: dave.mayhem23 at gmail.com (Dave May) Date: Mon, 14 Jun 2021 17:37:51 +0200 Subject: [petsc-users] Change Amat in FormJacobian In-Reply-To: References: <1e340ba2-32b7-f397-282b-ef0c8289c62d@uni-mainz.de> Message-ID: On Mon 14. Jun 2021 at 17:27, Anton Popov wrote: > > On 14.06.21 15:04, Dave May wrote: > > > Hi Anton, > > Hi Dave, > > > On Mon, 14 Jun 2021 at 14:47, Anton Popov wrote: > >> Hi Barry & Matt, >> >> thanks for your quick response. These options were exactly what I needed >> and expected: >> >> -pc_mg_galerkin pmat >> -pc_use_amat false >> >> I just assumed that it?s a default behavior of the PC object. >> >> So to clarify my case, I don't use nonlinear multigrid. Galerkin is >> expected to deal with Pmat only, and it's enough if Amat implements a >> matrix-vector product for the Krylov accelerator. >> >> Matt, the reason for switching Amat during the iteration is a quite >> common Picard-Newton combination. Jacobian matrix gives accurate updates >> close to the solution, but is rather unstable far form the solution. Picard >> matrix (approximate Jacobian) is quite the opposite ? it?s kind of stable, >> but slow. So the idea is to begin the iteration with Picard matrix, and >> switch to the Jacobian later. >> >> If the assembled matrices are used, then the standard SNES interface is >> just perfect. I can decide how to fill the matrices. But I don?t bother >> with Jacobian assembly and want to use a built-in MFFD approximation >> instead. I did quite a few tests previously and figured out that MFFD is >> practically the same as closed-from matrix-free Jacobian for the later >> stages of the iteration. The Picard matrix still does a good job as a >> preconditioner. But it is important to start the iteration with Picard and >> only change to MFFD later. >> >> Is my workaround with the shell matrix acceptable, or there is a better >> solution? >> > > Given what you write, it sounds like you already have a good heuristic for > when to switch from Picard to Newton, > Hence I think the simplest option is just to use two seperate SNES objects > - one for performing Picard and one for Newton. > > > Yes, I considered this option initially. Sometimes it is necessary to > switch back and forth between the methods, so it becomes a bit messy to > organize this in the code. > > But maybe if Newton fails after Picard, I should just reduce the time step > and restart, instead of switching back to Picard. Thanks, Dave. > > Oh yeah, for a nasty multiphysics problem a single SNES is likely not the end of the story! Definitely there is almost certainly an entire other outer loop of several nonlinear solver strategies slapped together in some problem specific manner in an effort to get the monolithic nonlinear residual down. Aborting the time step and dropping dt is often the thing you fall back to when all of those fail. Thanks, Dave Thanks, > > Anton > > > The stopping condition for the Picard object would encode your heuristic > to abort earlier when the solution was deemed sufficiently accurate. > > Thanks, > Dave > > >> >> Thanks, >> Anton >> On 13.06.21 20:52, Barry Smith wrote: >> >> >> Anton, >> >> -pc_mg_galerkin pmat >> >> Though it seems simple, there is some subtly in swapping out matrices >> with SNES. >> >> When using multigrid with SNES there are at least five distinct uses of >> the Jacobian operator. >> >> - Perform matrix-vector product in line search to check Wolf >> sufficient decrease convergence criteria >> - Perform the matrix-vector product for the Krylov accelerator of >> the system >> - Perform smoothing on the finest level of MG >> - Perform the matrix-vector product needed on the finest level of >> MG to compute the residual that will be restricted to the coarser level of >> multigrid >> - When using Galerkin products to compute the coarser grid >> operator performing the Galerkin matrix triple product >> >> >> when one swaps out the mat, which of these do they wish to change? The >> first two seem to naturally go together as do the last three. In your case >> I presume you want to swap for the first two, but always use pmat for the >> last three? To achieve this you will also need -pc_use_amat false >> >> If you run with -info and -snes_view it will print out some of the >> information about which operator it is using for each case, but probably >> not all of them. >> >> Note: if the pmat is actually an accurate computation of the Jacobian >> then it is likely best not to ever use a matrix-free product. It is only >> when pmat is approximated in some specific way that using the matrix-free >> product would be useful to insure the "Newton" method actually computes a >> Newton step. >> >> Barry >> >> >> >> On Jun 13, 2021, at 11:21 AM, Matthew Knepley wrote: >> >> On Sun, Jun 13, 2021 at 10:55 AM Anton Popov wrote: >> >>> Hi, >>> >>> I want a simple(?) thing. I want to decide and be able to assign the >>> Jacobian matrix (Amat) on the fly within the FormJacobian function (i.e. >>> during Newton iteration) to one of the following options: >>> >>> 1) built-in MFFD approximation >>> 2) assembled preconditioner matrix (Pmat) >>> >>> I have not found this scenario demonstrated in any example, therefore >>> I?m asking how to do that. >>> >>> Currently I do the following: >>> >>> 1) setup Amat as a shell matrix with a MATOP_MULT operation that simply >>> retrieves a matrix object form its context and calls MatMult on it. >>> >>> 2) if I need MFFD, I put a matrix generated with MatCreateSNESMF in the >>> Amat context (of course I also call MatMFFDComputeJacobian before that). >>> >>> 3) if I need Pmat, I simply put Pmat in the Amat context. >>> >>> 4) call MatAssemblyBegin/End on Amat >>> >>> So far so good. >>> >>> However, shell Amat and assembled Pmat generate a problem if Galerkin >>> multigrid is requested as a preconditioner (I just test on 1 CPU): >>> >>> [0]PETSC ERROR: MatPtAP requires A, shell, to be compatible with P, >>> seqaij (Misses composed function MatPtAP_shell_seqaij_C) >>> [0]PETSC ERROR: #1 MatPtAP() >>> [0]PETSC ERROR: #2 MatGalerkin() >>> [0]PETSC ERROR: #3 PCSetUp_MG() >>> [0]PETSC ERROR: #4 PCSetUp() >>> [0]PETSC ERROR: #5 KSPSetUp() >>> [0]PETSC ERROR: #6 KSPSolve() >>> [0]PETSC ERROR: #7 SNESSolve_NEWTONLS() >>> [0]PETSC ERROR: #8 SNESSolve() >>> >>> It seems like PETSc tries to apply Galerkin coarsening to the shell Amat >>> matrix instead of the assembled Pmat. Why? >>> >>> Pmat is internally generated by SNES using a DMDA attached to SNES, so >>> it has everything to define Galerkin coarsening. And it actually works >>> if I just get rid of the shell Amat. >>> >>> I can get around this problem by wrapping a PC object in a shell matrix >>> and pass it as Pmat. This is a rather ugly approach, so I wonder how to >>> resolve the situation in a better way, if it is possible. >>> >> >> Hi Anton, >> >> You are correct that there is no specific test, so I can believe that a >> bug might be lurking here. >> I believe the way it is supposed to work is that you set the type of >> Galerkin coarsening that you >> want >> >> >> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCMGSetGalerkin.html >> >> So I am thinking you want 'pmat' only, and you would be in charge of >> making the coarse MFFD operator >> and telling PCMG about it. I could imagine that you might want us to >> automatically do that, but we would >> need some way to know that it is MFFD, and with the scheme above we do >> not have that. >> >> What is the advantage of switching representations during the Newton >> iteration? >> >> Thanks, >> >> Matt >> >> >>> Thanks. >>> Anton >>> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Mon Jun 14 11:38:06 2021 From: mfadams at lbl.gov (Mark Adams) Date: Mon, 14 Jun 2021 12:38:06 -0400 Subject: [petsc-users] reproducibility Message-ID: I am working on reproducibility for SC and I have two machines that are using the same branch that is up-to-date as far as I can tell but they seem to have different SHA1: out10_1_1_Cuda_CPU.txt:Using Petsc Development GIT revision: v3.15.0-792-gacb395f GIT Date: 2021-06-13 22:56:54 +0000 out11_1_1_Cuda_CPU.txt:Using Petsc Development GIT revision: v3.15.0-794-g4bf14ff56a GIT Date: 2021-06-14 07:47:59 -0400 Does this look right? Thanks, Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Jun 14 11:55:45 2021 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 14 Jun 2021 12:55:45 -0400 Subject: [petsc-users] reproducibility In-Reply-To: References: Message-ID: On Mon, Jun 14, 2021 at 12:38 PM Mark Adams wrote: > I am working on reproducibility for SC and I have two machines that are > using the same branch that is up-to-date as far as I can tell but they seem > to have different SHA1: > > out10_1_1_Cuda_CPU.txt:Using Petsc Development GIT revision: > v3.15.0-792-gacb395f GIT Date: 2021-06-13 22:56:54 +0000 > > > out11_1_1_Cuda_CPU.txt:Using Petsc Development GIT revision: > v3.15.0-794-g4bf14ff56a GIT Date: 2021-06-14 07:47:59 -0400 > > Does this look right? > They definitely look like they are on different change sets. Can you do git log -1 Thanks, Matt > Thanks, > Mark > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Mon Jun 14 12:22:35 2021 From: mfadams at lbl.gov (Mark Adams) Date: Mon, 14 Jun 2021 13:22:35 -0400 Subject: [petsc-users] reproducibility In-Reply-To: References: Message-ID: Same SHA1 (?): 13:18 adams/cusparse-cpu-solve= /gpfs/alpine/csc314/scratch/adams/petsc/src/ts/utils/dmplexlandau/tutorials/tput$ git log -1 commit 4bf14ff56aaf20873dd75467a8e4c029914a4549 Author: Mark Adams Date: Sat May 15 12:22:00 2021 -0700 turn cpu solve back on commit 4bf14ff56aaf20873dd75467a8e4c029914a4549 (HEAD -> adams/cusparse-cpu-solve, origin/adams/cusparse-cpu-solve) Author: Mark Adams Date: Sat May 15 12:22:00 2021 -0700 turn cpu solve back on lines 1-5/5 (END) On Mon, Jun 14, 2021 at 12:55 PM Matthew Knepley wrote: > On Mon, Jun 14, 2021 at 12:38 PM Mark Adams wrote: > >> I am working on reproducibility for SC and I have two machines that are >> using the same branch that is up-to-date as far as I can tell but they seem >> to have different SHA1: >> >> out10_1_1_Cuda_CPU.txt:Using Petsc Development GIT revision: >> v3.15.0-792-gacb395f GIT Date: 2021-06-13 22:56:54 +0000 >> >> >> out11_1_1_Cuda_CPU.txt:Using Petsc Development GIT revision: >> v3.15.0-794-g4bf14ff56a GIT Date: 2021-06-14 07:47:59 -0400 >> >> Does this look right? >> > > They definitely look like they are on different change sets. Can you do > > git log -1 > > Thanks, > > Matt > > >> Thanks, >> Mark >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefano.zampini at gmail.com Mon Jun 14 12:45:28 2021 From: stefano.zampini at gmail.com (Stefano Zampini) Date: Mon, 14 Jun 2021 20:45:28 +0300 Subject: [petsc-users] reproducibility In-Reply-To: References: Message-ID: Mark I presume in your first message you report the SHA1 as listed by log_view. That string is populated at configure time, not at runtime. -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Mon Jun 14 12:51:38 2021 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 14 Jun 2021 12:51:38 -0500 Subject: [petsc-users] reproducibility In-Reply-To: References: Message-ID: <41d8f9d-1028-fcec-da78-19ec8af6429@mcs.anl.gov> The commits match v3.15.0-794-g4bf14ff56a So in the repo that gave v3.15.0-792-gacb395f in the log - configure was run with an earlier snapshot Best to rerun configure [and build] to have the correct hash string [of the commit] encoded in the library. Satish On Mon, 14 Jun 2021, Mark Adams wrote: > Same SHA1 (?): > > 13:18 adams/cusparse-cpu-solve= > /gpfs/alpine/csc314/scratch/adams/petsc/src/ts/utils/dmplexlandau/tutorials/tput$ > git log -1 > commit 4bf14ff56aaf20873dd75467a8e4c029914a4549 > Author: Mark Adams > Date: Sat May 15 12:22:00 2021 -0700 > > turn cpu solve back on > > > > commit 4bf14ff56aaf20873dd75467a8e4c029914a4549 (HEAD -> > adams/cusparse-cpu-solve, origin/adams/cusparse-cpu-solve) > Author: Mark Adams > Date: Sat May 15 12:22:00 2021 -0700 > > turn cpu solve back on > lines 1-5/5 (END) > > On Mon, Jun 14, 2021 at 12:55 PM Matthew Knepley wrote: > > > On Mon, Jun 14, 2021 at 12:38 PM Mark Adams wrote: > > > >> I am working on reproducibility for SC and I have two machines that are > >> using the same branch that is up-to-date as far as I can tell but they seem > >> to have different SHA1: > >> > >> out10_1_1_Cuda_CPU.txt:Using Petsc Development GIT revision: > >> v3.15.0-792-gacb395f GIT Date: 2021-06-13 22:56:54 +0000 > >> > >> > >> out11_1_1_Cuda_CPU.txt:Using Petsc Development GIT revision: > >> v3.15.0-794-g4bf14ff56a GIT Date: 2021-06-14 07:47:59 -0400 > >> > >> Does this look right? > >> > > > > They definitely look like they are on different change sets. Can you do > > > > git log -1 > > > > Thanks, > > > > Matt > > > > > >> Thanks, > >> Mark > >> > >> > > > > -- > > What most experimenters take for granted before they begin their > > experiments is infinitely more interesting than any results to which their > > experiments lead. > > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ > > > > > From mfadams at lbl.gov Mon Jun 14 13:01:23 2021 From: mfadams at lbl.gov (Mark Adams) Date: Mon, 14 Jun 2021 14:01:23 -0400 Subject: [petsc-users] reproducibility In-Reply-To: <41d8f9d-1028-fcec-da78-19ec8af6429@mcs.anl.gov> References: <41d8f9d-1028-fcec-da78-19ec8af6429@mcs.anl.gov> Message-ID: Ah, I see . I need to force configure. Thanks, Mark On Mon, Jun 14, 2021 at 1:51 PM Satish Balay wrote: > The commits match v3.15.0-794-g4bf14ff56a > > So in the repo that gave v3.15.0-792-gacb395f in the log - configure was > run with an earlier snapshot > > Best to rerun configure [and build] to have the correct hash string [of > the commit] encoded in the library. > > Satish > > On Mon, 14 Jun 2021, Mark Adams wrote: > > > Same SHA1 (?): > > > > 13:18 adams/cusparse-cpu-solve= > > > /gpfs/alpine/csc314/scratch/adams/petsc/src/ts/utils/dmplexlandau/tutorials/tput$ > > git log -1 > > commit 4bf14ff56aaf20873dd75467a8e4c029914a4549 > > Author: Mark Adams > > Date: Sat May 15 12:22:00 2021 -0700 > > > > turn cpu solve back on > > > > > > > > commit 4bf14ff56aaf20873dd75467a8e4c029914a4549 (HEAD -> > > adams/cusparse-cpu-solve, origin/adams/cusparse-cpu-solve) > > Author: Mark Adams > > Date: Sat May 15 12:22:00 2021 -0700 > > > > turn cpu solve back on > > lines 1-5/5 (END) > > > > On Mon, Jun 14, 2021 at 12:55 PM Matthew Knepley > wrote: > > > > > On Mon, Jun 14, 2021 at 12:38 PM Mark Adams wrote: > > > > > >> I am working on reproducibility for SC and I have two machines that > are > > >> using the same branch that is up-to-date as far as I can tell but > they seem > > >> to have different SHA1: > > >> > > >> out10_1_1_Cuda_CPU.txt:Using Petsc Development GIT revision: > > >> v3.15.0-792-gacb395f GIT Date: 2021-06-13 22:56:54 +0000 > > >> > > >> > > >> out11_1_1_Cuda_CPU.txt:Using Petsc Development GIT revision: > > >> v3.15.0-794-g4bf14ff56a GIT Date: 2021-06-14 07:47:59 -0400 > > >> > > >> Does this look right? > > >> > > > > > > They definitely look like they are on different change sets. Can you do > > > > > > git log -1 > > > > > > Thanks, > > > > > > Matt > > > > > > > > >> Thanks, > > >> Mark > > >> > > >> > > > > > > -- > > > What most experimenters take for granted before they begin their > > > experiments is infinitely more interesting than any results to which > their > > > experiments lead. > > > -- Norbert Wiener > > > > > > https://www.cse.buffalo.edu/~knepley/ > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Mon Jun 14 14:45:43 2021 From: bsmith at petsc.dev (Barry Smith) Date: Mon, 14 Jun 2021 14:45:43 -0500 Subject: [petsc-users] Copy stiffness matrix to jacobian matrix in form jacobian In-Reply-To: References: <72658765-A7DC-4C47-84F0-BD366EF22A4B@petsc.dev> Message-ID: It is best to cut and paste the entire error message, not just two lines. We are missing all context to see where the error is occurring and possible causes. > On Jun 14, 2021, at 6:20 AM, Kaushik Vijaykumar wrote: > > Thanks for the suggestion Barry, I did do that by setting it in multiple ways first by setting > > Formfunction(snes,Vec x,Vec F,void* ctx) > { > ierr = SNESGetJacobian(snes,&jac,NULL,NULL,NULL);CHKERRQ(ierr); > ierr = MatZeroEntries(jac);CHKERRQ(ierr); > > ..... > > > ierr = MatGetOwnershipRange(jac,&istart,&iend); > > for(i=istart;i { > val = 0.0; > ierr = MatSetValue(jac, i, i, val, ADD_VALUES);CHKERRQ(ierr); > } > ierr = MatAssemblyBegin(jac,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > ierr = MatAssemblyEnd(jac,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > } > I get the same error as before for a single processor run. > "[0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Object is in wrong state > [0]PETSC ERROR: Matrix is missing diagonal entry 0" > > > Thanks > Kaushik > > On Fri, Jun 11, 2021 at 11:33 AM Barry Smith > wrote: > >> When I execute this I get a Petsc error, object in wrong state missing diagonal entry. > > When do you get this error? During a TSSolve? While KSP is building a preconditioner. > > Some parts of PETSc require that you put entries (even if they are zero) on all diagonal entries in the Jacobian. So likely you need to just make sure that your populate jac routine puts/adds 0 to all diagonal entries. > > Barry > > > >> On Jun 11, 2021, at 10:16 AM, Kaushik Vijaykumar > wrote: >> >> Thanks Barry for your comments. Based on your suggestion I tried the following >> >> Main() >> { >> ierr = SNESSetFunction(snes,r1,FormFunction,&this_ctx);CHKERRQ(ierr); >> ierr = SNESSetJacobian(snes,jac,NULL,FormJacobian,&this_ctx);CHKERRQ(ierr); >> ierr = SNESSetFromOptions(snes);CHKERRQ(ierr); >> ierr = SNESSolve(snes, NULL, u);CHKERRQ(ierr); >> } >> >> In formfunction I populate the jacobian "jac" >> >> Formfunction(snes,Vec x,Vec F,void* ctx) >> { >> ierr = SNESGetJacobian(snes,&jac,Null,NULL,NULL);CHKERRQ(ierr); >> >> // "Populate jac" >> ierr = MatAssemblyBegin(jac,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); >> ierr = MatAssemblyEnd(jac,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); >> >> } >> FormJacobian(SNES snes,Vec x,Mat jac,Mat B,void *ctx) >> { >> ierr = MatAssemblyBegin(B,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); >> ierr = MatAssemblyEnd(B,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); >> >> ierr = MatAssemblyBegin(jac,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); >> ierr = MatAssemblyEnd(jac,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); >> } >> >> When I execute this I get a Petsc error, object in wrong state missing diagonal entry. >> >> I am not sure what I am missing here? >> >> Thanks >> Kaushik >> >> >> On Thu, Jun 10, 2021 at 6:37 PM Barry Smith > wrote: >> >> >> > On Jun 10, 2021, at 3:29 PM, Kaushik Vijaykumar > wrote: >> > >> > Hi everyone, >> > >> > I am trying to copy the stiffness matrix that I generated in form function to the jacobian matrix jac in form jacobian. The piece of code for that: >> > >> > >> If you have created jac and B in your main program and passed them with SNESSetJacobian() then you should be able to just use >> >> > ierr = MatCopy(ptr->K,jac,SAME_NONZERO_PATTERN);CHKERRQ(ierr); >> >> Generally jac and B are the same matrix so you don't need the second copy. >> > ierr = MatCopy(ptr->K,B,SAME_NONZERO_PATTERN);CHKERRQ(ierr); >> >> If they are sometimes the same and sometime not you can do >> >> if (jac != B) {ierr = MatCopy(ptr->K,B,SAME_NONZERO_PATTERN);CHKERRQ(ierr); } >> >> >> The following won't work because you are creating new local matrices in your form jacobian that don't exist outside of that routine. >> >> > >> > ierr = MatDuplicate(ptr->K,MAT_COPY_VALUES,&jac);CHKERRQ(ierr); >> > ierr = MatDuplicate(ptr->K,MAT_COPY_VALUES,&B);CHKERRQ(ierr); >> > >> > ierr = MatCopy(ptr->K,jac,SAME_NONZERO_PATTERN);CHKERRQ(ierr); >> > ierr = MatCopy(ptr->K,B,SAME_NONZERO_PATTERN);CHKERRQ(ierr); >> > >> > >> If the matrix you are creating in form function is the Jacobian then you don't need to use the memory and time to make copies. You can just form the matrix directly into the Jacobian you will use for SNES. You can obtain this with a call >> >> SNESGetJacobian(snes,&jac,NULL,NULL,NULL); >> >> in your form function and just put the matrix values directly into jac. Then your form Jacobian does not need to do anything but return since the jac already contains the Jacobian. >> >> On the off-change you are using the iteration A(x^n) x^{n+1} = b(x^n} and not using Newtons' method (which is why you compute A in form function, then you might be better off using SNESSetPicard() instead of SNESSetFunction(), SNESSetJacobian(). >> >> > >> > Thanks >> > Kaushik >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.croucher at auckland.ac.nz Mon Jun 14 18:47:27 2021 From: a.croucher at auckland.ac.nz (Adrian Croucher) Date: Tue, 15 Jun 2021 11:47:27 +1200 Subject: [petsc-users] flux vector In-Reply-To: References: <2e0e287c-77cf-c375-3b92-c80816c30724@auckland.ac.nz> <17d35df1-5ae0-5cf4-4cda-3592e56776d9@auckland.ac.nz> Message-ID: hi Matt, On 14/06/21 9:54 pm, Matthew Knepley wrote: > Okay, I think it is not so hard to get what you want in parallel. > There are only two kinds of faces with supportSize == 1: > > ? a) Faces on the global boundary > > ? b) Faces which are "shared" I think there is an unfortunate corner case in which faces can be both on the global boundary *and* shared. For example: consider a square 2D mesh which is partitioned down the middle, so one process has the left half. With overlap = 1, a row of ghost cells will be added along the right-hand edge of this half of the mesh. The two faces at top and bottom of this row of ghost cells are on the global boundary and also shared. With your algorithm these would be labelled as flux faces, but they shouldn't be (as they are on the global boundary). I can't see a way to eliminate those kinds of shared faces - can you? - Adrian > > It is the second set that is somewhat confusing because PetscSF does > not have 2-sided information by default. However, it can make it. > There is a two-step check for "shared": > > ? 1) Is the face in the PetscSF? Here you just check for it in the > sorted "locals" array from PetscSFGetGraph() > > ? 2) Is the face ghosted on another process? You can get this from > PetscSFGetRootRanks(). > > I just wrote a small function to check for "shared" points. After > that, I think you can just run > > ? 1) After distribution, loop overall faces on current process > > ? ? ?If face on open boundary, label face as flux face > > ? ? ?else: > > ? ? ? ?if face has supportSize != 1 or (supportSize == 1 && shared), > label face as flux face > > ? Thanks, > > ? ? ?Matt > -- Dr Adrian Croucher Senior Research Fellow Department of Engineering Science University of Auckland, New Zealand email: a.croucher at auckland.ac.nz tel: +64 (0)9 923 4611 -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Jun 14 19:21:49 2021 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 14 Jun 2021 20:21:49 -0400 Subject: [petsc-users] flux vector In-Reply-To: References: <2e0e287c-77cf-c375-3b92-c80816c30724@auckland.ac.nz> <17d35df1-5ae0-5cf4-4cda-3592e56776d9@auckland.ac.nz> Message-ID: On Mon, Jun 14, 2021 at 7:47 PM Adrian Croucher wrote: > hi Matt, > On 14/06/21 9:54 pm, Matthew Knepley wrote: > > Okay, I think it is not so hard to get what you want in parallel. There > are only two kinds of faces with supportSize == 1: > > a) Faces on the global boundary > > b) Faces which are "shared" > > I think there is an unfortunate corner case in which faces can be both on > the global boundary *and* shared. > > For example: consider a square 2D mesh which is partitioned down the > middle, so one process has the left half. With overlap = 1, a row of ghost > cells will be added along the right-hand edge of this half of the mesh. The > two faces at top and bottom of this row of ghost cells are on the global > boundary and also shared. > > With your algorithm these would be labelled as flux faces, but they > shouldn't be (as they are on the global boundary). I can't see a way to > eliminate those kinds of shared faces - can you? > I admit that it is becoming complicated, but this can also be handled. The corner case only shows up when we distribute with overlap. Thus we could, 1) Distribute the mesh without overlap 2) Mark the global boundary using the former algorithm 3) Distribute overlap = 1 using DMPlexDistributeOverlap(). DMPlexDistribute() just does these two steps at once if you ask for overlap. 4) Use the former algorithm, with the boundary label we just made, to mark flux faces. Thanks, Matt > - Adrian > > > It is the second set that is somewhat confusing because PetscSF does not > have 2-sided information by default. However, it can make it. > There is a two-step check for "shared": > > 1) Is the face in the PetscSF? Here you just check for it in the sorted > "locals" array from PetscSFGetGraph() > > 2) Is the face ghosted on another process? You can get this from > PetscSFGetRootRanks(). > > I just wrote a small function to check for "shared" points. After that, I > think you can just run > > 1) After distribution, loop overall faces on current process > > If face on open boundary, label face as flux face > > else: > > if face has supportSize != 1 or (supportSize == 1 && shared), label > face as flux face > > Thanks, > > Matt > > -- > Dr Adrian Croucher > Senior Research Fellow > Department of Engineering Science > University of Auckland, New Zealand > email: a.croucher at auckland.ac.nz > tel: +64 (0)9 923 4611 > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Mon Jun 14 20:04:25 2021 From: bsmith at petsc.dev (Barry Smith) Date: Mon, 14 Jun 2021 20:04:25 -0500 Subject: [petsc-users] Copy stiffness matrix to jacobian matrix in form jacobian In-Reply-To: References: <72658765-A7DC-4C47-84F0-BD366EF22A4B@petsc.dev> Message-ID: How are you obtaining the original form of the copied matrix? With MatDuplicate(), do you use MAT_DO_NOT_COPY_VALUES? When you do the MatCopy() what do you pass for the MatStructure argument? SAME_NONZERO_PATTERN? It seems like the options you are using is causing the zero locations to not be copied over. You are using a rather old PETSc, it should work ok but it is always best, when possible to develop code using the newest release. Barry > On Jun 14, 2021, at 4:13 PM, Kaushik Vijaykumar wrote: > > Thanks Matt and Barry for your response. I did not copy paste the entire error as it was very long, please see the error below. > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Object is in wrong state > [0]PETSC ERROR: Matrix is missing diagonal entry 0 > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.11.2, May, 18, 2019 > [0]PETSC ERROR: ./openspaf090.x on a named nid00060 by de764124 Mon Jun 14 07:18:56 2021 > [0]PETSC ERROR: Configure options --known-has-attribute-aligned=1 --known-mpi-int64_t=0 --known-bits-per-byte=8 --known-64-bit-blas-indices=0 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 --known-level1-dcache-assoc=0 --known-level1-dcache-linesize=32 --known-level1-dcache-size=32768 --known-memcmp-ok=1 --known-mpi-c-double-complex=1 --known-mpi-long-double=1 --known-mpi-shared-libraries=0 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-sizeof-char=1 --known-sizeof-double=8 --known-sizeof-float=4 --known-sizeof-int=4 --known-sizeof-long-long=8 --known-sizeof-long=8 --known-sizeof-short=2 --known-sizeof-size_t=8 --known-sizeof-void-p=8 --with-ar=ar --with-batch=1 --with-cc=cc --with-clib-autodetect=0 --with-cxx=CC --with-cxx-dialect=C++11 --with-cxxlib-autodetect=0 --with-debugging=0 --with-dependencies=0 --with-fc=ftn --with-fortran-datatypes=0 --with-fortran-interfaces=0 --with-fortranlib-autodetect=0 --with-ranlib=ranlib --with-scalar-type=real --with-shared-ld=ar --with-etags=0 --with-dependencies=0 --with-x=0 --with-ssl=0 --with-shared-libraries=0 --with-dependencies=0 --with-mpi-lib="[]" --with-mpi-include="[]" --with-blaslapack-lib="-L/opt/cray/pe/libsci/19.04.1.1/INTEL/16.0/x86_64/lib -lsci_intel_mp" --with-superlu=1 --with-superlu-include=/opt/cray/pe/tpsl/19.06.1/INTEL/19.0/sandybridge/include --with-superlu-lib="-L/opt/cray/pe/tpsl/19.06.1/INTEL/19.0/sandybridge/lib -lsuperlu" --with-superlu_dist=1 --with-superlu_dist-include=/opt/cray/pe/tpsl/19.06.1/INTEL/19.0/sandybridge/include --with-superlu_dist-lib="-L/opt/cray/pe/tpsl/19.06.1/INTEL/19.0/sandybridge/lib -lsuperlu_dist" --with-parmetis=1 --with-parmetis-include=/opt/cray/pe/tpsl/19.06.1/INTEL/19.0/sandybridge/include --with-parmetis-lib="-L/opt/cray/pe/tpsl/19.06.1/INTEL/19.0/sandybridge/lib -lparmetis" --with-metis=1 --with-metis-include=/opt/cray/pe/tpsl/19.06.1/INTEL/19.0/sandybridge/include --with-metis-lib="-L/opt/cray/pe/tpsl/19.06.1/INTEL/19.0/sandybridge/lib -lmetis" --with-ptscotch=1 --with-ptscotch-include=/opt/cray/pe/tpsl/19.06.1/INTEL/19.0/sandybridge/include --with-ptscotch-lib="-L/opt/cray/pe/tpsl/19.06.1/INTEL/19.0/sandybridge/lib -lptscotch -lscotch -lptscotcherr -lscotcherr" --with-scalapack=1 --with-scalapack-include=/opt/cray/pe/libsci/19.04.1.1/INTEL/16.0/x86_64/include --with-scalapack-lib="-L/opt/cray/pe/libsci/19.04.1.1/INTEL/16.0/x86_64/lib -lsci_intel_mpi_mp -lsci_intel_mp" --with-mumps=1 --with-mumps-include=/opt/cray/pe/tpsl/19.06.1/INTEL/19.0/sandybridge/include --with-mumps-lib="-L/opt/cray/pe/tpsl/19.06.1/INTEL/19.0/sandybridge/lib -L/opt/cray/pe/mpt/7.7.7/gni/mpich-intel/16.0/lib -lcmumps -ldmumps -lesmumps -lsmumps -lzmumps -lmumps_common -lptesmumps -lesmumps -lpord -lfmpich" --with-hdf5=1 --with-hdf5-include=/opt/cray/pe/hdf5-parallel/1.10.5.0/intel/18.0/include --with-hdf5-lib="-L/opt/cray/pe/hdf5-parallel/1.10.5.0/intel/18.0/lib -lhdf5_parallel -lz -ldl" --CFLAGS="-xavx -qopenmp -O3 -fpic" --CPPFLAGS= --CXXFLAGS="-xavx -qopenmp -O3 -fpic" --FFLAGS="-xavx -qopenmp -O3 -fpic" --LIBS=-lstdc++ --CXX_LINKER_FLAGS= --PETSC_ARCH=sandybridge --prefix=/opt/cray/pe/petsc/3.11.2.0/real/INTEL/19.0/sandybridge --with-hypre=1 --with-hypre-include=/opt/cray/pe/tpsl/19.06.1/INTEL/19.0/sandybridge/include --with-hypre-lib="-L/opt/cray/pe/tpsl/19.06.1/INTEL/19.0/sandybridge/lib -lHYPRE" --with-sundials=1 --with-sundials-include=/opt/cray/pe/tpsl/19.06.1/INTEL/19.0/sandybridge/include --with-sundials-lib="-L/opt/cray/pe/tpsl/19.06.1/INTEL/19.0/sandybridge/lib -lsundials_cvode -lsundials_cvodes -lsundials_ida -lsundials_idas -lsundials_kinsol -lsundials_nvecparallel -lsundials_nvecserial" > [0]PETSC ERROR: #1 MatILUFactorSymbolic_SeqAIJ() line 1687 in src/mat/impls/aij/seq/aijfact.c > [0]PETSC ERROR: #2 MatILUFactorSymbolic() line 6759 in src/mat/interface/matrix.c > [0]PETSC ERROR: #3 PCSetUp_ILU() line 144 in src/ksp/pc/impls/factor/ilu/ilu.c > [0]PETSC ERROR: #4 PCSetUp() line 932 in src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #5 KSPSetUp() line 391 in src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #6 KSPSolve() line 725 in src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #7 SNESSolve_NEWTONLS() line 225 in src/snes/impls/ls/ls.c > [0]PETSC ERROR: #8 SNESSolve() line 4560 in src/snes/interface/snes.c > [0]PETSC ERROR: #9 main() line 601 in spaf090_main.c > [0]PETSC ERROR: No PETSc Option Table entries > [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- > Rank 0 [Mon Jun 14 07:18:57 2021] [c0-0c0s15n0] application called MPI_Abort(MPI_COMM_WORLD, 73) - process 0 > _pmiu_daemon(SIGCHLD): [NID 00060] [c0-0c0s15n0] [Mon Jun 14 07:18:57 2021] PE RANK 0 exit signal Aborted > Application 11927 exit codes: 134 > Application 11927 resources: utime ~0s, stime ~1s, Rss ~783564, inblocks ~0, outblocks ~0 > > > In my previous approach, I was copying the jacobian in formjacobian and I did not encounter this error, of Matrix is missing a diagonal entry 0 > > If I copy my jacobian using MatGetValue and MatSetValue, the above error disappears > > formjacobian() > { > > my_ctx* ptr = (my_ctx*) ctx; // cast the pointer to void into pointer to struct > > ierr = MatGetOwnershipRange(jac,&istart,&iend); > ierr = MatGetSize(ptr->K,&m,&n); CHKERRQ(ierr); > > ierr = MatAssemblyBegin(ptr->K,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > ierr = MatAssemblyEnd(ptr->K,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > > for(i=istart;i { > for(j=0;j { > ierr = MatGetValue(ptr->K, i, j, &v);CHKERRQ(ierr); > ierr = MatSetValue(jac, i, j, v, INSERT_VALUES);CHKERRQ(ierr); > ierr = MatSetValue(B, i, j, v, INSERT_VALUES);CHKERRQ(ierr); > } > } > > ... > } > > > Thanks > Kaushik > > On Mon, Jun 14, 2021 at 3:45 PM Barry Smith > wrote: > > It is best to cut and paste the entire error message, not just two lines. We are missing all context to see where the error is occurring and possible causes. > > > >> On Jun 14, 2021, at 6:20 AM, Kaushik Vijaykumar > wrote: >> >> Thanks for the suggestion Barry, I did do that by setting it in multiple ways first by setting >> >> Formfunction(snes,Vec x,Vec F,void* ctx) >> { >> ierr = SNESGetJacobian(snes,&jac,NULL,NULL,NULL);CHKERRQ(ierr); >> ierr = MatZeroEntries(jac);CHKERRQ(ierr); >> >> ..... >> >> >> ierr = MatGetOwnershipRange(jac,&istart,&iend); >> >> for(i=istart;i> { >> val = 0.0; >> ierr = MatSetValue(jac, i, i, val, ADD_VALUES);CHKERRQ(ierr); >> } >> ierr = MatAssemblyBegin(jac,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); >> ierr = MatAssemblyEnd(jac,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); >> } >> I get the same error as before for a single processor run. >> "[0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> [0]PETSC ERROR: Object is in wrong state >> [0]PETSC ERROR: Matrix is missing diagonal entry 0" >> >> >> Thanks >> Kaushik >> >> On Fri, Jun 11, 2021 at 11:33 AM Barry Smith > wrote: >> >>> When I execute this I get a Petsc error, object in wrong state missing diagonal entry. >> >> When do you get this error? During a TSSolve? While KSP is building a preconditioner. >> >> Some parts of PETSc require that you put entries (even if they are zero) on all diagonal entries in the Jacobian. So likely you need to just make sure that your populate jac routine puts/adds 0 to all diagonal entries. >> >> Barry >> >> >> >>> On Jun 11, 2021, at 10:16 AM, Kaushik Vijaykumar > wrote: >>> >>> Thanks Barry for your comments. Based on your suggestion I tried the following >>> >>> Main() >>> { >>> ierr = SNESSetFunction(snes,r1,FormFunction,&this_ctx);CHKERRQ(ierr); >>> ierr = SNESSetJacobian(snes,jac,NULL,FormJacobian,&this_ctx);CHKERRQ(ierr); >>> ierr = SNESSetFromOptions(snes);CHKERRQ(ierr); >>> ierr = SNESSolve(snes, NULL, u);CHKERRQ(ierr); >>> } >>> >>> In formfunction I populate the jacobian "jac" >>> >>> Formfunction(snes,Vec x,Vec F,void* ctx) >>> { >>> ierr = SNESGetJacobian(snes,&jac,Null,NULL,NULL);CHKERRQ(ierr); >>> >>> // "Populate jac" >>> ierr = MatAssemblyBegin(jac,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); >>> ierr = MatAssemblyEnd(jac,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); >>> >>> } >>> FormJacobian(SNES snes,Vec x,Mat jac,Mat B,void *ctx) >>> { >>> ierr = MatAssemblyBegin(B,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); >>> ierr = MatAssemblyEnd(B,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); >>> >>> ierr = MatAssemblyBegin(jac,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); >>> ierr = MatAssemblyEnd(jac,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); >>> } >>> >>> When I execute this I get a Petsc error, object in wrong state missing diagonal entry. >>> >>> I am not sure what I am missing here? >>> >>> Thanks >>> Kaushik >>> >>> >>> On Thu, Jun 10, 2021 at 6:37 PM Barry Smith > wrote: >>> >>> >>> > On Jun 10, 2021, at 3:29 PM, Kaushik Vijaykumar > wrote: >>> > >>> > Hi everyone, >>> > >>> > I am trying to copy the stiffness matrix that I generated in form function to the jacobian matrix jac in form jacobian. The piece of code for that: >>> > >>> > >>> If you have created jac and B in your main program and passed them with SNESSetJacobian() then you should be able to just use >>> >>> > ierr = MatCopy(ptr->K,jac,SAME_NONZERO_PATTERN);CHKERRQ(ierr); >>> >>> Generally jac and B are the same matrix so you don't need the second copy. >>> > ierr = MatCopy(ptr->K,B,SAME_NONZERO_PATTERN);CHKERRQ(ierr); >>> >>> If they are sometimes the same and sometime not you can do >>> >>> if (jac != B) {ierr = MatCopy(ptr->K,B,SAME_NONZERO_PATTERN);CHKERRQ(ierr); } >>> >>> >>> The following won't work because you are creating new local matrices in your form jacobian that don't exist outside of that routine. >>> >>> > >>> > ierr = MatDuplicate(ptr->K,MAT_COPY_VALUES,&jac);CHKERRQ(ierr); >>> > ierr = MatDuplicate(ptr->K,MAT_COPY_VALUES,&B);CHKERRQ(ierr); >>> > >>> > ierr = MatCopy(ptr->K,jac,SAME_NONZERO_PATTERN);CHKERRQ(ierr); >>> > ierr = MatCopy(ptr->K,B,SAME_NONZERO_PATTERN);CHKERRQ(ierr); >>> > >>> > >>> If the matrix you are creating in form function is the Jacobian then you don't need to use the memory and time to make copies. You can just form the matrix directly into the Jacobian you will use for SNES. You can obtain this with a call >>> >>> SNESGetJacobian(snes,&jac,NULL,NULL,NULL); >>> >>> in your form function and just put the matrix values directly into jac. Then your form Jacobian does not need to do anything but return since the jac already contains the Jacobian. >>> >>> On the off-change you are using the iteration A(x^n) x^{n+1} = b(x^n} and not using Newtons' method (which is why you compute A in form function, then you might be better off using SNESSetPicard() instead of SNESSetFunction(), SNESSetJacobian(). >>> >>> > >>> > Thanks >>> > Kaushik >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.croucher at auckland.ac.nz Mon Jun 14 22:34:07 2021 From: a.croucher at auckland.ac.nz (Adrian Croucher) Date: Tue, 15 Jun 2021 15:34:07 +1200 Subject: [petsc-users] flux vector In-Reply-To: References: <2e0e287c-77cf-c375-3b92-c80816c30724@auckland.ac.nz> <17d35df1-5ae0-5cf4-4cda-3592e56776d9@auckland.ac.nz> Message-ID: <516129f2-557b-641a-2110-c84872fed2c9@auckland.ac.nz> hi On 15/06/21 12:21 pm, Matthew Knepley wrote: > > I admit that it is becoming?complicated, but this can also be handled. > > The corner case only shows up when we distribute with overlap. Thus we > could, > > 1) Distribute the mesh without overlap > > 2) Mark the global boundary using the former algorithm > > 3) Distribute overlap = 1 using DMPlexDistributeOverlap(). > DMPlexDistribute() just does these two steps at once if you ask for > overlap. > > 4) Use the former algorithm, with the boundary?label we just made, to > mark flux faces. That sounds like it might work. Do you think this perhaps slightly simpler version might also work? 1) distribute mesh without overlap 2) label boundary faces using DMPlexMarkBoundaryFaces() 3) DMPlexDistributeOverlap() 4) Un-label boundary faces with support size > 1 (i.e. the shared faces on the non-overlapped mesh- I think) - so the boundary label is now just the global boundary 5) loop over faces as in my original algorithm, labelling flux faces if they are either on the open boundary, or are not on the global boundary Unfortunately I can't test this at the moment - it looks like the Fortran interface for DMPlexDistributeOverlap() is missing? - Adrian -- Dr Adrian Croucher Senior Research Fellow Department of Engineering Science University of Auckland, New Zealand email: a.croucher at auckland.ac.nz tel: +64 (0)9 923 4611 -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Jun 15 06:18:04 2021 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 15 Jun 2021 07:18:04 -0400 Subject: [petsc-users] flux vector In-Reply-To: <516129f2-557b-641a-2110-c84872fed2c9@auckland.ac.nz> References: <2e0e287c-77cf-c375-3b92-c80816c30724@auckland.ac.nz> <17d35df1-5ae0-5cf4-4cda-3592e56776d9@auckland.ac.nz> <516129f2-557b-641a-2110-c84872fed2c9@auckland.ac.nz> Message-ID: On Mon, Jun 14, 2021 at 11:34 PM Adrian Croucher wrote: > hi > On 15/06/21 12:21 pm, Matthew Knepley wrote: > > > I admit that it is becoming complicated, but this can also be handled. > > The corner case only shows up when we distribute with overlap. Thus we > could, > > 1) Distribute the mesh without overlap > > 2) Mark the global boundary using the former algorithm > > 3) Distribute overlap = 1 using DMPlexDistributeOverlap(). > DMPlexDistribute() just does these two steps at once if you ask for overlap. > > 4) Use the former algorithm, with the boundary label we just made, to mark > flux faces. > > That sounds like it might work. Do you think this perhaps slightly simpler > version might also work? > > 1) distribute mesh without overlap > > 2) label boundary faces using DMPlexMarkBoundaryFaces() > > 3) DMPlexDistributeOverlap() > > 4) Un-label boundary faces with support size > 1 (i.e. the shared faces on > the non-overlapped mesh- I think) - so the boundary label is now just the > global boundary > > 5) loop over faces as in my original algorithm, labelling flux faces if > they are either on the open boundary, or are not on the global boundary > > Unfortunately I can't test this at the moment - it looks like the Fortran > interface for DMPlexDistributeOverlap() is missing? > Yes, that does sound simpler. I will get the Fotran interface up today. I did not use the automatic version since we need to check for NULL. Thanks, Matt > - Adrian > -- > > Dr Adrian Croucher > Senior Research Fellow > Department of Engineering Science > University of Auckland, New Zealand > email: a.croucher at auckland.ac.nz > tel: +64 (0)9 923 4611 > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Thu Jun 17 23:48:41 2021 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 17 Jun 2021 23:48:41 -0500 Subject: [petsc-users] New PETSc website, documentation and 3.15.1 patch release Message-ID: <786ff0b6-7e8d-df3-8afc-ce358374afbf@mcs.anl.gov> Dear PETSc users, We are migrating PETSc website and docs to a new location. https://petsc.org/release/ Let us know via petsc-users or petsc-maint if you encounter any issues. For now, the old website https://www.mcs.anl.gov/petsc/ will be accessible - but the content is frozen. The patch release petsc-3.15.1 is also available for download. https://petsc.org/release/download/ Satish From eijkhout at tacc.utexas.edu Fri Jun 18 08:12:44 2021 From: eijkhout at tacc.utexas.edu (Victor Eijkhout) Date: Fri, 18 Jun 2021 13:12:44 +0000 Subject: [petsc-users] [petsc-dev] New PETSc website, documentation and 3.15.1 patch release In-Reply-To: <786ff0b6-7e8d-df3-8afc-ce358374afbf@mcs.anl.gov> References: <786ff0b6-7e8d-df3-8afc-ce358374afbf@mcs.anl.gov> Message-ID: On , 2021Jun17, at 23:48, Satish Balay via petsc-dev > wrote: https://petsc.org/release/ The ?Documentation? link is https://petsc.org/release/# Maybe a more informative URL? Victor. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wence at gmx.li Fri Jun 18 08:33:35 2021 From: wence at gmx.li (Lawrence Mitchell) Date: Fri, 18 Jun 2021 14:33:35 +0100 Subject: [petsc-users] [petsc-dev] New PETSc website, documentation and 3.15.1 patch release In-Reply-To: References: <786ff0b6-7e8d-df3-8afc-ce358374afbf@mcs.anl.gov> Message-ID: > On 18 Jun 2021, at 14:12, Victor Eijkhout wrote: > > > >> On , 2021Jun17, at 23:48, Satish Balay via petsc-dev wrote: >> >> https://petsc.org/release/ > > The ?Documentation? link is > > https://petsc.org/release/# > > Maybe a more informative URL? https://petsc.org/release/documentation/ ? From eijkhout at tacc.utexas.edu Fri Jun 18 08:37:59 2021 From: eijkhout at tacc.utexas.edu (Victor Eijkhout) Date: Fri, 18 Jun 2021 13:37:59 +0000 Subject: [petsc-users] [petsc-dev] New PETSc website, documentation and 3.15.1 patch release In-Reply-To: References: <786ff0b6-7e8d-df3-8afc-ce358374afbf@mcs.anl.gov> Message-ID: <4B45F69B-8C90-4E7F-9172-5AEC410607C9@tacc.utexas.edu> On , 2021Jun18, at 08:33, Lawrence Mitchell > wrote: https://petsc.org/release/documentation/ How do I find that? Clicking ?Documentation? on the main page does not go there. Victor. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wence at gmx.li Fri Jun 18 08:43:11 2021 From: wence at gmx.li (Lawrence Mitchell) Date: Fri, 18 Jun 2021 14:43:11 +0100 Subject: [petsc-users] [petsc-dev] New PETSc website, documentation and 3.15.1 patch release In-Reply-To: <4B45F69B-8C90-4E7F-9172-5AEC410607C9@tacc.utexas.edu> References: <786ff0b6-7e8d-df3-8afc-ce358374afbf@mcs.anl.gov> <4B45F69B-8C90-4E7F-9172-5AEC410607C9@tacc.utexas.edu> Message-ID: <77A8C5C4-1B07-458C-B14B-C9C83D966B9B@gmx.li> > On 18 Jun 2021, at 14:37, Victor Eijkhout wrote: > >> >> On , 2021Jun18, at 08:33, Lawrence Mitchell wrote: >> >> https://petsc.org/release/documentation/ > > How do I find that? Clicking ?Documentation? on the main page does not go there. I clicked "Documentation" in the "menu" at the top. So don't know what happened for you. Lawrence From eijkhout at tacc.utexas.edu Fri Jun 18 08:57:10 2021 From: eijkhout at tacc.utexas.edu (Victor Eijkhout) Date: Fri, 18 Jun 2021 13:57:10 +0000 Subject: [petsc-users] [petsc-dev] New PETSc website, documentation and 3.15.1 patch release In-Reply-To: <77A8C5C4-1B07-458C-B14B-C9C83D966B9B@gmx.li> References: <786ff0b6-7e8d-df3-8afc-ce358374afbf@mcs.anl.gov> <4B45F69B-8C90-4E7F-9172-5AEC410607C9@tacc.utexas.edu> <77A8C5C4-1B07-458C-B14B-C9C83D966B9B@gmx.li> Message-ID: <99B4ECE4-3F0D-4FA4-B67D-FF419DB1B312@tacc.utexas.edu> On , 2021Jun18, at 08:43, Lawrence Mitchell > wrote: I clicked "Documentation" in the "menu" at the top. There is also a ?Documentation? in the right sidebar. The page is also titled ?Petsc 315. Documentation? And scroll a little down to ?Main topics? and there is another link to Documentation?. I?m assuming the design will go through another iteration. Glad to have been your focus group :-) Victor. -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Fri Jun 18 09:12:23 2021 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 18 Jun 2021 09:12:23 -0500 Subject: [petsc-users] [petsc-dev] New PETSc website, documentation and 3.15.1 patch release In-Reply-To: <4B45F69B-8C90-4E7F-9172-5AEC410607C9@tacc.utexas.edu> References: <786ff0b6-7e8d-df3-8afc-ce358374afbf@mcs.anl.gov> <4B45F69B-8C90-4E7F-9172-5AEC410607C9@tacc.utexas.edu> Message-ID: <32e25296-dcf5-d49e-6d26-b69e362c6a3@mcs.anl.gov> On Fri, 18 Jun 2021, Victor Eijkhout wrote: > > > On , 2021Jun18, at 08:33, Lawrence Mitchell > wrote: > > https://petsc.org/release/documentation/ > > How do I find that? Clicking ?Documentation? on the main page does not go there. I see the confusion. There is 'PETSc 3.15 Documentation' [on the right side] also listed in 'On this page' subsection links. This should be changed.. Satish From balay at mcs.anl.gov Fri Jun 18 09:19:54 2021 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 18 Jun 2021 09:19:54 -0500 Subject: [petsc-users] [petsc-dev] New PETSc website, documentation and 3.15.1 patch release In-Reply-To: <32e25296-dcf5-d49e-6d26-b69e362c6a3@mcs.anl.gov> References: <786ff0b6-7e8d-df3-8afc-ce358374afbf@mcs.anl.gov> <4B45F69B-8C90-4E7F-9172-5AEC410607C9@tacc.utexas.edu> <32e25296-dcf5-d49e-6d26-b69e362c6a3@mcs.anl.gov> Message-ID: <52fa7823-e688-62f8-4e3f-4141222e31c2@mcs.anl.gov> On Fri, 18 Jun 2021, Satish Balay via petsc-dev wrote: > On Fri, 18 Jun 2021, Victor Eijkhout wrote: > > > > > > > On , 2021Jun18, at 08:33, Lawrence Mitchell > wrote: > > > > https://petsc.org/release/documentation/ > > > > How do I find that? Clicking ?Documentation? on the main page does not go there. > > I see the confusion. There is 'PETSc 3.15 Documentation' [on the right side] also listed in 'On this page' subsection links. > > This should be changed.. I think it came from the fact that we needed to distinguish [and associate the version numbers to] https://petsc.org/release vs https://petsc.org/main Satish From eijkhout at tacc.utexas.edu Fri Jun 18 09:45:37 2021 From: eijkhout at tacc.utexas.edu (Victor Eijkhout) Date: Fri, 18 Jun 2021 14:45:37 +0000 Subject: [petsc-users] [petsc-dev] New PETSc website, documentation and 3.15.1 patch release In-Reply-To: <52fa7823-e688-62f8-4e3f-4141222e31c2@mcs.anl.gov> References: <786ff0b6-7e8d-df3-8afc-ce358374afbf@mcs.anl.gov> <4B45F69B-8C90-4E7F-9172-5AEC410607C9@tacc.utexas.edu> <32e25296-dcf5-d49e-6d26-b69e362c6a3@mcs.anl.gov> <52fa7823-e688-62f8-4e3f-4141222e31c2@mcs.anl.gov> Message-ID: <6D393649-342C-4662-9634-0C68361305DF@tacc.utexas.edu> On , 2021Jun18, at 09:19, Satish Balay > wrote: https://petsc.org/release If you called it ?Petsc 3.15? instead of ?Petsc 3.15 documentation? it would be perfectly clear. Victor. -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Fri Jun 18 10:28:43 2021 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 18 Jun 2021 10:28:43 -0500 Subject: [petsc-users] [petsc-dev] New PETSc website, documentation and 3.15.1 patch release In-Reply-To: <6D393649-342C-4662-9634-0C68361305DF@tacc.utexas.edu> References: <786ff0b6-7e8d-df3-8afc-ce358374afbf@mcs.anl.gov> <4B45F69B-8C90-4E7F-9172-5AEC410607C9@tacc.utexas.edu> <32e25296-dcf5-d49e-6d26-b69e362c6a3@mcs.anl.gov> <52fa7823-e688-62f8-4e3f-4141222e31c2@mcs.anl.gov> <6D393649-342C-4662-9634-0C68361305DF@tacc.utexas.edu> Message-ID: On Fri, 18 Jun 2021, Victor Eijkhout wrote: > > > On , 2021Jun18, at 09:19, Satish Balay > wrote: > > https://petsc.org/release > > If you called it ?Petsc 3.15? instead of ?Petsc 3.15 documentation? it would be perfectly clear. Fix at: https://gitlab.com/petsc/petsc/-/merge_requests/4109 Satish From ellen.price at cfa.harvard.edu Sun Jun 20 15:51:17 2021 From: ellen.price at cfa.harvard.edu (Ellen M. Price) Date: Sun, 20 Jun 2021 15:51:17 -0500 Subject: [petsc-users] Tracking a NaN in FVM gradient computation Message-ID: <700e829f-5417-4c7f-df4c-a54d29a1d0df@cfa.harvard.edu> Hi there PETSc, I am working my way through ex11.c and have encountered a problem. On the first pass through mesh adaption, the gradient computation in PETSc triggers a NaN, even though none of the input data are NaN. To reproduce: PETSc v3.15 with latest MPICH, no external libraries, debugging on Compile ex11.c using GCC 9.3 on Ubuntu 20.04 Run as: ./ex11 -ufv_use_amr -fp_trap Running under GDB shows that the offending line is: src/dm/impls/plex/plexfvm.c:111 This originates from the call to DMPlexReconstructGradientsFVM. I only noticed this because I was trying to resolve a discontinuity in my own initial condition for a Sod shock tube problem, but then I found that it occurs in the example without any modification, too. Is this somehow intended? If not, what steps can one take to make sure the gradients are actual numeric values? It doesn't make sense to me that they would be undefined on the first step when nothing else is NaN. Or is it that AMR in ex11 doesn't quite work yet? Thanks, Ellen Price From jed at jedbrown.org Sun Jun 20 22:59:19 2021 From: jed at jedbrown.org (Jed Brown) Date: Sun, 20 Jun 2021 21:59:19 -0600 Subject: [petsc-users] Tracking a NaN in FVM gradient computation In-Reply-To: <700e829f-5417-4c7f-df4c-a54d29a1d0df@cfa.harvard.edu> References: <700e829f-5417-4c7f-df4c-a54d29a1d0df@cfa.harvard.edu> Message-ID: <871r8vdg14.fsf@jedbrown.org> Looks like this is the relevant code. for (d = 0; d < dim; ++d) { if (cgrad[0]) cgrad[0][pd*dim+d] += fg->grad[0][d] * delta; if (cgrad[1]) cgrad[1][pd*dim+d] -= fg->grad[1][d] * delta; } I ran in a debugger and found there was already nan here: Thread 1 "ex11" received signal SIGFPE, Arithmetic exception. 0x00007ffff6c88e31 in DMPlexReconstructGradients_Internal (dm=0x555555b21230, fvm=0x5555558cc4b0, fStart=25, fEnd=41, faceGeometry=0x555555e3b3a0, cellGeometry=0x555555e54560, locX=0x555555edb7f0, grad =0x555555ee6ce0) at src/dm/impls/plex/plexfvm.c:111 111 if (cgrad[0]) cgrad[0][pd*dim+d] += fg->grad[0][d] * delta; (gdb) p fg->grad[0][d] $1 = nan(0x000000002) (gdb) p d $2 = 0 That indicates memory corruption because if the nan had been computed in an earlier step, we would have trapped there. Indeed, I see Valgrind errors. I'm adding Toby, who developed much of this and might be able to debug more efficiently. It would be useful to file an issue on GitLab. $ valgrind mpich-clang/tests/ts/tutorials/ex11 -ufv_use_amr -fp_trap ==599057== Memcheck, a memory error detector ==599057== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al. ==599057== Using Valgrind-3.17.0 and LibVEX; rerun with -h for copyright info ==599057== Command: mpich-clang/tests/ts/tutorials/ex11 -ufv_use_amr -fp_trap ==599057== ==599057== Conditional jump or move depends on uninitialised value(s) ==599057== at 0x122A31: adaptToleranceFVM (ex11.c:1608) ==599057== by 0x11A89B: main (ex11.c:1927) ==599057== ==599057== Conditional jump or move depends on uninitialised value(s) ==599057== at 0x122A77: adaptToleranceFVM (ex11.c:1609) ==599057== by 0x11A89B: main (ex11.c:1927) ==599057== ==599057== Conditional jump or move depends on uninitialised value(s) ==599057== at 0x4F3B293: VecTaggerComputeIS_FromBoxes (tagger.c:465) ==599057== by 0x4F3AB34: VecTaggerComputeIS (tagger.c:422) ==599057== by 0x122E91: adaptToleranceFVM (ex11.c:1619) ==599057== by 0x11A89B: main (ex11.c:1927) ==599057== ==599057== Conditional jump or move depends on uninitialised value(s) ==599057== at 0x4F3B293: VecTaggerComputeIS_FromBoxes (tagger.c:465) ==599057== by 0x4F3AB34: VecTaggerComputeIS (tagger.c:422) ==599057== by 0x122F23: adaptToleranceFVM (ex11.c:1620) ==599057== by 0x11A89B: main (ex11.c:1927) ==599057== ==599057== Conditional jump or move depends on uninitialised value(s) ==599057== at 0x4F3B2CB: VecTaggerComputeIS_FromBoxes (tagger.c:472) ==599057== by 0x4F3AB34: VecTaggerComputeIS (tagger.c:422) ==599057== by 0x122F23: adaptToleranceFVM (ex11.c:1620) ==599057== by 0x11A89B: main (ex11.c:1927) "Ellen M. Price" writes: > Hi there PETSc, > > I am working my way through ex11.c and have encountered a problem. On > the first pass through mesh adaption, the gradient computation in PETSc > triggers a NaN, even though none of the input data are NaN. > > To reproduce: > > PETSc v3.15 with latest MPICH, no external libraries, debugging on > Compile ex11.c using GCC 9.3 on Ubuntu 20.04 > Run as: ./ex11 -ufv_use_amr -fp_trap > > Running under GDB shows that the offending line is: > src/dm/impls/plex/plexfvm.c:111 > This originates from the call to DMPlexReconstructGradientsFVM. > > I only noticed this because I was trying to resolve a discontinuity in > my own initial condition for a Sod shock tube problem, but then I found > that it occurs in the example without any modification, too. > > Is this somehow intended? If not, what steps can one take to make sure > the gradients are actual numeric values? It doesn't make sense to me > that they would be undefined on the first step when nothing else is NaN. > Or is it that AMR in ex11 doesn't quite work yet? > > Thanks, > Ellen Price From zjorti at lanl.gov Tue Jun 22 15:29:52 2021 From: zjorti at lanl.gov (Jorti, Zakariae) Date: Tue, 22 Jun 2021 20:29:52 +0000 Subject: [petsc-users] Question about PCFieldSplit In-Reply-To: <037C3F53-8329-4950-B83B-63AC118526E3@msu.edu> References: <9bd268fa58a14c8ba55abffa3661dad1@lanl.gov> <8A8FE6DE-F995-4242-92C7-878C48ED1A70@gmail.com>, <037C3F53-8329-4950-B83B-63AC118526E3@msu.edu> Message-ID: <2e1a71b944a949c99c24ad85bed9350a@lanl.gov> Hello, I am working on DMStag and I have one dof on vertices (let us call it V), one dof on edges (let us call it E), one dof on faces ((let us call it F)) and one dof on cells (let us call it C). I build a matrix on this DM, and I was wondering if there was a way to get blocks (or sub matrices) of this matrix corresponding to specific degrees of freedom, for example rows corresponding to V dofs and columns corresponding to E dofs. I already asked this question before and the answer I got was I could call PCFieldSplitSetDetectSaddlePoint with the diagonal entries being of the matrix being zero or nonzero. That worked well. Nonetheless, I am curious to know if there was another alternative that does not require creating a dummy matrix with appropriate diagonal entries and solving a dummy linear system with this matrix to define the splits. Many thanks. Best regards, Zakariae ________________________________ From: petsc-users on behalf of Tang, Qi Sent: Sunday, April 18, 2021 11:51:59 PM To: Patrick Sanan Cc: petsc-users at mcs.anl.gov; Tang, Xianzhu Subject: [EXTERNAL] Re: [petsc-users] Question about PCFieldSplit Thanks a lot, Patrick. We appreciate your help. Qi On Apr 18, 2021, at 11:30 PM, Patrick Sanan > wrote: We have this functionality in a branch, which I'm working on cleaning up to get to master. It doesn't use PETScSection. Sorry about the delay! You can only use PCFieldSplitSetDetectSaddlePoint when your diagonal entries being zero or non-zero defines the splits correctly. Am 17.04.2021 um 21:09 schrieb Matthew Knepley >: On Fri, Apr 16, 2021 at 8:39 PM Jorti, Zakariae via petsc-users > wrote: Hello, I have a DMStag grid with one dof on each edge and face center. I want to use a PCFieldSplit preconditioner on a Jacobian matrix that I assume is already split but I am not sure how to determine the fields. In the DMStag examples (ex2.c and ex3.c), the function PCFieldSplitSetDetectSaddlePoint is used to determine those fields based on zero diagonal entries. In my case, I have a Jacobian matrix that does not have zero diagonal entries. Can I use that PCFieldSplitSetDetectSaddlePoint in this case? If not, how should I do? Should I do like this example (https://www.mcs.anl.gov/petsc/petsc-master/src/ksp/ksp/tutorials/ex43.c.html): const PetscInt Bfields[1] = {0},Efields[1] = {1}; KSPGetPC(ksp,&pc); PCFieldSplitSetBlockSize(pc,2); PCFieldSplitSetFields(pc,"B",1,Bfields,Bfields); PCFieldSplitSetFields(pc,"E",1,Efields,Efields); where my B unknowns are defined on face centers and E unknowns are defined on edge centers? That will not work.That interface only works for colocated fields that you get from DMDA. Patrick, does DMSTAG use PetscSection? Then the field split would be automatically calculated. If not, does it maintain the field division so that it could be given to PCFIELDSPLIT as ISes? Thanks, Matt One last thing, I do not know which field comes first. Is it the one defined for face dofs or edge dofs. Thank you. Best regards, Zakariae -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From patrick.sanan at gmail.com Tue Jun 22 23:51:37 2021 From: patrick.sanan at gmail.com (Patrick Sanan) Date: Wed, 23 Jun 2021 06:51:37 +0200 Subject: [petsc-users] Question about PCFieldSplit In-Reply-To: <2e1a71b944a949c99c24ad85bed9350a@lanl.gov> References: <9bd268fa58a14c8ba55abffa3661dad1@lanl.gov> <8A8FE6DE-F995-4242-92C7-878C48ED1A70@gmail.com> <037C3F53-8329-4950-B83B-63AC118526E3@msu.edu> <2e1a71b944a949c99c24ad85bed9350a@lanl.gov> Message-ID: <3667C2C9-C4E7-48C7-B7FC-B48DDA210BCF@gmail.com> Hi Zakariae - The usual way to do this is to define an IS (index set) with the degrees of freedom of interest for the rows, and another one for the columns, and then use MatCreateSubmatrix [1] . There's not a particularly convenient way to create an IS with the degrees of freedom corresponding to a particular "stratum" (i.e. elements, faces, edges, or vertices) of a DMStag, but fortunately I believe we have some code to do exactly this in a development branch. I'll track it down and see if it can quickly be added to the main branch. [1]: https://petsc.org/release/docs/manualpages/Mat/MatCreateSubMatrix.html > Am 22.06.2021 um 22:29 schrieb Jorti, Zakariae : > > Hello, > > I am working on DMStag and I have one dof on vertices (let us call it V), one dof on edges (let us call it E), one dof on faces ((let us call it F)) and one dof on cells (let us call it C). > I build a matrix on this DM, and I was wondering if there was a way to get blocks (or sub matrices) of this matrix corresponding to specific degrees of freedom, for example rows corresponding to V dofs and columns corresponding to E dofs. > I already asked this question before and the answer I got was I could call PCFieldSplitSetDetectSaddlePoint with the diagonal entries being of the matrix being zero or nonzero. > That worked well. Nonetheless, I am curious to know if there was another alternative that does not require creating a dummy matrix with appropriate diagonal entries and solving a dummy linear system with this matrix to define the splits. > > Many thanks. > > Best regards, > > Zakariae > From: petsc-users on behalf of Tang, Qi > Sent: Sunday, April 18, 2021 11:51:59 PM > To: Patrick Sanan > Cc: petsc-users at mcs.anl.gov; Tang, Xianzhu > Subject: [EXTERNAL] Re: [petsc-users] Question about PCFieldSplit > > Thanks a lot, Patrick. We appreciate your help. > > Qi > > > >> On Apr 18, 2021, at 11:30 PM, Patrick Sanan > wrote: >> >> We have this functionality in a branch, which I'm working on cleaning up to get to master. It doesn't use PETScSection. Sorry about the delay! >> >> You can only use PCFieldSplitSetDetectSaddlePoint when your diagonal entries being zero or non-zero defines the splits correctly. >> >>> Am 17.04.2021 um 21:09 schrieb Matthew Knepley >: >>> >>> On Fri, Apr 16, 2021 at 8:39 PM Jorti, Zakariae via petsc-users > wrote: >>> Hello, >>> >>> I have a DMStag grid with one dof on each edge and face center. >>> I want to use a PCFieldSplit preconditioner on a Jacobian matrix that I assume is already split but I am not sure how to determine the fields. >>> In the DMStag examples (ex2.c and ex3.c), the function PCFieldSplitSetDetectSaddlePoint is used to determine those fields based on zero diagonal entries. In my case, I have a Jacobian matrix that does not have zero diagonal entries. >>> Can I use that PCFieldSplitSetDetectSaddlePoint in this case? >>> If not, how should I do? >>> Should I do like this example (https://www.mcs.anl.gov/petsc/petsc-master/src/ksp/ksp/tutorials/ex43.c.html ): >>> const PetscInt Bfields[1] = {0},Efields[1] = {1}; >>> KSPGetPC(ksp,&pc); >>> PCFieldSplitSetBlockSize(pc,2); >>> PCFieldSplitSetFields(pc,"B",1,Bfields,Bfields); PCFieldSplitSetFields(pc,"E",1,Efields,Efields); >>> where my B unknowns are defined on face centers and E unknowns are defined on edge centers? >>> That will not work.That interface only works for colocated fields that you get from DMDA. >>> >>> Patrick, does DMSTAG use PetscSection? Then the field split would be automatically calculated. If not, does it maintain the >>> field division so that it could be given to PCFIELDSPLIT as ISes? >>> >>> Thanks, >>> >>> Matt >>> One last thing, I do not know which field comes first. Is it the one defined for face dofs or edge dofs. >>> >>> Thank you. >>> Best regards, >>> >>> Zakariae >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Jun 23 05:15:29 2021 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 23 Jun 2021 06:15:29 -0400 Subject: [petsc-users] Question about PCFieldSplit In-Reply-To: <3667C2C9-C4E7-48C7-B7FC-B48DDA210BCF@gmail.com> References: <9bd268fa58a14c8ba55abffa3661dad1@lanl.gov> <8A8FE6DE-F995-4242-92C7-878C48ED1A70@gmail.com> <037C3F53-8329-4950-B83B-63AC118526E3@msu.edu> <2e1a71b944a949c99c24ad85bed9350a@lanl.gov> <3667C2C9-C4E7-48C7-B7FC-B48DDA210BCF@gmail.com> Message-ID: On Wed, Jun 23, 2021 at 12:51 AM Patrick Sanan wrote: > Hi Zakariae - > > The usual way to do this is to define an IS (index set) with the degrees > of freedom of interest for the rows, and another one for the columns, and > then use MatCreateSubmatrix [1] . > > There's not a particularly convenient way to create an IS with the degrees > of freedom corresponding to a particular "stratum" (i.e. elements, faces, > edges, or vertices) of a DMStag, but fortunately I believe we have some > code to do exactly this in a development branch. > > I'll track it down and see if it can quickly be added to the main branch. > Note that an easy way to keep track of this would be to create a section with the different locations as fields. This Section could then easily create the ISes, and could automatically interface with PCFIELDSPLIT. Thanks, Matt > > [1]: > https://petsc.org/release/docs/manualpages/Mat/MatCreateSubMatrix.html > > Am 22.06.2021 um 22:29 schrieb Jorti, Zakariae : > > Hello, > > I am working on DMStag and I have one dof on vertices (let us call it V), one > dof on edges (let us call it E), one dof on faces ((let us call it F)) and > one dof on cells (let us call it C). > I build a matrix on this DM, and I was wondering if there was a way to > get blocks (or sub matrices) of this matrix corresponding to specific > degrees of freedom, for example rows corresponding to V dofs and columns > corresponding to E dofs. > I already asked this question before and the answer I got was I could call PCFieldSplitSetDetectSaddlePoint > with the diagonal entries being of the matrix being zero or nonzero. > That worked well. Nonetheless, I am curious to know if there was another > alternative that does not require creating a dummy matrix with > appropriate diagonal entries and solving a dummy linear system with this > matrix to define the splits. > > > Many thanks. > > Best regards, > > Zakariae > ------------------------------ > *From:* petsc-users on behalf of Tang, > Qi > *Sent:* Sunday, April 18, 2021 11:51:59 PM > *To:* Patrick Sanan > *Cc:* petsc-users at mcs.anl.gov; Tang, Xianzhu > *Subject:* [EXTERNAL] Re: [petsc-users] Question about PCFieldSplit > > Thanks a lot, Patrick. We appreciate your help. > > Qi > > > > On Apr 18, 2021, at 11:30 PM, Patrick Sanan > wrote: > > We have this functionality in a branch, which I'm working on cleaning up > to get to master. It doesn't use PETScSection. Sorry about the delay! > > You can only use PCFieldSplitSetDetectSaddlePoint when your diagonal > entries being zero or non-zero defines the splits correctly. > > Am 17.04.2021 um 21:09 schrieb Matthew Knepley : > > On Fri, Apr 16, 2021 at 8:39 PM Jorti, Zakariae via petsc-users < > petsc-users at mcs.anl.gov> wrote: > >> Hello, >> >> I have a DMStag grid with one dof on each edge and face center. >> I want to use a PCFieldSplit preconditioner on a Jacobian matrix that I >> assume is already split but I am not sure how to determine the fields. >> In the DMStag examples (ex2.c and ex3.c), the >> function PCFieldSplitSetDetectSaddlePoint is used to determine those fields >> based on zero diagonal entries. In my case, I have a Jacobian matrix that >> does not have zero diagonal entries. >> Can I use that PCFieldSplitSetDetectSaddlePoint in this case? >> If not, how should I do? >> Should I do like this example ( >> https://www.mcs.anl.gov/petsc/petsc-master/src/ksp/ksp/tutorials/ex43.c.html >> >> ): >> const PetscInt Bfields[1] = {0},Efields[1] = {1}; >> KSPGetPC(ksp,&pc); >> PCFieldSplitSetBlockSize(pc,2); >> PCFieldSplitSetFields(pc,"B",1,Bfields,Bfields); >> PCFieldSplitSetFields(pc,"E",1,Efields,Efields); >> where my B unknowns are defined on face centers and E unknowns are >> defined on edge centers? >> > That will not work.That interface only works for colocated fields that you > get from DMDA. > > Patrick, does DMSTAG use PetscSection? Then the field split would be > automatically calculated. If not, does it maintain the > field division so that it could be given to PCFIELDSPLIT as ISes? > > Thanks, > > Matt > >> One last thing, I do not know which field comes first. Is it the one >> defined for face dofs or edge dofs. >> >> Thank you. >> Best regards, >> >> Zakariae >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.panneerselvam at mpie.de Thu Jun 24 03:43:55 2021 From: a.panneerselvam at mpie.de (Abisheik Panneerselvam) Date: Thu, 24 Jun 2021 10:43:55 +0200 Subject: [petsc-users] Getting nodal coordinates from a FEM computation Message-ID: Hello, Good Morning. My name is Abisheik Panneerselvam and I am presently working in Max-Planck D?sseldorf. I am working on a FE software development project. The FE computation code is already written in Fortran (using Petsc for solving) few years ago. I am currently working on post-processing the results of the computation (nodal displacements, integration point displacements etc.). I am facing problems in getting the coordinates of vertices. ?I tried the following call SNESGetDM(mechanical_snes,dm_local,ierr); CHKERRQ(ierr)?????????????????????????????????? !Gets the local DM from the SNES solve call DMGetLocalVector(dm_local,x_local,ierr); CHKERRQ(ierr)???????????? ????????????????????????????!Gets the local vector from the DM This functions successfully gets the coordinates of all the nodes. This works fine for linear elements as there are only nodes on cell vertices. But for higher order elements, the local vector (x_local) contains additional nodal points (I think its called ghost points in Petsc) on the edges. In this case how do I get the coordinates of only the cell vertices? ? I tried the below code for getting the vertex coordinates but its giving random values. call DMPlexGetDepthStratum(dm_local, 0, vStart, vEnd,ierr); CHKERRQ(ierr)??????????????????????? !Get the vertices ?call VecGetArrayF90(x_local, node_coords_local,ierr) ? v_count=1 ? do v=vStart, vEnd-1 ??? call DMPlexGetPointLocal(dm_local, v, x, y, ierr); CHKERRQ(ierr)?? ??????????????????????????????????????!Get the position of the vertice in the local DM ??? d_count=1 ??? do d=x,y-1 ????? node_coords(d_count,v_count)=node_coords_local(d) ????? d_count=d_count+1 ??? end do ??? v_count = v_count+1 ? end do It will be really helpful if you could advise me on this. Best regards Abisheik ------------------------------------------------- Max-Planck-Institut f?r Eisenforschung GmbH Max-Planck-Stra?e 1 D-40237 D?sseldorf Handelsregister B 2533 Amtsgericht D?sseldorf Gesch?ftsf?hrung Prof. Dr. Gerhard Dehm Prof. Dr. J?rg Neugebauer Prof. Dr. Dierk Raabe Dr. Kai de Weldige Ust.-Id.-Nr.: DE 11 93 58 514 Steuernummer: 105 5891 1000 Please consider that invitations and e-mails of our institute are only valid if they end with ?@mpie.de. If you are not sure of the validity please contact rco at mpie.de Bitte beachten Sie, dass Einladungen zu Veranstaltungen und E-Mails aus unserem Haus nur mit der Endung ?@mpie.de g?ltig sind. In Zweifelsf?llen wenden Sie sich bitte an rco at mpie.de ------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Jun 24 14:16:29 2021 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 24 Jun 2021 15:16:29 -0400 Subject: [petsc-users] Getting nodal coordinates from a FEM computation In-Reply-To: References: Message-ID: On Thu, Jun 24, 2021 at 10:34 AM Abisheik Panneerselvam < a.panneerselvam at mpie.de> wrote: > Hello, > > > > Good Morning. My name is Abisheik Panneerselvam and I am presently working > in Max-Planck D?sseldorf. > > I am working on a FE software development project. The FE computation code > is already written in Fortran (using Petsc for solving) few years ago. I am > currently working on post-processing the results of the computation (nodal > displacements, integration point displacements etc.). I am facing problems > in getting the coordinates of vertices. I tried the following > > > > call SNESGetDM(mechanical_snes,dm_local,ierr); CHKERRQ(ierr) > !Gets the local DM from the SNES solve > > call DMGetLocalVector(dm_local,x_local,ierr); CHKERRQ(ierr) > !Gets the local vector from the DM > > > > This functions successfully gets the coordinates of all the nodes. This > works fine for linear elements as there are only nodes on cell vertices. > But for higher order elements, the local vector (x_local) contains > additional nodal points (I think its called ghost points in Petsc) on the > edges. In this case how do I get the coordinates of only the cell vertices? > > > > > I tried the below code for getting the vertex coordinates but its giving > random values. > > > > call DMPlexGetDepthStratum(dm_local, 0, vStart, vEnd,ierr); CHKERRQ(ierr) > !Get the vertices > > call VecGetArrayF90(x_local, node_coords_local,ierr) > > v_count=1 > > do v=vStart, vEnd-1 > > call DMPlexGetPointLocal(dm_local, v, x, y, ierr); CHKERRQ(ierr) !Get > the position of the vertice in the local DM > > d_count=1 > > do d=x,y-1 > > node_coords(d_count,v_count)=node_coords_local(d) > > d_count=d_count+1 > > end do > > v_count = v_count+1 > > end do > > > > > > It will be really helpful if you could advise me on this. > Hi Abisheik, One question to start off. Are you using DMPlex to describe the grid? It sounded like your FEM code was in Fortran. If you are only using PETSc for solving the equations, it will not know the difference between unknowns on vertices and edges. Thanks, Matt > > > Best regards > > Abisheik > > > ------------------------------ > ------------------------------------------------- > Max-Planck-Institut f?r Eisenforschung GmbH > Max-Planck-Stra?e 1 > D-40237 D?sseldorf > > Handelsregister B 2533 > Amtsgericht D?sseldorf > > Gesch?ftsf?hrung > Prof. Dr. Gerhard Dehm > Prof. Dr. J?rg Neugebauer > Prof. Dr. Dierk Raabe > Dr. Kai de Weldige > > Ust.-Id.-Nr.: DE 11 93 58 514 > Steuernummer: 105 5891 1000 > > > Please consider that invitations and e-mails of our institute are > only valid if they end with ?@mpie.de. > If you are not sure of the validity please contact rco at mpie.de > > Bitte beachten Sie, dass Einladungen zu Veranstaltungen und E-Mails > aus unserem Haus nur mit der Endung ?@mpie.de g?ltig sind. > In Zweifelsf?llen wenden Sie sich bitte an rco at mpie.de > ------------------------------------------------- > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bantingl at myumanitoba.ca Thu Jun 24 14:44:43 2021 From: bantingl at myumanitoba.ca (Lucas Banting) Date: Thu, 24 Jun 2021 19:44:43 +0000 Subject: [petsc-users] Any workarounds for TAO in a complex scalar type code? Message-ID: Hi, I have a complex valued optimization code that requires non-linear least squares curve fitting at each iteration. I wanted to use Tao to do the non-linear least squares problem, but just discovered that Tao can only be used with real scalar types. Is there any workarounds? I was wondering if maybe I could somehow use two different Petsc installations at once, or is there any other ways to hack tao into working? Thanks, Lucas -------------- next part -------------- An HTML attachment was scrubbed... URL: From adener at anl.gov Thu Jun 24 15:25:26 2021 From: adener at anl.gov (Dener, Alp) Date: Thu, 24 Jun 2021 20:25:26 +0000 Subject: [petsc-users] Any workarounds for TAO in a complex scalar type code? In-Reply-To: References: Message-ID: Hi Lucas, In theory, you can attempt to solve complex-valued problems by concatenating the real and imaginary parts into a single real-valued solution vector and solving it as if it?s a real-valued problem. There?s a more practical issue of the PETSc build excluding the TAO source code entirely for complex builds. I think this is done because we cannot guarantee that TAO code has been completely rigorous in its distinction of PetscReal from PetscScalar. There may be places where we used Scalar and assumed it?s Real. The methods developed in the last few years have paid more attention to this but there?s a lot of older code floating around that might break even if you hacked the build into compiling TAO for complex configurations. I?m not entirely sure what?s necessary to enable TAO builds for complex PETSc. Perhaps Todd, Satish or Barry will have a better idea. If you want to give it a try though I?d be happy to help you resolve the errors as they come up so that it can be compiled for complex builds. Can?t provide convergence guarantees but it would at least let you attempt a solution. Alp On Jun 24, 2021, at 2:44 PM, Lucas Banting > wrote: Hi, I have a complex valued optimization code that requires non-linear least squares curve fitting at each iteration. I wanted to use Tao to do the non-linear least squares problem, but just discovered that Tao can only be used with real scalar types. Is there any workarounds? I was wondering if maybe I could somehow use two different Petsc installations at once, or is there any other ways to hack tao into working? Thanks, Lucas -------------- next part -------------- An HTML attachment was scrubbed... URL: From junchao.zhang at gmail.com Thu Jun 24 16:02:12 2021 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Thu, 24 Jun 2021 16:02:12 -0500 Subject: [petsc-users] Any workarounds for TAO in a complex scalar type code? In-Reply-To: References: Message-ID: On Thu, Jun 24, 2021 at 3:25 PM Dener, Alp via petsc-users < petsc-users at mcs.anl.gov> wrote: > Hi Lucas, > > In theory, you can attempt to solve complex-valued problems by > concatenating the real and imaginary parts into a single real-valued > solution vector and solving it as if it?s a real-valued problem. > > There?s a more practical issue of the PETSc build excluding the TAO source > code entirely for complex builds. I think this is done because we cannot > guarantee that TAO code has been completely rigorous in its distinction of > PetscReal from PetscScalar. There may be places where we used Scalar and > assumed it?s Real. The methods developed in the last few years have paid > more attention to this but there?s a lot of older code floating around that > might break even if you hacked the build into compiling TAO for complex > configurations. > If an expression does not support PetscComplex, shouldn't it report a compilation error? > > I?m not entirely sure what?s necessary to enable TAO builds for complex > PETSc. Perhaps Todd, Satish or Barry will have a better idea. If you want > to give it a try though I?d be happy to help you resolve the errors as they > come up so that it can be compiled for complex builds. Can?t provide > convergence guarantees but it would at least let you attempt a solution. > > Alp > > On Jun 24, 2021, at 2:44 PM, Lucas Banting > wrote: > > Hi, > > I have a complex valued optimization code that requires non-linear least > squares curve fitting at each iteration. I wanted to use Tao to do the > non-linear least squares problem, but just discovered that Tao can only be > used with real scalar types. Is there any workarounds? > > I was wondering if maybe I could somehow use two different Petsc > installations at once, or is there any other ways to hack tao into working? > > Thanks, > Lucas > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Thu Jun 24 22:11:07 2021 From: bsmith at petsc.dev (Barry Smith) Date: Thu, 24 Jun 2021 22:11:07 -0500 Subject: [petsc-users] Any workarounds for TAO in a complex scalar type code? In-Reply-To: References: Message-ID: $ git grep requiresscalar bound/makefile:#requiresscalar real complementarity/makefile:#requiresscalar real constrained/makefile:#requiresscalar real leastsquares/makefile:#requiresscalar real linesearch/impls/makefile:#requiresscalar real pde_constrained/makefile:#requiresscalar real quadratic/makefile:#requiresscalar real unconstrained/makefile:#requiresscalar real We added these requirements when Tao was brought back into the PETSc fold as Alp mentioned. It would be good to start eliminating the requirements when possible. I have removed the leastsquares/makefile:#requiresscalar real and have started a build to see how it goes. If everything goes well maybe we can get it working for complex without too much difficulty. Barry > On Jun 24, 2021, at 4:02 PM, Junchao Zhang wrote: > > > > > On Thu, Jun 24, 2021 at 3:25 PM Dener, Alp via petsc-users > wrote: > Hi Lucas, > > In theory, you can attempt to solve complex-valued problems by concatenating the real and imaginary parts into a single real-valued solution vector and solving it as if it?s a real-valued problem. > > There?s a more practical issue of the PETSc build excluding the TAO source code entirely for complex builds. I think this is done because we cannot guarantee that TAO code has been completely rigorous in its distinction of PetscReal from PetscScalar. There may be places where we used Scalar and assumed it?s Real. The methods developed in the last few years have paid more attention to this but there?s a lot of older code floating around that might break even if you hacked the build into compiling TAO for complex configurations. > If an expression does not support PetscComplex, shouldn't it report a compilation error? > > I?m not entirely sure what?s necessary to enable TAO builds for complex PETSc. Perhaps Todd, Satish or Barry will have a better idea. If you want to give it a try though I?d be happy to help you resolve the errors as they come up so that it can be compiled for complex builds. Can?t provide convergence guarantees but it would at least let you attempt a solution. > > Alp > >> On Jun 24, 2021, at 2:44 PM, Lucas Banting > wrote: >> >> Hi, >> >> I have a complex valued optimization code that requires non-linear least squares curve fitting at each iteration. I wanted to use Tao to do the non-linear least squares problem, but just discovered that Tao can only be used with real scalar types. Is there any workarounds? >> >> I was wondering if maybe I could somehow use two different Petsc installations at once, or is there any other ways to hack tao into working? >> >> Thanks, >> Lucas > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Thu Jun 24 22:29:27 2021 From: bsmith at petsc.dev (Barry Smith) Date: Thu, 24 Jun 2021 22:29:27 -0500 Subject: [petsc-users] Any workarounds for TAO in a complex scalar type code? In-Reply-To: References: Message-ID: Least squares has two supported types: brgn and pounders. They will not compile with complex. I suspect the problem, at least with brgn, is its handling of bound constraints. Perhaps for complex numbers the handling of constraints can simply be #ifdef out? If one does not need constraints. Barry You can start by removing the leastsquares/makefile:#requiresscalar real adding it into the pounders directory and then compiling and "fixing" the code in brng.c to work with complex. But BRNG is a wrapper around TAOBNTL so you need to get that to also build correctly with complex. > On Jun 24, 2021, at 10:11 PM, Barry Smith wrote: > > > $ git grep requiresscalar > bound/makefile:#requiresscalar real > complementarity/makefile:#requiresscalar real > constrained/makefile:#requiresscalar real > leastsquares/makefile:#requiresscalar real > linesearch/impls/makefile:#requiresscalar real > pde_constrained/makefile:#requiresscalar real > quadratic/makefile:#requiresscalar real > unconstrained/makefile:#requiresscalar real > > We added these requirements when Tao was brought back into the PETSc fold as Alp mentioned. It would be good to start eliminating the requirements when possible. > > I have removed the leastsquares/makefile:#requiresscalar real and have started a build to see how it goes. If everything goes well maybe we can get it working for complex without too much difficulty. > > Barry > > >> On Jun 24, 2021, at 4:02 PM, Junchao Zhang > wrote: >> >> >> >> >> On Thu, Jun 24, 2021 at 3:25 PM Dener, Alp via petsc-users > wrote: >> Hi Lucas, >> >> In theory, you can attempt to solve complex-valued problems by concatenating the real and imaginary parts into a single real-valued solution vector and solving it as if it?s a real-valued problem. >> >> There?s a more practical issue of the PETSc build excluding the TAO source code entirely for complex builds. I think this is done because we cannot guarantee that TAO code has been completely rigorous in its distinction of PetscReal from PetscScalar. There may be places where we used Scalar and assumed it?s Real. The methods developed in the last few years have paid more attention to this but there?s a lot of older code floating around that might break even if you hacked the build into compiling TAO for complex configurations. >> If an expression does not support PetscComplex, shouldn't it report a compilation error? >> >> I?m not entirely sure what?s necessary to enable TAO builds for complex PETSc. Perhaps Todd, Satish or Barry will have a better idea. If you want to give it a try though I?d be happy to help you resolve the errors as they come up so that it can be compiled for complex builds. Can?t provide convergence guarantees but it would at least let you attempt a solution. >> >> Alp >> >>> On Jun 24, 2021, at 2:44 PM, Lucas Banting > wrote: >>> >>> Hi, >>> >>> I have a complex valued optimization code that requires non-linear least squares curve fitting at each iteration. I wanted to use Tao to do the non-linear least squares problem, but just discovered that Tao can only be used with real scalar types. Is there any workarounds? >>> >>> I was wondering if maybe I could somehow use two different Petsc installations at once, or is there any other ways to hack tao into working? >>> >>> Thanks, >>> Lucas >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.croucher at auckland.ac.nz Tue Jun 29 21:04:17 2021 From: a.croucher at auckland.ac.nz (Adrian Croucher) Date: Wed, 30 Jun 2021 14:04:17 +1200 Subject: [petsc-users] Boundary ghost cell types in PETSc 3.15 Message-ID: <4d88edb0-9a30-6462-02e7-2f6d3b52d4e8@auckland.ac.nz> hi I've been testing building my code with PETSc 3.15 and have hit a weird problem. If I create a DMPlex and add boundary ghost cells using DMPlexConstructGhostCells(), I expect the boundary ghost cells to have cell type DM_POLYTOPE_FV_GHOST, if I check the cell types using DMPlexGetCellTypeLabel() and DMLabelGetValue(). That is true if I build using PETSc 3.14. But with 3.15 it appears the boundary ghost cells are ending up with cell type DM_POLYTOPE_INTERIOR_GHOST. These types are defined in include/petscdmtypes.h and currently DM_POLYTOPE_FV_GHOST = 11 and DM_POLYTOPE_INTERIOR_GHOST = 12. I've attached a minimal example program in Fortran. It creates a box mesh and adds boundary ghost cells, then counts how many have type DM_POLYTOPE_FV_GHOST. With PETSc 3.14 it is 24, the same as the number of ghost cells. With PETSc 3.15 it is zero, and the boundary ghost cells all have type 12. Any clues? Regards, Adrian -- Dr Adrian Croucher Senior Research Fellow Department of Engineering Science University of Auckland, New Zealand email: a.croucher at auckland.ac.nz tel: +64 (0)9 923 4611 -------------- next part -------------- A non-text attachment was scrubbed... Name: testbdy.F90 Type: text/x-fortran Size: 1284 bytes Desc: not available URL: From knepley at gmail.com Wed Jun 30 06:13:43 2021 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 30 Jun 2021 06:13:43 -0500 Subject: [petsc-users] Boundary ghost cell types in PETSc 3.15 In-Reply-To: <4d88edb0-9a30-6462-02e7-2f6d3b52d4e8@auckland.ac.nz> References: <4d88edb0-9a30-6462-02e7-2f6d3b52d4e8@auckland.ac.nz> Message-ID: On Tue, Jun 29, 2021 at 9:04 PM Adrian Croucher wrote: > hi > > I've been testing building my code with PETSc 3.15 and have hit a weird > problem. > > If I create a DMPlex and add boundary ghost cells using > DMPlexConstructGhostCells(), I expect the boundary ghost cells to have > cell type DM_POLYTOPE_FV_GHOST, if I check the cell types using > DMPlexGetCellTypeLabel() and DMLabelGetValue(). > > That is true if I build using PETSc 3.14. But with 3.15 it appears the > boundary ghost cells are ending up with cell type > DM_POLYTOPE_INTERIOR_GHOST. > > These types are defined in include/petscdmtypes.h and currently > DM_POLYTOPE_FV_GHOST = 11 and DM_POLYTOPE_INTERIOR_GHOST = 12. > > I've attached a minimal example program in Fortran. It creates a box > mesh and adds boundary ghost cells, then counts how many have type > DM_POLYTOPE_FV_GHOST. > > With PETSc 3.14 it is 24, the same as the number of ghost cells. With > PETSc 3.15 it is zero, and the boundary ghost cells all have type 12. > > Any clues? > Yes, src/dm/f90-mod/petscdm.h is missing PYRAMID. I will make a fix right now. Thanks, Matt > Regards, Adrian > > -- > Dr Adrian Croucher > Senior Research Fellow > Department of Engineering Science > University of Auckland, New Zealand > email: a.croucher at auckland.ac.nz > tel: +64 (0)9 923 4611 > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Jun 30 06:17:28 2021 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 30 Jun 2021 06:17:28 -0500 Subject: [petsc-users] Boundary ghost cell types in PETSc 3.15 In-Reply-To: References: <4d88edb0-9a30-6462-02e7-2f6d3b52d4e8@auckland.ac.nz> Message-ID: On Wed, Jun 30, 2021 at 6:13 AM Matthew Knepley wrote: > On Tue, Jun 29, 2021 at 9:04 PM Adrian Croucher > wrote: > >> hi >> >> I've been testing building my code with PETSc 3.15 and have hit a weird >> problem. >> >> If I create a DMPlex and add boundary ghost cells using >> DMPlexConstructGhostCells(), I expect the boundary ghost cells to have >> cell type DM_POLYTOPE_FV_GHOST, if I check the cell types using >> DMPlexGetCellTypeLabel() and DMLabelGetValue(). >> >> That is true if I build using PETSc 3.14. But with 3.15 it appears the >> boundary ghost cells are ending up with cell type >> DM_POLYTOPE_INTERIOR_GHOST. >> >> These types are defined in include/petscdmtypes.h and currently >> DM_POLYTOPE_FV_GHOST = 11 and DM_POLYTOPE_INTERIOR_GHOST = 12. >> >> I've attached a minimal example program in Fortran. It creates a box >> mesh and adds boundary ghost cells, then counts how many have type >> DM_POLYTOPE_FV_GHOST. >> >> With PETSc 3.14 it is 24, the same as the number of ghost cells. With >> PETSc 3.15 it is zero, and the boundary ghost cells all have type 12. >> >> Any clues? >> > > Yes, src/dm/f90-mod/petscdm.h is missing PYRAMID. I will make a fix right > now. > Here it is: https://gitlab.com/petsc/petsc/-/merge_requests/4140 Thanks, Matt > Thanks, > > Matt > > >> Regards, Adrian >> >> -- >> Dr Adrian Croucher >> Senior Research Fellow >> Department of Engineering Science >> University of Auckland, New Zealand >> email: a.croucher at auckland.ac.nz >> tel: +64 (0)9 923 4611 >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From pjool at mek.dtu.dk Wed Jun 30 06:49:09 2021 From: pjool at mek.dtu.dk (=?iso-8859-1?Q?Peder_J=F8rgensgaard_Olesen?=) Date: Wed, 30 Jun 2021 11:49:09 +0000 Subject: [petsc-users] Scatter parallel Vec to sequential Vec on non-zeroth process Message-ID: <93521e6acde64da2af7c415ceee9273c@mek.dtu.dk> Hello I have a parallel vector, and I'd like to copy its data to a sequential vector located on a given process that is not necessarily the 0th process, so using VecScatterCreateToZero wouldn't do here. How might one approach this? Med venlig hilsen / Best Regards Peder J?rgensgaard Olesen PhD Student, Turbulence Research Lab Dept. of Mechanical Engineering Technical University of Denmark Koppels All? Bygning 403, Rum 105 DK-2800 Kgs. Lyngby -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Jun 30 06:58:49 2021 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 30 Jun 2021 06:58:49 -0500 Subject: [petsc-users] Scatter parallel Vec to sequential Vec on non-zeroth process In-Reply-To: <93521e6acde64da2af7c415ceee9273c@mek.dtu.dk> References: <93521e6acde64da2af7c415ceee9273c@mek.dtu.dk> Message-ID: On Wed, Jun 30, 2021 at 6:49 AM Peder J?rgensgaard Olesen via petsc-users < petsc-users at mcs.anl.gov> wrote: > Hello > > > I have a parallel vector, and I'd like to copy its data to a sequential > vector located on a given process that is not necessarily the > 0th process, so using VecScatterCreateToZero wouldn't do here. > > > How might one approach this? > The function is very short: https://www.mcs.anl.gov/petsc/petsc-current/src/vec/is/sf/interface/vscat.c.html#VecScatterCreateToZero Just go in and replace '0' by the rank you want. Thanks, Matt > Med venlig hilsen / Best Regards > > > Peder J?rgensgaard Olesen > > PhD Student, Turbulence Research Lab > > Dept. of Mechanical Engineering > > Technical University of Denmark > > Koppels All? > > Bygning 403, Rum 105 > > DK-2800 Kgs. Lyngby > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From pjool at mek.dtu.dk Wed Jun 30 07:13:13 2021 From: pjool at mek.dtu.dk (=?iso-8859-1?Q?Peder_J=F8rgensgaard_Olesen?=) Date: Wed, 30 Jun 2021 12:13:13 +0000 Subject: [petsc-users] Scatter parallel Vec to sequential Vec on non-zeroth process In-Reply-To: References: <93521e6acde64da2af7c415ceee9273c@mek.dtu.dk>, Message-ID: <61644b633c624282a29f4e0ea80e61c7@mek.dtu.dk> Dear Matt Thank you for your reply. I'm guessing that the line if (rank) N = 0; should be changed to something like if (rank != target_rank) N = 0; such that an empty SEQVEC and scatter are created on each rank other than target_rank. Med venlig hilsen / Best regards Peder ________________________________ Fra: Matthew Knepley Sendt: 30. juni 2021 13:58:49 Til: Peder J?rgensgaard Olesen Cc: petsc-users at mcs.anl.gov Emne: Re: [petsc-users] Scatter parallel Vec to sequential Vec on non-zeroth process On Wed, Jun 30, 2021 at 6:49 AM Peder J?rgensgaard Olesen via petsc-users > wrote: Hello I have a parallel vector, and I'd like to copy its data to a sequential vector located on a given process that is not necessarily the 0th process, so using VecScatterCreateToZero wouldn't do here. How might one approach this? The function is very short: https://www.mcs.anl.gov/petsc/petsc-current/src/vec/is/sf/interface/vscat.c.html#VecScatterCreateToZero Just go in and replace '0' by the rank you want. Thanks, Matt Med venlig hilsen / Best Regards Peder J?rgensgaard Olesen PhD Student, Turbulence Research Lab Dept. of Mechanical Engineering Technical University of Denmark Koppels All? Bygning 403, Rum 105 DK-2800 Kgs. Lyngby -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Jun 30 07:24:13 2021 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 30 Jun 2021 07:24:13 -0500 Subject: [petsc-users] Scatter parallel Vec to sequential Vec on non-zeroth process In-Reply-To: <61644b633c624282a29f4e0ea80e61c7@mek.dtu.dk> References: <93521e6acde64da2af7c415ceee9273c@mek.dtu.dk> <61644b633c624282a29f4e0ea80e61c7@mek.dtu.dk> Message-ID: On Wed, Jun 30, 2021 at 7:13 AM Peder J?rgensgaard Olesen wrote: > Dear Matt > > > Thank you for your reply. I'm guessing that the line > > if (rank) N = 0; > > should be changed to something like > > if (rank != target_rank) N = 0; > > such that an empty SEQVEC and scatter are created on each rank other than > target_rank. > Yes Thanks, Matt > Med venlig hilsen / Best regards > > Peder > ------------------------------ > *Fra:* Matthew Knepley > *Sendt:* 30. juni 2021 13:58:49 > *Til:* Peder J?rgensgaard Olesen > *Cc:* petsc-users at mcs.anl.gov > *Emne:* Re: [petsc-users] Scatter parallel Vec to sequential Vec on > non-zeroth process > > On Wed, Jun 30, 2021 at 6:49 AM Peder J?rgensgaard Olesen via petsc-users < > petsc-users at mcs.anl.gov> wrote: > >> Hello >> >> >> I have a parallel vector, and I'd like to copy its data to a sequential >> vector located on a given process that is not necessarily the >> 0th process, so using VecScatterCreateToZero wouldn't do here. >> >> >> How might one approach this? >> > The function is very short: > https://www.mcs.anl.gov/petsc/petsc-current/src/vec/is/sf/interface/vscat.c.html#VecScatterCreateToZero > Just go in and replace '0' by the rank you want. > > Thanks, > > Matt > >> Med venlig hilsen / Best Regards >> >> >> Peder J?rgensgaard Olesen >> >> PhD Student, Turbulence Research Lab >> >> Dept. of Mechanical Engineering >> >> Technical University of Denmark >> >> Koppels All? >> >> Bygning 403, Rum 105 >> >> DK-2800 Kgs. Lyngby >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From junchao.zhang at gmail.com Wed Jun 30 09:00:25 2021 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Wed, 30 Jun 2021 09:00:25 -0500 Subject: [petsc-users] Scatter parallel Vec to sequential Vec on non-zeroth process In-Reply-To: <61644b633c624282a29f4e0ea80e61c7@mek.dtu.dk> References: <93521e6acde64da2af7c415ceee9273c@mek.dtu.dk> <61644b633c624282a29f4e0ea80e61c7@mek.dtu.dk> Message-ID: That should work. But VecScatterCreateToZero() is more efficient. So if possible, better refactor your code in such a way. --Junchao Zhang On Wed, Jun 30, 2021 at 7:13 AM Peder J?rgensgaard Olesen via petsc-users < petsc-users at mcs.anl.gov> wrote: > Dear Matt > > > Thank you for your reply. I'm guessing that the line > > if (rank) N = 0; > > should be changed to something like > > if (rank != target_rank) N = 0; > > such that an empty SEQVEC and scatter are created on each rank other than > target_rank. > > > Med venlig hilsen / Best regards > > Peder > ------------------------------ > *Fra:* Matthew Knepley > *Sendt:* 30. juni 2021 13:58:49 > *Til:* Peder J?rgensgaard Olesen > *Cc:* petsc-users at mcs.anl.gov > *Emne:* Re: [petsc-users] Scatter parallel Vec to sequential Vec on > non-zeroth process > > On Wed, Jun 30, 2021 at 6:49 AM Peder J?rgensgaard Olesen via petsc-users < > petsc-users at mcs.anl.gov> wrote: > >> Hello >> >> >> I have a parallel vector, and I'd like to copy its data to a sequential >> vector located on a given process that is not necessarily the >> 0th process, so using VecScatterCreateToZero wouldn't do here. >> >> >> How might one approach this? >> > The function is very short: > https://www.mcs.anl.gov/petsc/petsc-current/src/vec/is/sf/interface/vscat.c.html#VecScatterCreateToZero > Just go in and replace '0' by the rank you want. > > Thanks, > > Matt > >> Med venlig hilsen / Best Regards >> >> >> Peder J?rgensgaard Olesen >> >> PhD Student, Turbulence Research Lab >> >> Dept. of Mechanical Engineering >> >> Technical University of Denmark >> >> Koppels All? >> >> Bygning 403, Rum 105 >> >> DK-2800 Kgs. Lyngby >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pjool at mek.dtu.dk Wed Jun 30 09:07:17 2021 From: pjool at mek.dtu.dk (=?iso-8859-1?Q?Peder_J=F8rgensgaard_Olesen?=) Date: Wed, 30 Jun 2021 14:07:17 +0000 Subject: [petsc-users] Scatter parallel Vec to sequential Vec on non-zeroth process In-Reply-To: References: <93521e6acde64da2af7c415ceee9273c@mek.dtu.dk> <61644b633c624282a29f4e0ea80e61c7@mek.dtu.dk>, Message-ID: <3285e9ab0ba941e583998a3bb7a5c67c@mek.dtu.dk> Dear Junchao Zhang I'm distributing a set of independent tasks over different processes, so I'm afraid sending everything to the zeroth process would rather thoroughly defeat the purpose of what I'm doing. Med venlig hilsen / Best regards Peder ________________________________ Fra: Junchao Zhang Sendt: 30. juni 2021 16:00:25 Til: Peder J?rgensgaard Olesen Cc: Matthew Knepley; petsc-users at mcs.anl.gov Emne: Re: [petsc-users] Scatter parallel Vec to sequential Vec on non-zeroth process That should work. But VecScatterCreateToZero() is more efficient. So if possible, better refactor your code in such a way. --Junchao Zhang On Wed, Jun 30, 2021 at 7:13 AM Peder J?rgensgaard Olesen via petsc-users > wrote: Dear Matt Thank you for your reply. I'm guessing that the line if (rank) N = 0; should be changed to something like if (rank != target_rank) N = 0; such that an empty SEQVEC and scatter are created on each rank other than target_rank. Med venlig hilsen / Best regards Peder ________________________________ Fra: Matthew Knepley > Sendt: 30. juni 2021 13:58:49 Til: Peder J?rgensgaard Olesen Cc: petsc-users at mcs.anl.gov Emne: Re: [petsc-users] Scatter parallel Vec to sequential Vec on non-zeroth process On Wed, Jun 30, 2021 at 6:49 AM Peder J?rgensgaard Olesen via petsc-users > wrote: Hello I have a parallel vector, and I'd like to copy its data to a sequential vector located on a given process that is not necessarily the 0th process, so using VecScatterCreateToZero wouldn't do here. How might one approach this? The function is very short: https://www.mcs.anl.gov/petsc/petsc-current/src/vec/is/sf/interface/vscat.c.html#VecScatterCreateToZero Just go in and replace '0' by the rank you want. Thanks, Matt Med venlig hilsen / Best Regards Peder J?rgensgaard Olesen PhD Student, Turbulence Research Lab Dept. of Mechanical Engineering Technical University of Denmark Koppels All? Bygning 403, Rum 105 DK-2800 Kgs. Lyngby -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From junchao.zhang at gmail.com Wed Jun 30 09:34:38 2021 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Wed, 30 Jun 2021 09:34:38 -0500 Subject: [petsc-users] Scatter parallel Vec to sequential Vec on non-zeroth process In-Reply-To: <3285e9ab0ba941e583998a3bb7a5c67c@mek.dtu.dk> References: <93521e6acde64da2af7c415ceee9273c@mek.dtu.dk> <61644b633c624282a29f4e0ea80e61c7@mek.dtu.dk> <3285e9ab0ba941e583998a3bb7a5c67c@mek.dtu.dk> Message-ID: OK, then we should optimize this case. VecScatterCreateToZero() uses MPI_Gather/Gatherv. But in your case, petsc will use individual MPI send/recvs, though it is possible to use Gather/Gatherv. I created a follow-up issue https://gitlab.com/petsc/petsc/-/issues/950 Thanks. --Junchao Zhang On Wed, Jun 30, 2021 at 9:07 AM Peder J?rgensgaard Olesen wrote: > Dear Junchao Zhang > > > I'm distributing a set of independent tasks over different processes, so > I'm afraid sending everything to the zeroth process would rather > thoroughly defeat the purpose of what I'm doing. > > > Med venlig hilsen / Best regards > > Peder > ------------------------------ > *Fra:* Junchao Zhang > *Sendt:* 30. juni 2021 16:00:25 > *Til:* Peder J?rgensgaard Olesen > *Cc:* Matthew Knepley; petsc-users at mcs.anl.gov > *Emne:* Re: [petsc-users] Scatter parallel Vec to sequential Vec on > non-zeroth process > > That should work. But VecScatterCreateToZero() is more efficient. So if > possible, better refactor your code in such a way. > > --Junchao Zhang > > On Wed, Jun 30, 2021 at 7:13 AM Peder J?rgensgaard Olesen via petsc-users < > petsc-users at mcs.anl.gov> wrote: > >> Dear Matt >> >> >> Thank you for your reply. I'm guessing that the line >> >> if (rank) N = 0; >> >> should be changed to something like >> >> if (rank != target_rank) N = 0; >> >> such that an empty SEQVEC and scatter are created on each rank other than >> target_rank. >> >> >> Med venlig hilsen / Best regards >> >> Peder >> ------------------------------ >> *Fra:* Matthew Knepley >> *Sendt:* 30. juni 2021 13:58:49 >> *Til:* Peder J?rgensgaard Olesen >> *Cc:* petsc-users at mcs.anl.gov >> *Emne:* Re: [petsc-users] Scatter parallel Vec to sequential Vec on >> non-zeroth process >> >> On Wed, Jun 30, 2021 at 6:49 AM Peder J?rgensgaard Olesen via petsc-users >> wrote: >> >>> Hello >>> >>> >>> I have a parallel vector, and I'd like to copy its data to a sequential >>> vector located on a given process that is not necessarily the >>> 0th process, so using VecScatterCreateToZero wouldn't do here. >>> >>> >>> How might one approach this? >>> >> The function is very short: >> https://www.mcs.anl.gov/petsc/petsc-current/src/vec/is/sf/interface/vscat.c.html#VecScatterCreateToZero >> Just go in and replace '0' by the rank you want. >> >> Thanks, >> >> Matt >> >>> Med venlig hilsen / Best Regards >>> >>> >>> Peder J?rgensgaard Olesen >>> >>> PhD Student, Turbulence Research Lab >>> >>> Dept. of Mechanical Engineering >>> >>> Technical University of Denmark >>> >>> Koppels All? >>> >>> Bygning 403, Rum 105 >>> >>> DK-2800 Kgs. Lyngby >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Wed Jun 30 09:41:25 2021 From: jed at jedbrown.org (Jed Brown) Date: Wed, 30 Jun 2021 08:41:25 -0600 Subject: [petsc-users] Scatter parallel Vec to sequential Vec on non-zeroth process In-Reply-To: <3285e9ab0ba941e583998a3bb7a5c67c@mek.dtu.dk> References: <93521e6acde64da2af7c415ceee9273c@mek.dtu.dk> <61644b633c624282a29f4e0ea80e61c7@mek.dtu.dk> <3285e9ab0ba941e583998a3bb7a5c67c@mek.dtu.dk> Message-ID: <87y2arpg8a.fsf@jedbrown.org> Peder J?rgensgaard Olesen via petsc-users writes: > I'm distributing a set of independent tasks over different processes, so I'm afraid sending everything to the zeroth process would rather thoroughly defeat the purpose of what I'm doing. It sounds like you're going to run this a bunch of times. Does it have to be sequential (gather to one rank at a time) or can it be an alltoall? From a.croucher at auckland.ac.nz Wed Jun 30 21:17:32 2021 From: a.croucher at auckland.ac.nz (Adrian Croucher) Date: Thu, 1 Jul 2021 14:17:32 +1200 Subject: [petsc-users] Boundary ghost cell types in PETSc 3.15 In-Reply-To: References: <4d88edb0-9a30-6462-02e7-2f6d3b52d4e8@auckland.ac.nz> Message-ID: <6ae64a30-3d19-e5d7-5b62-01440e79a551@auckland.ac.nz> On 30/06/21 11:17 pm, Matthew Knepley wrote: > > Yes, src/dm/f90-mod/petscdm.h is missing PYRAMID. I will make a > fix right now. > > > Here it is: https://gitlab.com/petsc/petsc/-/merge_requests/4140 > > Aha, yes that's fixed it. Thanks Matt! - Adrian -- Dr Adrian Croucher Senior Research Fellow Department of Engineering Science University of Auckland, New Zealand email: a.croucher at auckland.ac.nz tel: +64 (0)9 923 4611 -------------- next part -------------- An HTML attachment was scrubbed... URL: From aduarteg at utexas.edu Wed Jun 30 21:17:55 2021 From: aduarteg at utexas.edu (Alfredo J Duarte Gomez) Date: Wed, 30 Jun 2021 21:17:55 -0500 Subject: [petsc-users] Visualization and I/O Message-ID: Good evening, My group and I are in the process of developing an application using PETSc, and we are currently planning and figuring out the best objects to do this. I currently have some questions about visualization and I/O. Some of the most important features that we want in the process of developing this code is the ability for easy checkpoint/restarts and visualization. As we are working with a time dependent PDE, I have stumbled upon the "ts" and "tstrajectory" objects that seem to have some very helpful features. However, I am having some trouble extracting information out of the the "tstrajectory" object in an HDF5 format. I initially thought something like this would work, where tj was obtained using TSGetTrajectory(ts,&tj) after a successful solve of the ts object: PetscViewer viewer; PetscViewerHDF5Open(PETSC_COMM_WORLD,"output.h5",FILE_MODE_WRITE,&viewer);CHKERRQ(ierr); TSTrajectoryView(tj,viewer);CHKERRQ(ierr); PetscViewerDestroy(&viewer);CHKERRQ(ierr); However, the output file comes out empty. Do I have to extract the solutions one by one? How would I do that? Does the HDF5 viewer only work with vector objects or can it be generalized to other objects? I am trying to maintain the highest degree of abstraction possible (i.e. multiple 2d ode solutions in one hdf5 file). This seems favorable to me because it would be easy to use the HDF5 file for postprocessing/visualization and restarts, but let me know if you think this is a bad idea and it would be better to just use the HDF5 for output/postprocess and use native PETSc binary files for restarts. Additionally, as I move towards large visualizations, I would like to use Paraview. I saw that the best way of doing this is to use the petsc_gen_xdmf.py script on an HDF5 solution file. Is this still the best way for PETSc visualization with Paraview? Thank you, -- Alfredo Duarte Graduate Research Assistant The University of Texas at Austin -------------- next part -------------- An HTML attachment was scrubbed... URL: From hongzhang at anl.gov Wed Jun 30 23:37:02 2021 From: hongzhang at anl.gov (Zhang, Hong) Date: Thu, 1 Jul 2021 04:37:02 +0000 Subject: [petsc-users] Visualization and I/O In-Reply-To: References: Message-ID: <2E66C4CD-0B25-4FC6-85C8-6654AD492803@anl.gov> Alfredo, TSTrajectory currently supports binary format only. See https://www.mcs.anl.gov/petsc/petsc-current/src/ts/trajectory/impls/basic/trajbasic.c.html#TSTrajectorySet_Basic https://www.mcs.anl.gov/petsc/petsc-current/src/ts/trajectory/impls/visualization/trajvisualization.c.html#TSTrajectorySet_Visualization It should not be difficult to extend TSTrajectory for HDF5 support. You are welcome to contribute an MR to add this feature. Thanks, Hong (Mr.) On Jun 30, 2021, at 9:17 PM, Alfredo J Duarte Gomez > wrote: Good evening, My group and I are in the process of developing an application using PETSc, and we are currently planning and figuring out the best objects to do this. I currently have some questions about visualization and I/O. Some of the most important features that we want in the process of developing this code is the ability for easy checkpoint/restarts and visualization. As we are working with a time dependent PDE, I have stumbled upon the "ts" and "tstrajectory" objects that seem to have some very helpful features. However, I am having some trouble extracting information out of the the "tstrajectory" object in an HDF5 format. I initially thought something like this would work, where tj was obtained using TSGetTrajectory(ts,&tj) after a successful solve of the ts object: PetscViewer viewer; PetscViewerHDF5Open(PETSC_COMM_WORLD,"output.h5",FILE_MODE_WRITE,&viewer);CHKERRQ(ierr); TSTrajectoryView(tj,viewer);CHKERRQ(ierr); PetscViewerDestroy(&viewer);CHKERRQ(ierr); However, the output file comes out empty. Do I have to extract the solutions one by one? How would I do that? Does the HDF5 viewer only work with vector objects or can it be generalized to other objects? I am trying to maintain the highest degree of abstraction possible (i.e. multiple 2d ode solutions in one hdf5 file). This seems favorable to me because it would be easy to use the HDF5 file for postprocessing/visualization and restarts, but let me know if you think this is a bad idea and it would be better to just use the HDF5 for output/postprocess and use native PETSc binary files for restarts. Additionally, as I move towards large visualizations, I would like to use Paraview. I saw that the best way of doing this is to use the petsc_gen_xdmf.py script on an HDF5 solution file. Is this still the best way for PETSc visualization with Paraview? Thank you, -- Alfredo Duarte Graduate Research Assistant The University of Texas at Austin -------------- next part -------------- An HTML attachment was scrubbed... URL: